The Critical Need for Validation

Unlike A/B tests, quasi-experiments lack the fundamental guarantee of unbiasedness provided by randomization. This makes validation, diagnostics, and trust-building absolutely critical. If we cannot randomize the treatment assignment, we must rely on a large set of control variables (confounders) to adjust for pre-exposure differences between the treatment and control groups. This adjustment process makes the analysis highly susceptible to bias if not correctly executed and rigorously checked. The core validation pillars are centered on managing confounders and monitoring model diagnostics, as we’ll explore in the next section.

The Validation Engine: Confounders and Diagnostics

Since quasi-experiments lack the inherent randomization of A/B tests, we must prove the validity of our causal estimates through rigorous inputs and explicit model diagnostics. We build trust in every single result through Confounder Management and a Diagnostic Scorecard.

1. Rigorous Data and Confounder Inputs

The greatest threat to a quasi-experiment’s validity is selection bias — users with certain characters are more likely to be treated by the policy, making the control group not a proper counterfactual for the treatment group. The only way to correct for this is by meticulously identifying and controlling for these pre-existing differences, known as confounders. At Lyft, we have developed a quasi-experimentation platform that makes this process mandatory and customizable.

a) The Confounder Set Requirement

The analysis requires potentially hundreds of features/confounders to adequately reduce bias. The quasi-experimentation platform doesn’t allow users to skip this step; users are required to select or define one confounder set for AIPW analysis.

• Pre-defined sets: Users can choose from established sets on the platform that are pre-defined as SQL queries to pull the relevant data.
• Customization is key: Crucially, the platform exposes the underlying SQL query, allowing users to customize, modify, add, or remove variables within a set to perfectly match their specific use case.
• Preventing leakage: The system automatically ensures confounder data is gathered from before the user’s first exposure date to the treatment, preventing leaky covariates that would incorrectly attribute the treatment’s effect to a non-causal variable.

b) The Balancing Act: Correcting for Downsampling Bias

A core task in data preparation for AIPW is balancing the size of the treatment and control groups in the presence of imbalance data. When the one group is smaller than the other, the system randomly downsamples the larger group to achieve balance. However, this random downsampling introduces a new scientific challenge:

• Non-Representative Samples: Even if the original sample satisfies model assumptions across treatment groups, taking a random subset of the larger group may make that subset non-representative of the true population distribution with respect to the confounders.

To recover the true population-level ATE, we apply a correction in the AIPW estimates per Ballinari (2024), which involves two related concepts:

i. Propensity Score Correction: We must convert the sample-estimated propensity score, p_s(X), back into the true population propensity score, p(X), using a conversion formula that accounts for the downsampling ratio L. This ensures the model uses the real probability of being treated in the population, not just in the sample.

ii. Outcome Reweighting: After the propensity score correction, the efficient scores must be subjected to a weighted average based on the sampling ratio. Specifically, for every observation in the downsampled control group, we must account for the fact that it represents 1/L copies of the original population. This process involves uniformly rescaling the weights of each observation so they average to 1.

This reweighting of outcomes is a critical scientific refinement currently being implemented to debias the results and ensure the final ATE estimate accurately reflects the total impact on the original population.

2. Model Diagnostics and Assumptions

The output of every AIPW analysis on our quasi experimentation platform is a scorecard with two essential tabs: the Scores Tab and the Diagnostic Tab. The Diagnostic Tab is where we evaluate the model’s health to look for clues that its fundamental assumptions hold, providing visual proof of the estimation quality. Below are two of the tens of diagnostics we show users:

i. Checking Common Support (Propensity Overlap)

The AIPW model relies on the common support assumption (or overlapping assumption), meaning that for any given set of confounders, there must be a non-zero probability of being in either the treatment or control group (0<e(X)<1). If this assumption is violated, the inverse propensity score weights explode. This means that the outliers will receive an extreme weight and take over the overall effect.

The platform provides a histogram visualizing the Propensity Overlap between the control and treatment groups. Below are examples of sufficient and insufficient overlaps between treated and control groups

If the overlap is poor, the causal estimates are unreliable. The analysis is further refined by applying a user-defined trim level to discard extreme propensity score values and satisfy the common support assumption.

ii. Ensuring Covariate Balance

The primary goal of using confounders is to achieve a state where the treatment and control groups are statistically similar on all observed characteristics.

A key diagnostic graph illustrates the Covariate Balance before and after adjustment.

• Before Adjustment (Original): Shows the initial difference (bias) in each feature between the groups.
• After Adjustment (IPW/IPW Trimmed): Shows how well the model has reduced the difference. Successful adjustment moves the metrics close to the dotted zero line, confirming that the confounders are properly balanced.