Lecture 7: Causality via 2SLS, Difference-in-Differences, and Synthetic Controls [FINISHED]

Synthetic Control methods are the "most important development in program evaluation in the last decade."

-- Susan Athey and Guido Imbens (2016) "The State of Applied Econometrics - Causality and Policy Evaluation."

We know that correlation is not causation but it can be tricky to identify the latter from the former. Causation is measuring the real impact on Y because of X; e.g., What is the effect of ad campaigns on the sales of a product?

It is critical to precisely understand the causal effects of these interventions on the subject. One of the main threats to causal inference is the confounding effect from other variables. In the case of ad campaigns, it could be a reduction in price of the product, change in the overall economy or various other factors that could be inducing the change in sales at the same time. So how do we correctly attribute the change in sales because of the ad campaign?

There are two ways to estimate the true causal impact of the intervention on the subject.

• Randomized Experiment: The researcher designs an experiment such that participants are assigned the treatment at random. Since the treatment is randomized, the group of participants which receive the treatment are observationally identical to residents who do not receive the treatment (e.g., vaccine trials). The researcher follows the two groups to evaluate whether the treatment affects outcomes of interest. This approach is the most reliable method to infer the actual causal impact of the treatment a lot of interesting and important questions cannot be addressed via experimentation for a variety of logistical and ethical reasons.

• Causal Inference: The alternative is to apply statistical procedures to existing data to estimate causality indepent of confounding factors. This will be our focus. Note that instrumental variables (IV) regression falls in this category.

Specifically, we'll discuss the following in this lecture:

1. Estimating a Demand Curve
   
   

2. The Data: Terrorist Bombings in the Basque Country
   
   

3. First Difference
   
   

4. Difference in Differences
   
   

5. Synthetic Control

Class Announcements

PS2 is due by end of day today (Tuesday) to eLC. Extra office hours Tuesday afternoon until 4 pm.

1. Estimating a Demand Curve (top)

1.1 Correlation¶

We load the data as a Pandas dataframe and estimate the demand curve equation by OLS using the HC3 estimator for the standard errors, print the summary table of the results.

We're interested in the estimated value of the price elasticity of demand.

OLS Estimation¶

Practice ¶

1. Explain why the interaction of supply and demand will make our OLS estimates biased. Can you sign the bias?

Answer:

Prices and quantities are determined by the intersection of supply and demand in market equilibrium. Due to this simultaneity problem, we cannot simply run OLS to estimate the demand curve. If we do run OLS, it is unclear which curve we are actually estimating -- in some sense we are not (consistently) estimating either supply or demand!

Alternatively, you could think of $u_i$ as an unobserved factor that shifts demand. If $u_i$ is positive, then demand is shifted to the right, which would increase both price and quantity when supply is held fixed. Thus $Cov(\ln(P_i), u_i) > 0$ and the OLS coefficient is biased upwards.

If you would like to know more, this note by Douglas Steigerwald is provides a more detailed explanation.

1.2 Causality¶

We separate causation from correlation by introducing an instrument. Intuitively, we need something that shifts supply but not demand. If we have a variable that exogenously shifts the supply curve, we can use these shifts to trace out the demand curve. An exogenous change in firm costs (due to changes in input prices, taxes, regulations) is a great instrument because it elicits a change in the firm's pricing decision.

A subtle and interesting point is all products are connected in the (pricing) equilibrium:

• a cost change for one product of a multiproduct firm effectively changes prices for all products sold by the firm.
• a cost change for one firm leads to other firms changing their prices.

Back to the JEC case

We could use $cartel_i$ (see JEC_data_description.pdf for this variable's definition) as an instrumental variable for $\ln(P_i)$.

Does this (plausibly) satisfy the two conditions for a valid instrument?

For $cartel_i$ to be a valid instrument for $\ln(P_i)$, it must satisfy the following two conditions:

1. Relevance: $Cov(cartel_i,\ln(P_i)) \neq 0$

1. Exogeneity: $Cov(cartel_i, u_i) = 0$

The first condition is satisfied: the cartel acts to restrict quantity and increase prices, so $Cov(cartel_i,\ln(P_i)) > 0$.

The second condition requires that the operation of the cartel was unrelated to other unobserved factors that shift the demand for rail transport of grain. We know that the cartel faced an information asymmetry problem and would randomly penalize itself so it seems plausible that $Cov(cartel_i, u_i) = 0$.

2SLS Estimation¶

We'll estimate the demand equation using two stage least squares with $cartel_i$ as an instrument for $\ln(P_i)$. We're interested in the demand price elasticity which amounts to the estimated coefficient on $\ln(P)$.

NOTE: You might need to install the linearmodels package, i.e.,

pip install linearmodels

**Here is a subtle but important point -- a good price instrument makes demand more elastic. We know this from economic theory.**

2. Terrorist Bombings in the Basque Country (top)¶

We're going to estimate the economic impact of terrorism in the Basque region -- an autonomous community in Spain. To do this, we'll use information from 17 other Spanish regions where our underlying assumption will be that terrorism affected economic activity in the Basque region but not elsewhere whereas other economic shocks are aggregate and affect all of Spain.

Here are the details:

• Coverage from 1955–1997 for 18 Spanish regions. One of the data "regions" is all of Spain which we won't use.
• The treatment region is “Basque Country (Pais Vasco)”.
• The "treatment" year is 1975 since there were several bombings around that year.
• We will measure "economic impact" via GDP per capita (in thousands).

Background: Euskadi Ta Askatasuna (ETA), was an armed leftist Basque nationalist and separatist organization in the Basque Country (in northern Spain and southwestern France). The group was founded in 1959 and later evolved from a group promoting traditional Basque culture to a paramilitary group engaged in a violent campaign of bombing, assassinations and kidnappings in the Southern Basque Country and throughout Spanish territory. Its goal was gaining independence for the Basque Country. ETA was the main group within the Basque National Liberation Movement and was the most important Basque participant in the Basque conflict. While its terrorist activities spanned several decades, the death of Spanish dictator Francisco Franco in 1975 led to a substantial increase in bombings. (Wikipedia)

Loading Data from R¶

We've learned how to load data from csv, excel, and stata. Another popular programming language (and to some degree a rival of python) is the open-source language R. The data we want to access is from the following academic paper

Abadie, A. and Gardeazabal, J. (2003) "Economic Costs of Conflict: A Case Study of the Basque Country." American Economic Review 93 (1) 113--132.

where the data was saved in the R programming language. We'll do this by leveraging python's own open-source nature and load a user-created package called pyreadr which we install as usual in the terminal:

pip install pyreadr

We then use pyreadr to load basque.RData.

Before proceeding into the econometrics, it's useful to graph the data variation we're most interested in. In this project, we're interested in evaluating the effect of terrororism on GDP per capita where we're using the bombings in 1975 as a "natural experiment." We'll do so using Plotly.

We observe that per capita GDP decreases after the 1975 increase in ETA terrorist bombings and then increases. How much of that decrease is due to terrorism and how much is due to general economic uncertainty due to Franco's death?

3. First-Difference (top)¶

Thus The far : Terrorist Bombings in the Basque Countrywe've learned how to load data from csv, excel, and stata. Another popular programming language (and to some degree a riva

We'll begin by comparing the per capit GDP before and after the 1975 treament year. One way to do this is to construct a "first difference" equation by having per capita GDP as the dependent variable and adding a pre-post indicator as the independent variable.

where $1_{\{t>1975\}}$ is equal to one for all years after 1975 and zero otherwise.

The regression indicates that the ETA terrorishm increased GDP per capita by $\approx$2.5 units. Seems odd. What's happening?

Compare the graph and the regression. The 2.5 unit increase is because of trend growth but GDP could be growing for a lot reasons and frankly what we're interested in is understanding how ETA terrorism may have inhibited that growth. What we need to do is separate the effect of the bombings from all of the things which could be confounding the results.

BTW, I hope some part of you is thinking "This regression is dumb because all it's really doing is comparing average per capita GDP before 1975 to average per capita GDP after wards." To see this, let's simply compute these averages and compare:

Per capita GDP after treatment is 7.87 while before treatment is 5.38 with a difference of -- not surprisingly -- 2.48!

Let's see what happened graphically:

4. Difference-in-Differences (DiD) (top)¶

We'll establish causality by comparing the trend of a "control" group to the Basque region. This is like conducting a an experiment where you give some patients a vaccine, drug, etc. while you give other patients a placebo. In that experimental setting, we design the experiment such that the control and treatment groups are identical but for the treament applied. We then observe whether or not outcomes-of-interest are significantly different between the two groups.

The same rationale holds here but of course we don't have an experimental setting so we choose a control group after the fact under the assumption this group provides an adequate proxy for the trend that would have been observed in the treatment group in the absence of treatment. if true, the difference in change of slope would be the actual treatment effect. Hence, the "difference in differences" terminology. Note that our "identification strategy" requires the treatment and control groupsfollow the same pre-period (i.e., pre-treatment) trend which is something we can easilly show.

In fact, we'll choose our control group using the following criterion: lowest variation in % difference of pre-period per capita GDP across years between each region and the Basque country.

Aragon follows the Basque country not only pre-period but also post-period. Look at a map and you observe that the Basque country and Aragon are neighbors so it seems reasonable that GDP effects related to ETA activities may have also had an impact on Aragon so the latter is not a great control. Cataluna, however, is farther away so it better achieves our "exclusion restriction". Let's redo the analysis using the DiD estimator as follows:

Discussion: Using the Cataluna region as control indicates that ETA-related terrorism reduced per capita GDP by 0.14 units. Moreover, this decline is statistically significant.

5. Synthetic Control (top)¶

Generally speaking (umm... writing), the "synthetic control" method allows for estimation in settings where a single unit (a state, country, firm, etc.) is exposed to an event or intervention but it's not obvious who or what the control group should be. Specifically, synthetic control provides a data-driven procedure to construct a control group using a convex combination of comparison units. The idea is that this generated (ie, synthetic) control group approximates the characteristics of the unit of interest prior to treatment. The thought (hope) is that a combination of comparison units provides a better comparison for the unit exposed to the intervention than any single comparison unit.

In our example above, we identfied Cataluna as our control variable by generating a metric to assess pre-period trends. Fortunately, Catalunya turned out to be a nice fit -- how fortunate! In many (most?) cases, we will have too much data to identify a control group by "eyeballing" the data and we'll likely not be able to satisfy the parallel trend pre-period assumption. What if instead we could create a control group via a convex combination of potential candidates? That's the idea behind "synthetic controls" -- we create a control group.

Synthetic control is a technique which is very similar to DiD in estimating the true impact of a treatment. Both the methods use the help of control groups to construct a counter-factual of the treated group giving us an idea of what the trend is if the treatment had not happened. The counter-factual GDP of the treated group would be predicted by the GDP of the control groups and also other possible covariates in the control group.

Just as the DiD approach used the control (Catalunya) to construct a counter-factual of the treated group giving us an idea of what the trend is if the treatment had not happened, "synthetic control" predicts the counter-factual by assigning weights to the regressors in the control groups identify individual regressors and their influence in prediction. Ultimately, the true causal impact is the difference in GDP between actual GDP and the counter-factual GDP if the treatment had not happened which is the same idea as DiD.

As before, let's look at the raw data before proceeding:

Idea: What we'll be doing is picking the convex combination of regions to best match the evolution of per capita GDP prior to the ETA bombings (i.e., before 1975). We'll then hold these weights fixed and generate per capita GDP for the synthetic post-treatment control group. That's it. Identifying the causal effect then follows our DiD regression where we use the synthetic control rather than a specific control (e.g., Cataluna). A nice feature of this approach is that we can be hands-off in selection of our control group. Plus, construction of our synthetical control is not based on the treatment period so there's minimal risk of inadvertantly baking-in our results.

Caveat Emptor: The following code use our knowledge of machine learning (LASSO) to generate the candidate weights. While machine learning algorithms like this have become increasingly popular to generate weights for the synthetic control group, the traditional approach is to estimate weights which sum to one and are non-negative. LASSO imposes neither. If you want to get into the weeds, the conditions of non-negative weights which also sum to one are important to guarantee the weights used to construct the synthetic control group are unique (i.e., there is only one way to generate the pre-sample so there is also only one match for constructing the synthetical control after treatment). That said, I like my approach as it instills greater intuition, connects to what we've done before, and does not involve a big fancy black-box function (and the accompanying annoying syntax). **At the end of the notebook we construct the synthetic control group "properly."**

Let's get to work...

The graph demonstrates the following:

1. The synthetic control fits pre-period by construction.

1. The actual effect is then the divergence post-period. In the graph this is the difference between the blue and red lines.

The difference in trends in the post-period will be our "average treatment effect" which we will quantify via the DiD approach using the "synthetic control" (i.e., post-period prediction using pre-period LASSO coefficients as weights) as the control group.

Discussion: We find that the synthetic control method reduces the negative effect of ETA bombings on the Basque economy from -0.14 (DiD) to -0.10. These point-estimates are not statisticsally different from eachother (though both are statistically different than zero) which indicates that in this case the added toil of synthetic controls didn't add a lot quantitatively. Why?

Look back at the graph of GDP per capita for all regions and note that the regions look very similar to Cataluna. Hence, generating a convex combination of these others is pretty much the same as using Cataluna. Of course, this result is particular to our example and not generally true but it demonstrates that the Synthetic Control method nests the Difference-in-Differences approach.

The Effect of ETA Bombings The above approach demonstrates the average effect but we can estimate the treatment effect as the gap between treated and the synthetic control outcomes:

where $y_{lt}$ and $\hat y_{t}^S$ are the actual and (synthetic) control outcomes, resptively, and $\tau_{t}$ is the effect we're after. Let's plot it:

Practice ¶

Let's try an example on your own to practice this technique.

You will estimate the effect of cigarette taxation on consumption:

• Taxes will increase the cost of cigarettes and lower their demand.
• Since cigarettes cause addiction, consumers will be unresponsive so a change in their price won’t effect demand much and the only effect of the policy is to tax users (which may be good if such funds are used to offset medicare expeninditure related to smoking).

Whether such a policy is pro-active and leads consumers to consumer less tobacco will be your task to find out.

The Data

In 1988, California passed the Tobacco Tax and Health Protection Act also known as Proposition 99. The primary effect was to

impose a 25-cent per pack state excise tax on the sale of tobacco cigarettes within California, with approximately equivalent excise taxes similarly imposed on the retail sale of other commercial tobacco products, such as cigars and chewing tobacco. Additional restrictions placed on the sale of tobacco include a ban on cigarette vending machines in public areas accessible by juveniles and a ban on the individual sale of single cigarettes. Revenue generated by the act was earmarked for various environmental and health care programs, and anti-tobacco advertisements.

To evaluate whether this tax reduced smoking, have data from 1970 to 2000 from 39 states.

1. Load smoking.csv. Drop variables lnincome, beer, and age15to24. Convert year to datetime and make it the index. The variable cigsales is average annual per capita packs sold.

1. Plot cigarette sales (cigsale) across time for (a) California and (b) the average for the other states. Be sure to indicate 1988 when Proposition 99 goes in to effect. What trends do you see?

1. Use LASSO to create a Synthetic Control group using the 38 other states as the control group. Use 5-fold cross-validation to choose the penalty parameter $\alpha$ (ie, cv=5).

1. Plot cigarette sales for California and the Synthetic Control you created. Be sure to indicate 1988 when Proposition 99 goes in to effect. What trends do you see?

1. Use the Synthetic Control approach to estimate the effect of Proposition 99 for all states. This requires running creating a synthetic control group for every state so create a loop across states in the data. Plot $\tau_t$ for each state and discuss the results.
   • For California, this is the treatment effect.
     
     
   • For the other states, this is a placebo effect since passage of Proposition 99 in California should not have effected cigarette sales in -- say -- Nevada. If it does, we have a problem and our control group needs to be refined.

Looks promising but messy. There are a lot of outliers in the pre-period. Let's drop them.

1. Report the estimated effect of Proposition 99 on cigarette smoking in 2000.

1. Build a synthetic control group using the elastic net model. Report the estimated effect of Proposition 99 on cigarette smoking in 2000. Again, use 5-folds cross-validation to solve for the penalty parameters.

Constraining the Weights to Construct the Synthetic Control¶

We actually should have constrained the weights to be positive and sum to one. The idea is that we want to force the algorithm (ie, fitting exercise) to only do interpolatation. Restricting the weights to be positive and sum up one makes the synthetic control a convex combination of the units of candidate controls.

Practically, doing this procedure requires us to construct our own minimization routine -- which I do below.