Natural experiments help answer important questions - The price in economic sciences 2021 - Article review

This document contains article review "Natural experiments help answer important questions" The price in economic sciences 2021
To order to read the article select: https://www.nobelprize.org/uploads/2021/10/popular-economicsciencesprize2021-3.pdf

Contents

Reflection

Page 1

Introduction

They (Natural experiments) have also clarified exactly which conclusions about cause and effect can be drawn using this research approach.
The use of the word exactly is misleading. Concepts like cause and effect are always surrounded by a cloud of uncertainty. They cannot be described exactly
If we are to make good decisions, we must understand the consequences of our choices.
This is true for all our decisions. The main issue is that we don't know if they are good or bad.
However, answering broad questions about cause and effect is not easy, because we will never know what would have happened if we had made a different choice.
The main problem is that any invention often can be used with two intentions: good and bad.
One way of establishing causality is to use randomised experiments, where researchers allocate individuals to treatment groups by a random draw. This method is used to investigate the efficacy of new medicines, among other things, but is not suitable for investigating many societal issues - for example, we cannot have a randomised experiment determining who gets to attend upper-secondary school and who does not.
The use of the word randomised is unlucky. The issue is if we can use experiments for medical experiments and for educational purposes. The answer is: Yes. But both involve certain issues.
To test the efficacy of the first a Covid vaccine by random draws on individuals is unethical.
These natural experiments may be due to natural random variations, institutional rules or policy changes. In pioneering work from the early 1990s, David Card analysed some central questions in labour economics - such as the effects of a minimum wage, immigration and education - using this approach
For all the 3 questions it is important if experiments can be used to answer the effects on both small and large scale.

Page 2

2. One example of a natural experiment

For example, for men born in the US during the 1930s, earnings were, on average, seven per cent higher for those with one additional year of education.
That is considered a fact. However, there is no indication how many people were involved in each age group.
So, can we conclude that an extra year of education adds an extra seven per cent on your income?
On average the answer is Yes.
The answer to this question is no - people who choose a long education differ in many ways from those who choose a short education. For example, some people may be talented at studying and at working. These people are likely to continue studying, but they would still probably have had a high income even if they hadn't. It may also be the case that only people who expect education to pay off choose to study longer.
This are answers on completely different questions. It like asking the question: What happens if all men born in the 1930's followed 16 years of education. Did they all earn more?
If one earned less the answer is: No.

Page 3

So, how can we use a natural experiment to examine whether additional years of education affect future income? In the US etc some children who are born in a particular calendar year start school on the same date, children who are born early in the year can leave school sooner than children born later in the year. etc.
A comparison between people born in the first and fourth quarters of the year, revealed that the first group had, on average, spent less time in education.
It is completely logical that they receive less education if they leave school at the first quarter.
People born in the first quarter also had lower incomes than those born in the fourth quarter.
That is what you can expect, but is a much more difficult effect to measure, specific if you measure their income 20 years after they leave school compared to 2 years.
As adults they thus had both less education and lower incomes than those born late in the year.
Specific the issue of lower incomes is less straight forward. The solution is that people should only leave school at the end of a school year.

Because chance decides exactly when a person is born, Angrist and Krueger were able to use this natural experiment to establish a causal relationship showing that more education leads to higher earnings: the effect of an additional year of education on income was nine per cent.
The experiment discussed consists of two groups. What makes the experiment special is that the people who participate cannot influence to which group they belong.
The outcome of the experiment is that you should make the school system everywhere the same.
But when that is the case, it is still interesting to test the causal relationship that more education leads to higher earnings.
It was surprising that this effect was stronger than the association between education and income, which amounted to seven per cent.
If ambitious and intelligent people have both high levels of education and high incomes (regardless of education) the result should have been the opposite; the correlation should have been stronger than the causal relationship
This sentence is not clear. You cannot compare 'correlation' and 'causal relationship'
This observation raised new questions about how to interpret the results of natural experiments - questions that were later answered by A and I

People born late in the year have more years of education and higher incomes

Additional years of education have a positive effect on income.
The figure uses data from Angrist and Krueger
In this particular example it is better to write: Additional months of education have a positive effect.
It would be easy to believe that situations which enable natural experiments are very unusual, especially those that can be used to answer important questions. Research conducted over the past 30 years has shown that this is not the case: natural experiments occur frequently.
The issue if to what extend situations or processes, have a clear dividing line (see next sentence) or set of parameters. At the same if you want to investigate this border line you need a monitoring system.
Any experiment, in general, can be performed at any moment. To decide that a certain experiment is actual a natural experiment you need 'luck'.
There is thus unintended randomness that divides people into control and treatment groups, providing researchers with opportunities for uncovering causal relationships.
My own experience, at elementary level, during my youth, is that the school system was more uniform, compared with the present. The current school system is much more complex. To investigate, in general, what the influence of elementary education is on each person its whole career, is very difficult. Repeating a class, I evaluate as beneficiary.

Page 4

3. Understanding labour markets

3.1. The effects of minimum wage

To investigate how increased minimum wages affect employment, Card and Krueger used a natural experiment. In the early 1990s, the minimum hourly wage in New Jersey was raised from 4.25 dollars to 5.05 dollars. Just studying what happened in New Jersey after this increase does not give a reliable answer to the question, as numerous other factors can influence how employment levels change over time. As with randomised experiments, a control group was needed, i.e., a group where wages didn't change but all the other factors were the same.
This is the same for all types of experiments. If you want to understand what is happening you should change only one parameter. All the others should be the same.

The effect of increasing the minimum wage

Card and Krueger used a natural experiment to study how increasing the minimum wage affects employment.
Okay.
The researchers identified a treatment group (restaurants in New Jersey) and a control group (restaurants in eastern Pennsylvania) to measure the effect of increasing the minimum wage.
The most important issue what effect they measured and if that effect is influenced by the wage its inhabitants receive.
Card and Krueger noted that there was no increase in neighbouring Pennsylvania.
Important. That means they were lucky. (Your author lived there.)
So, they studied the effects on employment in two neighbouring areas - New Jersey and eastern Pennsylvania - which have a similar labour market, but where the minimum wage was increased on one side of the border but not the other.
To perform so a study you must have the hindsight to monitor the situation in both states before the increase i.e., before 1990.

Page 5

There was no apparent reason to believe that any factor (such as the economic situation) apart from the increase in the minimum wage would affect employment trends differently on either side of the border.
The issue is that during the period of this experiment there are no other effects involved.
Thus, if a change in the number of employees was observed in New Jersey, and this differed from any change on the other side of the border, there was good reason to interpret this as an effect of the increase in the minimum wage.
This implies a true correlation.

Card and Krueger focused on employment in fast-food restaurants, an industry where pay is low and minimum wages matter. Contrary to previous research, they found that an increase in the minimum wage had no effect on the number of employees.
That means that the number of employees in in fast-food restaurants on both sides of the border did not change.
The opposite would be much more interesting.
5 The overall conclusion is that the negative effects of increasing the minimum wage are small, and significantly smaller than was believed 30 years ago.
This is an important conclusion and requires a detailed investigation.

3.2 Research on immigration and education

Another important issue is how the labour market is affected by immigration. To answer this question, we need to know what would have happened if there had not been any immigration.
That is not true. You can start to collect statistics any moment in time.
. A unique event in the history of the US gave rise to a natural experiment, which David Card used to investigate how immigration affects the labour market. In April 1980, Fidel Castro unexpectedly allowed all Cubans who wished to leave the country to do so.
Okay
To examine how this huge influx of workers affected the labour market in Miami, David Card compared the wage and employment trends in Miami with the evolution of wages and employment in four comparison cities.
Okay.

For example, follow-up studies have shown that increased immigration has a positive effect on income for many groups who were born in the country, while people who immigrated at an earlier time are negatively affected.
That means immigration has a negative effect.

Page 6

However, one problem was that previous work had not considered the possibility of compensatory resource allocation.
Okay

Local average treatment effect.

Joshua Angrist and Guido Imbens showed how natural experiments can be used to arrive at precise conclusions about cause and effect.
Okay
Natural experiments differ from clinical trials as the researcher does not have complete control over who receives the treatment.
The question is how important this is. At the same time, it can be very important to have certain criteria who can participate in an experiment and who not.
Within this section there is a computer screen. The screen consists of two parts. 
*------------------------------------------------------------------------------------------------*
|The left part contains the text "OFFERED COMPUTING COURSE". The right part "NOT OFFERED"        |       
*------------------------------------------------------------------------------------------------*
Each side contains 10 emoticons: 8 smilies, 1 smilie with the text "Would have taken the course any way", 1 smilie with the text: Not interested.
Below the screen sits a researcher.
The researcher wants to evaluate the course and receives data on who completed it.
Okay, but not okay.
However, some participants might have studied the course anyway, and are thus unaffected by the experiment.
Okay, but not okay.
Why are these participants not affected? Every one participating is always affected in some way.
The people who are affected are those who took the course because it was offered to them, but the researcher does not know who they are.
I assume that the researcher has no additional information about who the participants are. Etc. But that is not clearly stated.
Reflection 4 - Natural Experiment: Call centre training programme.

4. A new framework for studies of causal relationships.

In all realistic scenarios, the effect of an intervention - for example, the effect of additional schooling on earnings - varies between people.
What this sentence implies that there is no relation (among people) between extra schooling and one's earnings.
Moreover, individuals are affected differently by a natural experiment.
This means that natural experiments cannot be used to study experiments.
The opportunity to leave school at 16 will hardly affect those who had already planned to go to university.
The same reasoning.
See also: Reflection 1 - How important are Natural experiments.

Page 7

However, a researcher who analyses the data only knows who has participated, not why - there is no information about who participated solely because they were offered the opportunity, thanks to the natural experiment (or the randomised experiment), and who would have done so anyway.
This sentence is not clear. Why mention both: natural and randomised experiment?
How can a causal relationship between education and income be established?
This is complex question.
First the assumption is that there should be assumption.
Secondly what is exactly meant with education and income. What is the definition.
An even more 'realistic' question is: What are the benefits of education?
See also: Reflection 1 - How important are Natural experiments.

More specifically, they asked the following question: Under which conditions can we use a natural experiment to estimate the effects of a particular intervention, such as a computing course, when the effects vary between individuals and we do not have complete control of who participates? How can we estimate this effect and how should it be interpreted?
To answer this question (to estimate the effects) is difficult. What is meant with: 'effects' and why using the verb: 'to estimate'? Both should be clearly defined.
Angrist and Imbens showed that it is possible to estimate the effect of the programme by applying a two-step process.
Okay.
The first step investigates how the natural experiment affects the probability of programme participation.
More detail is required. Why using the word 'probability'?
The second step then considers this probability when evaluating the effect of the actual programme.
More detail is required. The difference between step 1 and step 2 is the word affects and effect See: Reflection 4 - Natural Experiment: Call centre training programme.
Given a few assumptions, which Imbens and Angrist formulated and discussed in detail, the researchers can thus estimate the impact of the programme, even when there is no information about who was actually affected by the natural experiment.
This seems IMO a very important conclusion about this paper, but there is a lack of detail.
One important conclusion is that it is only possible to estimate the effect among the people who changed their behaviour as a result of the natural experiment.
That is the same as saying: "it is only possible to estimate the effect among the people who are affected as a result of the natural experiment." Tricky!
But that can also be a valid result of an experiment.

Reflection 1 - How important are Natural experiments.

In all types of sciences, experiments are the most important tools to use in order to investigates the details about the evolution of almost any type of process. The best strategy is a sequence of individual experiments, when as part of each experiment only one parameter is changed. Experiments are used in physical sciences, medical sciences, social sciences and economics.

The purpose of any experiment is to investigate the details of a certain process. The first step is divide the process in smaller processes and describe the (causal) relations between these processes. For the follow-up experiments can be used.
In relation to this document two types of processes can be studied:
Processes where humans are nor active involved.
Processes which include human involvement. Examples of these processes are: Economical processes (production processes and financial processes), Social processes including education and medical processes.

It is my understanding that in any Natural experiment the strategy is to minimize active involvement of humans. That means, by preference, parameter selection should be random and not based on human preferences.
However certain questions remain.

Reflection 2 - Influence maximum speed limit of 100 km/h versus 120 km/h on safety in the Netherlands and Belgium.

The problem is that recently the speed limit in the Netherlands has been reduced from 120 km/hour to 100 km/hour and in Belgium stayed at 120 km/hour. The Netherlands and Belgium are two neighbouring and approximate 50% of its boarders each are connecting.
The question to ask is: can this be considered a Natural experiment.
In my opinion not. The influence of a speed limit is a world-wide problem and it is not only related to safety but involves many causes and has many effects.
Of course if the maximum speed is reduced in the Netherlands at day x, you can compare the situation before and after day x.
At the same time it is very important to what extend the speed limit is active controlled by the police within the borders of each country. When this is country dependent it can make a comparison less reliable.

Reflection 3 - Does it make sense to pay Russia in dollars or in rubles.

The background of this problem is more or less discussed in this article: https://www.cnbc.com/2022/06/23/russias-ruble-is-at-strongest-level-in-7-years-despite-sanctions.html "Russia's ruble hit its strongest level in 7 years despite massive sanctions. Here's why"

The reason of the question is that Russia wants to force certain countries to pay for gas in rubles.
The first issue is why buying gas from Russia? If there is no alternative source then you must buy from Russia, if you like it or not.
The second issue is if Russia asks you for a certain quantity of gas, delivered, to pay at a certain date, a certain amount in dollars than normal trade practices (?) enforces you to do that.
The same if Russia asks you to pay in rubles.
What this means if you have agreed for a certain price, at a certain date, for a certain currency it does not matter what the actual conversion rate is between the two currencies.
The third issue is, if what is written above is true what are the added benefits for Russia to ask for certain countries to pay in rubles and not in dollars.

This is a very complex problem. Does it make sense to use a Natural experiment to solve it?

Reflection 4 - Natural Experiment: Call centre training programme.

The purpose of the experiment to clarify the issues discussed at page 7
  1. In this example we study a natural experiment which involves a training course. The training course is part of a call centre. The experiment involves 20 pupils. 10 pupils will be trained how to answer questions. 10 pupils will not be trained. Each pupil has its own room. The pupils in an even room number will be trained for 2 hours. At the end of the day the pupils will be evaluated by means of feedback received from the users.
    The results are: Almost all the users of the pupils who received training are satisfied. Almost all the users of the pupils who did not receive training are not satisfied. This result is as expected.
  2. The second example is slightly different. Instead of training versus non training, we supply all the pupils with a training manual. All pupils are supposed to study this training manual, for 2 hours, from a screen. There is also a difference. 10 pupils will have a help function on their screen. This functions like a training assistance. 10 pupils will have not this a help function. Their only training is the training manual.
    The results are at the end of the day that the 10 pupils with the help function on average outperformed the 10 pupils without help. If completely satisfied indicates a 10 and completely dissatisfied a 0 than the results of the 10 pupils with help was between 6 and 9 and the pupils with no help between 4 and 7.
  3. The third example consists that all the 20 pupils will get training with a help function. The only difference with the previous is that all the pupils should receive a form with contains one question: which is your highest level of education. The results are at the end of the day that the pupils with the highest level of education on average will receive the highest level of satisfaction. There exists a clear linear correlation.
In de text we can read that there are two steps.
Both these lines are very difficult. How does effect A affects B and how does effect B affects A.

If you want to give a comment you can use the following form Comment form
Created: 1 August 2022

Go Back to Article Review
Back to my home page Index