How Much Should We Trust Observational Estimates? Accumulating Evidence Using Randomised Controlled Trials with Imperfect Compliance
Despite advances in our understanding of quasi-experimental methods, there will likely remain demand to evaluate programs using observational methods like regression and matching. To evaluate the observational bias in these methods we collected data from a large number of RCTs with imperfect compliance (ICRCTs) conducted over the last 20 years. We create comparable observational and experimental estimates of treatment effects, and use these to estimate bias in each study. We then use meta-analysis to quantify the average direction of bias and uncertainty about its size. We find little evidence of average bias but large uncertainty. We suggest adjusting standard confidence intervals to take this uncertainty into account. Our preferred estimates imply that a hypothetical infinite N observational study has an effective standard error of over 0.16 standard deviations and hence a minimal detectable effect of more than 0.3 standard deviations. We conclude that – given current evidence – observational studies cannot be used to provide information about the impact of many programs that in truth have important policy relevant effects, but that collecting data from more ICRCTs may help to reduce uncertainty and increase the effective power of observational program evaluation.

Written with David Rhys Bernard (Paris School of Economics), Sylvain Chabé-Ferret (Toulouse School of Economics), Jon de Quidt (Stockholm University), Jasmin Claire Fliegner (University of Manchester), Roland Rathelot (Institut Polytechnique de Paris (ENSAE)
Date: 11 October 2023, 12:30 (Wednesday, 1st week, Michaelmas 2023)
Venue: Manor Road Building, Manor Road OX1 3UQ
Venue Details: Lecture Theatre
Speaker: Gharad Bryan (London School of Economics)
Organising department: Department of Economics
Organisers: Alison Andrew (University of Oxford), Dennis Egger (University of Oxford), Niclas Moneke (University of Oxford)
Organiser contact email address: suzanne.george@economics.ox.ac.uk
Part of: CSAE Lunchtime Seminars
Booking required?: Not required
Audience: Public
Editors: Suzanne George, Fiona Morsia, Claire Goode