A detailed guide to one of the most popular causal inference techniques in the industry — It is now widely accepted that the gold standard technique to compute the causal effect of a treatment (a drug, ad, product, …) on an outcome of interest (a disease, firm revenue, customer satisfaction, …) is AB testing, a.k.a. randomized experiments. We randomly split a set of subjects (patients, users…

Introduction: The problem with tutorials This huge amount of data science tutorials on the internet has its pros and cons. The biggest problem, in my humble opinion, is that everyone wants to talk about data nowadays in order to get people — specially recruiters — attention. That being said, several mistakes are being made and…

How statistical decision making has evolved, yet most companies haven’t kept up — Over the past two decades, online, controlled experiments have become an increasingly core part of the way that many of the most successful tech companies develop products. Companies like Airbnb, Booking.com, Microsoft, Google, Netflix, Doordash, and Stitch Fix run tens of thousands of experiments per year, and as a result…

An in-depth guide to the state-of-the-art variance reduction technique for A/B tests — During my Ph.D., I spent a lot of time learning and applying causal inference methods to experimental and observational data. However, I was completely clueless when I first heard of CUPED (Controlled-Experiment using Pre-Experiment Data), a technique to increase the power of randomized controlled trials in A/B tests.

Co-authored with Shubha Shedthikere and Jairaj Sathyanarayana Introduction In most marketplaces, there is usually a need for ranking models to incorporate the multi-dimensional nature of relevance as well as business constraints. Traditional ML methods cannot easily handle such multiple objectives in a single model. …

Explaining how L1 and L2 work using gradient descent — (Jump right here to skip the introductions.) Changelog: 27 Mar 2020: Added absolute to the terms in 2-norm and p-norm. Thanks to Ricardo N Santos for pointing this out. Overfitting is a phenomenon that occurs when a machine learning or statistics model is tailored to a particular dataset and is…

We are part of the Facebook Feed Analytics team and we wanted to share methodologies that we’ve developed at Meta around how we create long-term value for people that use our products and technologies. The crux of the data science and data engineering work we do at Meta Analytics in…

Article written with @eduardotoce, Data Scientist at Glovo. Abstract We discuss the use of cluster-randomized experiments at Glovo, an experimental design useful for evaluating impact when randomization at the individual level is infeasible or when there are concerns about interference under individual-level randomization. We also discuss the analysis of these…

Martin Tingley with Wenjing Zheng, Simon Ejdemyr, Stephanie Lane, Colin McFarland, Andy Rhines, Sophia Liu, Mihir Tendulkar, Kevin Mercurio, Veronica Hannan, Ting-Po Lee Earlier posts in this series covered the basics of A/B tests (Part 1 and Part 2 ), core statistical concepts (Part 3 and Part 4), and how…

How offline policy evaluation works, examples on how to use it, and lessons learned from building OPE at Facebook — TL;DR you’re probably wasting time, resources, and revenue running unnecessary A/B tests. Offline policy evaluation can predict how changes to your production systems will affect metrics and help you A/B test only the most promising changes. A/B tests are essential for measuring the impact of a change or new feature…

Measuring the impact our models generate in the business can be as complex as training and deploying them — User loyalty is an issue that concerns all e-commerce companies; just a tiny fraction of the best users generate most of the revenue. This is why taking care of these users is so important to the business; thus, the need to measure the impact of the efforts made to retain…

Martin Tingley with Wenjing Zheng, Simon Ejdemyr, Stephanie Lane, and Colin McFarland This is the third post in a multi-part series on how Netflix uses A/B tests to inform decisions and continuously innovate on our products. Need to catch up? Have a look at Part 1 (Decision Making at Netflix)…