Experimentation Platforms

Books

• Trustworthy Online Controlled Experiments

Resources

• Top Challenges from the first Practical Online Controlled Experiments Summit

• A/B Testing Pitfalls: Getting Numbers You Can Trust is Hard

• USF Business Analytics Forum - Ron Kohavi

• A/B Testing at Scale: Accelerating Software Innovation

• Trustworthy Online Controlled Experiments at Large Scale

• Always Valid Inference: Continuous Monitoring of A/B Tests

Companies

Netflix

• Netflix Articles tagged Experimentation
• It’s All A/Bout Testing: The Netflix Experimentation Platform
• Reimagining Experimentation Analysis at Netflix
• Success stories from a democratized experimentation platform
• Key Challenges with Quasi Experiments at Netflix
• Data Compression for Large-Scale Streaming Experimentation

• Page Simulation for Better Offline Metrics at Netflix

• Streaming Video Experimentation at Netflix: Visualizing Practical and Statistical Significance
• Innovating Faster on Personalization Algorithms at Netflix Using Interleaving

Microsoft

• ExP Experimentation Platform Accelerating software innovation through trustworthy experimentation
• Online Experimentation at Microsoft
• Experimentation Platform
• A/B Testing and Covid-19: Data-Driven Decisions in Times of Uncertainty
• Patterns of Trustworthy Experimentation: Pre-Experiment Stage

Twitter

• Twitter experimentation: technical overview

Google

• Overlapping Experiment Infrastructure: More, Better, Faster Experimentation

Facebook

• PlanOut is a library and interpreter for designing online experiments.
• Adaptive Experimentation Platform

Spotify

• Spotify’s New Experimentation Platform part 1
• Spotify’s New Experimentation Platform part 2
• Large Scale Experimentation at Spotify

Tinder

• Phoenix — Tinder’s Testing Platform, Part — I
• Phoenix — Tinder’s Testing Platform — Part II
• Phoenix — Tinder’s Testing Platform — Part III

LinkedIn

• Our evolution towards T-REX: The prehistory of experimentation infrastructure at LinkedIn
• Making the LinkedIn experimentation engine 20x faster

Uber

• Under the Hood of Uber’s Experimentation Platform
• A/B testing at Uber: How we built a BYOM (bring your own metrics) platform
• Building an Intelligent Experimentation Platform with Uber Engineering

AirBnB

• 4 Principles for Making Experimentation Count
• Scaling Airbnb’s Experimentation Platform
• Experiment Reporting Framework

Instagram

• Lessons Learned at Instagram Stories and Feed Machine Learning

Go-Jek

• Introducing Litmus: GOJEK’s Own Experimentation Platform

InstaCart

• Randomized, controlled experiments and multivariate regression are used to continuously improve the grocery delivery engine

Pintrest

• Building Pinterest’s A/B testing platform

Conferences

• Experimentation Culture Awards

SASS solutions

• Split.io
• Optimizely

When you can’t run A/B Tests

• Quasi Experimentation at Netflix
• Key Challenges with Quasi Experiments at Netflix
• Mostly Harmless Econometrics: An Empiricist’s Companion

Notes

2020 CODE@MIT Experimentation platforms

• Gaussian processes
• Multi touch attributions
• Heterogenous treatment effect
• Interaction effects
• Overlapping experiments
• What are potential over evaluation criteria ?
• What are good guardrail metrics ?
• run A/A tests.

Canary Deploys by using Experimentation platform to tell when you break guard rail metrics