I keep coming back to this story because it annoys me in a productive way every time I read it. Really a company that is making billions in revenue have no staff or time to Test It?
In 2012 someone on the Bing team at Microsoft had this idea about ad headlines. Combine the second line of ad text with the title to make a longer headline. Small change. Would take maybe three days of engineering time to build and test. The program managers looked at it, looked at the other two hundred ideas in the queue, and put it in the low priority pile.
It stayed there for six months. None bothered to touch it.
Then one engineer saw it in the backlog, figured the code was maybe two or three days of work, and ran an A/B test without going through any approval chain. Just wrote it and shipped it as an experiment because he had the afternoon free and the task looked small enough.
Hours later the new headline format was pulling revenue numbers that triggered the system’s automated “too good to be true” alert. Everyone on the team assumed something had broken, because a headline change moving revenue that hard did not make sense to anyone who had looked at the idea and ranked it low priority six months earlier.
It was not a bug.
12% revenue increase. Over $100 million per year in the US alone. The most profitable idea in Bing’s entire history.
Ronny Kohavi, who was VP of Analysis and Experimentation at Microsoft and wrote about this in the Harvard Business Review back in 2017, said the experience rewired how Microsoft approached testing after that. The idea had been in the backlog the whole time. Six months of nothing because the people ranking priorities could not see what was inside it just by reading the description on a ticket.
85% of Experiments at Bing Fail. At Netflix and Google It Is 90%.
Kohavi published these numbers across multiple talks and papers and they are worth sitting with for a second.
- Bing: ~85% of experiments show no positive result
- Google Ads: ~90% fail
- Netflix: ~90% fail
- Microsoft overall: ~70% fail
Out of every batch of ideas that gets tested, only about a third turn out positive. Another third are flat, zero measurable difference either way. And the last third actually make things worse, meaning if they had shipped without testing they would have cost the company money.
I mean think about what that says about companies that skip testing altogether. Microsoft with all their infrastructure and experimentation culture still gets it wrong two thirds of the time. What is going on at places where product calls are made by whoever talks loudest in the standup?
Google Tested 41 Shades of Blue and Found $200 Million
This one is older but I still bring it up because people laugh when they first hear it and then stop laughing when they hear the number.
Google’s design team could not agree on which shade of blue to use for their search result links. Instead of debating it they tested 41 different shades against each other. The winning shade, a slightly purpler blue, outperformed the others enough to generate an estimated $200 million in additional annual ad revenue.
Forty-one shades of blue. Not a redesign. Not a new feature. Not a product pivot. A colour.
Kohavi also found separately that improving Bing’s page load time by just 0.1 seconds was worth $18 million per year. A tenth of a second. Eighteen million dollars.
These numbers keep saying the same thing over and over. Small tested changes find money that big untested changes miss entirely. And most companies never find their own version of the $100 million headline because the team building the product is too occupied shipping features and closing bug tickets to run the experiment that would surface it.
The Capacity Question Is Where Most of This Falls Apart
The Bing engineer who ran the headline test was not assigned to do it. He had his regular work. He spotted the idea in the backlog, noticed it looked quick, and squeezed it into a gap between his actual tasks. If his week had been busier he would have scrolled past it and the $100 million stays buried.
There is a saying in Latin America, “nadie sabe lo que tiene hasta que lo pierde.” Nobody knows what they have until it is gone. Bing was sitting on the most profitable idea they ever had and the backlog almost swallowed it because none of the engineers with the skills to test it had the calendar space to pick it up.
Most product teams are already at capacity. Roadmap commitments, customer escalations, security patches, quarterly goals. Asking those same engineers to also run speculative experiments on ideas from the low priority pile means the experiments will always lose to the work that has a deadline attached to it.
I was trying to find actual examples of companies splitting experiment capacity from delivery capacity and went through LinkedIn, I also checked some Hacker News threads, and a Lenny’s Podcast episode where Kohavi was the guest (worth watching if you are interested in this topic in depth). And as I had some connection with the N-iX team who handle staff augmentation in Latin America from a previous project, I asked them directly how their clients handle the experiment backlog problem specifically. Their take was that the companies getting real value from temporary engineers are not using them for regular delivery. They are using them for the testing pile, the ideas the internal team knows are worth checking but keeps bumping to next sprint because the committed work never lets up. Same timezone overlap with North American teams, fixed engagement window, and once the experiment concludes the capacity concludes with it. The core team never lost a sprint to it and the company either validated or killed an idea that would have sat there growing dust otherwise.
That reframing stuck with me. It is not about getting more hands to ship features faster. It is about having separate hands whose job is specifically to test the things the feature team cannot get to.
The Bing headline sat for six months because none of the engineers had a free afternoon to test it. $100 million of annual revenue was locked inside a backlog ticket that kept getting bumped. How many of those are sitting in Jira boards right now at companies that will never run the test because the sprint is already full and the next one is already planned.
