Google ends support for popular browser benchmark because devs were cheating!

Google is ending support for a widely used browser benchmark after determining that the test had actually become detrimental to web performance.

The benchmark, called Octane, has been widely used since 2012 by developers trying to assess how well a browser runs JavaScript, a core component of the web.

But as browsers got better and better at taking the test, Google saw something start to happen: “we began to notice that JavaScript optimizations which eked out higher Octane scores often had a detrimental effect on real-world scenarios.”

Part of that is simply that the test was designed for where the web was five years ago. Browsers are now efficient at what Octane was designed to test, and at the same time, websites have started being built in different ways.

Octane, Google writes, “didn’t capture important use cases for the modern web” and often doesn’t work “the way real-world code does.”

On top of that, developers were essentially cheating to get higher scores on the test, even though it resulted in worse real-world performance. In one case, Google says, developers began taking advantage of a bug that gave a 15 percent performance bump in Octane, even though it had zero effect on the actual web.

The test also penalized some optimizations that were good for real-world performance, encouraging developers to build a worse JavaScript engine.

Google warns to not rely on benchmarks:

Because of all that, Google is now ending support for Octane and generally telling developers to be careful when using benchmarks at all. “Unfortunately, similar issues exist in other static or synthetic benchmarks,” writes the team behind Chrome’s JavaScript engine.

The team is right that this isn’t a problem unique to Octane. While benchmarking tools are helpful, there’s always been a distinct cap on their usefulness, especially as they age.

In one example, in 2013, major Android phone manufacturers were caught rigging their phones to over-perform when running benchmarks, leading consumers to believe they may have been more capable in daily use than they actually were.

Google isn’t saying benchmarks should be put aside altogether, but its announcement this week is an important reminder that benchmarks can easily outlive their usefulness.(source-theverge)