You`ve heard of benchmark cheating, but what about benchmark avoidance?

It's more than a year since bloggers at Anandtech produced a definitive list of Android benchmark cheaters -- manufacturers who tweaked their devices to produce artificially high scores in GFX Bench,  AnTuTu and other synthetic mobile benchmarks. (Recap: it was just a couple of little-known vendors like Samsung and HTC.)

The issue of cheating has lessened since then, but gamers are still getting gamed -- this time through a tactic that can best be described as "benchmark avoidance." Fortunately, GameBench has a solution to this issue, just as it did to the first cheating scandal, but first let's look at how the avoidance tactic works.

Benchmark avoidance is when hardware companies deliberately disable parts of the Android operating system in order to make a mobile device more inscrutable to the most revealing types of benchmarks -- particularly to real-world benchmarks like GameBench.

Read More

The market’s first ‘usability’ benchmark

When you’re on a long flight, have you stopped yourself from playing a mobile game in order to conserve your battery? Have you experienced too many stutters when you play a visually intensive title? Do you think it’d be helpful if you could choose a phone to suit your specific needs, based on public data about how well each model handles the best games and apps?

Smartphone and tablet reviewers try to distill their sense of how well a phone performs, but no matter how good their intentions, their conclusions are inevitably subjective and anecdotal. That’s why a market has developed around benchmarking apps like GFX, CPUBench, AnTuTu etc.

Read More

How to make an ‘uncheatable’ benchmark

Is there such thing as an ‘uncheatable’ benchmark? Cheating isn’t new in benchmarking, as seen with SPEC (for PCs) or Dhyrstone (for embedded processors) from a decade ago. More recently, benchmark wars have resurfaced, given the news around how certain smartphone manufacturers like Samsung and HTC have been rigging the results on particular top end models. It appears that they detect when a benchmark that is running (such as GFX, Basemark, or AnTuTu), and then increase the chipset frequency and temperature constraints to give higher results.

Read More