- GameBench Staff
- 04. December 2014
It's more than a year since bloggers at Anandtech produced a definitive list of Android benchmark cheaters -- manufacturers who tweaked their devices to produce artificially high scores in GFX Bench, AnTuTu and other synthetic mobile benchmarks. (Recap: it was just a couple of little-known vendors like Samsung and HTC.)
The issue of cheating has lessened since then, but gamers are still getting gamed -- this time through a tactic that can best be described as "benchmark avoidance." Fortunately, GameBench has a solution to this issue, just as it did to the first cheating scandal, but first let's look at how the avoidance tactic works.
Benchmark avoidance is when hardware companies deliberately disable parts of the Android operating system in order to make a mobile device more inscrutable to the most revealing types of benchmarks -- particularly to real-world benchmarks like GameBench.
Take frame rates, for example: HTC has now made it very difficult to monitor the frame rate of a real, commercially available game while it is being played on an HTC-branded device. This used to be possible on both the One M7 and the One M8, until one day it was not.
At some point in the recent past, HTC suddenly decided to stop supporting an important bit of Google code called "Systrace" -- something that was specifically built by Google in order to allow greater transparency.
Why did HTC do this? Did it have anything to do with GameBench-based evidence about the underperformance of HTC phones? We cannot really say. But the direct impact was that game developers, product reviewers and phone buyers could not readily use GameBench to judge how well a particular game or app runs on an HTC device.
You could argue that HTC is under no obligation to open up its devices to public inspection. But then again, just imagine what would happen if a PC hardware company did this. If AMD or NVIDIA suddenly decided to prevent frame rate tools (such as FRAPs) from working on their graphics cards, there would be uproar. The move would rightly be deemed as going against the interests of consumers and the wider gaming ecosystem.
But mobile gaming is a less mature market, with a lack of consumer awareness about what benchmarks really mean and why the testing of real-world apps and games might be important.
That's why MediaTek has also managed to get away with removing access to Systrace on all phones and tablets carrying its chipsets. In this instance, MediaTek has told GameBench that it only removes Systrace in order to minimize the size of Android builds (because Systrace adds a bit of extra space -- a couple of MB). The chip maker has also promised us that it will add Systrace back on future chipsets, which should be taken as a sign of the company's good intentions.
It's important to say that GameBench has never been totally dependent on Systrace. It has always been possible to root a device in order to re-instate the necessary code. That's how we tested the MediaTek-powered Idol X+, which actually turned out to be a very strong gaming phone. But rooting is inconvenient, so root-only tools will only ever be of limited use in informing the Android ecosystem.
And that's why this last bit of good news is -- we hope -- pretty important: As of now, GameBench has been upgraded to Version 2.0. This major update finally removes any dependence on Systrace, even in unrooted devices, and regardless of whether hardware companies like it or not. Best of all, GameBench is still free and available directly from the Play store right now.