Few could have predicted the huge surge in popularity of MOBA games on mobile platforms, culminating in Tencent’s Honor of Kings becoming the world’s top-grossing mobile game this summer (with an estimated monthly revenue of $150 million).
Where other genres have failed, MOBAs (Multiplayer Online Battle Arenas) are succeeding in attracting millions of depth-seeking, not-so-casual strategy gamers who have traditionally been neglected on iOS and Android.
But there’s a catch: the progress of mobile MOBA gaming is at risk of being held back due to the genre’s typically huge computational, connectivity and power requirements. Game studios have to be innovative in optimising for these requirements on mobile chipsets if they’re to achieve the same quality of experience and commercial longevity as renowned PC-grade MOBAs like League of Legends and DOTA 2.
In this article, we’ll use GameBench to objectively compare the optimisation of two popular mobile MOBAs: Vainglory by Super Evil Megacorp and Arena of Valor by Tencent. (Arena actually happens to be a Westernised spin-off from Honor of Kings.) As we’ll see, there are big differences between these two titles in how effective they are at striking the right balance of performance and power.
If you want to skip all the analysis and just see raw frame rate and power consumption figures, then you can head straight to Reference Data pages for these two products:
How to spot a well-optimised MOBA
A good MOBA will throw at least six players into a lush, atmospheric battle arena and then encourage them to use a mix of strategy, reflexes and cooperation to win. This formula immediately creates a conflict between three competing requirements, all of which can readily be measured and compared using GameBench:
- Fast and fluid animation is essential in order to bring battle arenas and heroes to life. 30 frames per second (fps) is widely accepted as the minimum for playable animation, while 60fps is generally the optimum on mobile. GameBench captures this in the form of our “Median FPS” metric.
- The higher the frame rate, the more susceptible it is to momentary bottlenecks. But competitive MOBA gamers don’t have much patience for even temporary slowdowns; they’ll simply abandon a game that stutters or lags during frenetic battles. A stable frame rate is therefore just as important as a high frame rate, and GameBench captures this stability through the “Minimum FPS” metric (which describes the worst frame rate during gameplay) and the “Variability Index” (which describes the average amount of variability between consecutive frame rate readings).
- Finally, we can’t forget power efficiency. Serious MOBA players might not always regard this as a top concern, especially if they play on a tablet or phablet with a big battery, but a good mobile MOBA should be playable on the go despite its need for constant connectivity. GameBench captures accurate power consumption data from the instantaneous current readings produced by modern iOS and Android chipsets.
A Super Evil disclaimer
Super Evil Megacorp is a GameBench client. The company has been using our tools internally for a while now, to help its engineers to achieve the best possible frame rates on the widest possible range of devices.
Recently, the company asked us if we could test Vainglory for ourselves and publish our results on the GameBench Reference Data page. We made it clear to Super Evil that, as always, our methodology and data are objective, unbendable and user-verifiable (i.e. anyone can check our results if they’d like, using GameBench or other profiling tools). Super Evil had no qualms about accepting this. So, let’s get to the benchmarks…
Frame rate comparison
We tested Vainglory and Arena of Valor on a sample of 12 widely-owned devices, representing iOS and Android flagships going back as far as 2013. The devices were all standard, unrooted/unjailbroken, shop-bought products, and they were all tested by real people playing real PvP matches. Here’s the full list:
- iPhone 5S
- iPhone 6
- iPhone 6S
- iPhone 7 Plus
- iPad Pro 9.7
- iPad Pro 10.5
- HTC One M8
- OnePlus 3T
- LG G5
- Galaxy S7 (US version)
- Galaxy S8 (US version)
- Galaxy S8+ (EU version)
First, we tested both games with their default settings and in their primary game modes, (3v3 for Vainglory, 5v5 for Arena of Valor). This quickly revealed a clear difference in frame rates, as shown in the chart below.
Vainglory is a truly smooth and responsive 60fps experience, achieving an average of 59fps (which is within measurement error of 60fps) across our sample of devices. This smoothness doesn’t come at the expense of visual detail, as the game produced highly detailed graphics on every device we tested.
Vainglory’s frame rate also proved to be very stable, with average Variability of just 1.7fps and an average Minimum Fps of 54fps, which is a small enough dip that most players probably wouldn’t notice. Indeed, very few reviews on the app stores complain about lag in Vainglory, and many specifically praise the graphics and controls.
(The exception to the rule was the old HTC One M8, which produced a median FPS of 53fps and minimum FPS of 33fps.)
By contrast, Arena of Valor runs at 30fps on default settings, meaning its side-scrolling and action were half as smooth and half as responsive as Vainglory’s. Reading through reviews of Arena of Valor on the app stores, it does seem that gamers notice the game’s frame rate as there are many mentions of lag. This may be contributing to Arena’s slightly lower average user rating.
On the plus side, however, Arena of Valor sticks very closely to 30fps, producing a stable experience. On average, the frame rate stayed above 28fps and the moment-to-moment variability of 1.3fps was slightly better than with Vainglory -- even on the tetchy HTC One M8.
Remember, for device-specific results, you can check out our Reference Data pages for Vainglory and Arena of Valor.
Max settings
GameBench’s methodology requires games to be tested on their default graphical settings. However, Arena of Valor offers the user some control over graphical settings, including a “High Frame Rate” option, so long as that user is happy to burrow into a settings menu and click past a warning message about their device overheating.
To see if we could create a more apples-to-apples comparison, we re-tested Arena of Valor on a reduced sample of four top-end devices that allowed all graphical options to be pushed to max. We wanted to reveal whether the game could compete with Vainglory after such tweaking, and to further level the playing field we also did the second round of tests using Arena of Valor’s smaller and more Vainglory-ish 3v3 Valley Skirmish maps.
As you can see, Arena still couldn’t keep up. On our reduced device sample, which included the Galaxy S8 (US version), the game only mustered an average of 56fps and a minimum of 45fps. That’s a long way from a consistent 60fps experience, so most Arena of Valor players are likely to be better off sticking with default settings.
Power efficiency
Given that Arena is half as performant as Vainglory for most gamers, we wanted to find out whether it also uses half the amount of energy when played at default settings. If true, this would potentially give Tencent's game an edge when it comes to playing on the move.
As it turned out, Arena’s power advantage wasn’t quite that big, but it was still significant. On average, the game’s current draw was 127mA less than Vainglory’s. In practice, the saving varied considerably depending on device, and it tended to be greater on iOS hardware. On a Galaxy S8+ or OnePlus 3T, Arena only provided a few minutes of extra gameplay time, but on an iPhone 7 Plus you could expect an extra hour and half.
Wrap-up
To keep things simple, GameBench’s Reference Data table uses colour-coded badges to quickly grade a gaming experience:
Our green “Smooth” badge is only awarded to games that achieve a playable 30fps. Arena of Valor wins this badge on most devices and therefore represents a safe bet for certain types of gamer -- particularly those who own an older or cheaper Android device. Arena also gets a fair number of green badges for battery drain, which makes it a smart choice for those who prioritise battery life over everything else.
The GameBench “Ultra” badge, meanwhile, deliberately sets a higher bar that rewards only rock-solid 60fps experiences. Vainglory stands out for the number of Ultra badges it wins across a wide range of devices -- including, remarkably, every iPhone back to the 5S. Super Evil has built a game that delivers 100 percent more performance for only 20 percent more power consumption, compared to Arena. For someone seeking a premium MOBA experience with minimal lag in order to play competitively, our performance stats show that Vainglory is the obvious choice.
Interested in finding which device is best for playing Vainglory? We've done a quick ranking that will reveal all.