GameBench launches first touch latency benchmark for mobile gaming: iPhone XS Max vs. Galaxy Note 10 input lag

In any competitive multiplayer game, by definition, you're going to face enemies. These include rival players, pesky bots and often the toughest foe of all, the one that you can't kill: lag. Lag has multiple causes, including poor frame rates, which we've been benchmarking on mobile for many years. But today we're launching an entirely new benchmark for a different source of lag, called touch latency (aka input lag).

The longer the time gap between touching the screen and seeing the result, the harder it is to aim, shoot or do whatever else it takes to win -- so this is fundamental to mobile gaming as a whole. To our knowledge, GameBench is the first to produce a platform-agnostic benchmark for real-world gaming touch latency on mobile, so that gamers and developers can compare different devices and games on the basis of their responsiveness. Please read on to see our first batch of results and discover more about how we measured them.

For the following benchmarks, we tested touch latency on four of the most popular mobile games, PUBG Mobile, Game for Peace, (the Chinese version of PUBG, which allows higher-grade quality settings), QQ Speed and Arena of Valor. We tested these games across two top rival phones: the iPhone XS Max and the Galaxy Note 10 (UK Edition). 

Why is touch latency important?

If you're an experienced gamer or mobile app developer, you'll know this already. If not, check out this YouTube video from Microsoft Research and you'll get the picture. The video is from 2012, when technical expectations in the mobile industry were perhaps a bit lower than they are today. But when the narrator says that 50ms of lag is "better than most devices on the market," that's still very much in 2019 -- as you'll see in our results in a moment.  

Our methodology in a nutshell

For those who are curious, we've got some additional notes about our methodology at the end of this article. In a nutshell, however, we've built a precisely synchronised, high-speed camera system that captures the moment a screen is tapped and the moment the onscreen animation responds, with millisecond accuracy. We repeat each tap-test three times, and take the average time. We then use this time to work out how many whole frames of lag there were between a typical tap and the resulting animation. The more frames of lag, the less responsive a game will feel.

What phones and games have we tested? 

For the following benchmarks, we tested touch latency on four of the most popular mobile games, PUBG Mobile, Game for Peace, (the Chinese version of PUBG, which allows higher-grade quality settings), QQ Speed and Arena of Valor. We tested these games across two top rival phones: the iPhone XS Max and the Galaxy Note 10 (UK Edition). 

We've chosen games and in-game settings that produced a median frame rate of 60fps across all test games and devices. Matching the target frame rate in this way is important for a more apples-to-apples comparison, because lower frame rates like 30fps will often produce worse lag. This is something we will compare and illustrate in a future blog. We'll also add more devices to our comparisons in the coming weeks, including mid-tier devices, so that readers can see whether spending more on a phone makes any difference to touch latency. Please sign up for updates if you'd like to see these new results as soon as they're ready! 

How much touch latency is 'bad'?

For a long time now, GameBench has rated gameplay frame rates as Ultra (blue), Smooth (Green), Basic (Yellow) or Poor (Red). These same badges can help to bring some simple colour-coding to our touch latency results. Our testers, who are all gamers first and foremost, have spent the last couple of months playing mobile games with different latencies in order to set the boundaries for each badge, producing the table below. We can't claim it's the final word in touch latency ratings (that makes no sense in a fast-moving industry like ours), and ultimately any fixed threshold is quite arbitrary, but we think this is a solid start: 

Most of the badges speak for themselves. We think latency becomes Poor once you exceed 167ms (10 frames of latency on a 60Hz display) and this tallies pretty well with PC and console experiences. For example, the game Killzone 2  (2009) on the PS3, which was widely criticised for its input lag, had a latency of around 150ms for the controller, plus 30-50ms for a typical TV display from that epoch. (Note, any display latency is included in our thresholds, because most mobile gaming is done on the integrated display, as were our tests). It was only when Killzone's developers brought overall latency down to less than 167ms (including display lag) in Killzone 3 (2010) that the complaints subsided.

The Basic badge is a bit more nuanced, because its significance depends on the game genre. Legend of Zelda: Breath of the Wild has touch latency of around 150ms, which is not good, but it's at least acceptable for this type of open-world exploration game where reflexes and timing aren't so mission critical. The same latency on a first-person shooter would be a more serious matter. So ultimately it's up to the gamer to decide whether a Basic badge will be sufficient for their needs. 

The Smooth badge represents solid touch responsiveness that should suffice in all mainstream scenarios, including reflex-dependent games. Perhaps the only scenario where Smooth isn't good enough is in highly competitive or eSports-grade gaming, where every extra frame of lag carries a huge penalty for competitiveness.

This is why we also have an Ultra badge, which rewards what we regard as competition-grade latency. On current-gen console games like Call of Duty: Modern Warfare 2 and Battlefield 4, latency has been pushed down as far as 40-56ms according to Eurogamer. Even factoring an extra 10-15ms of display lag for a modern gaming TV, you're still looking at only 3 or 4 frames of lag overall, which helps to set our target for Ultra. Apple believes there's a theoretical limit of two frames of lag on its mobile devices, which further lends some support to our Ultra threshold has being ambitious but realistic.

Overall Results 

The first thing to say is that our results show that these top-end iOS and Android phones are very evenly matched for touch latency -- there's really nothing between them. This isn't very exciting in terms of shouty headlines declaring a winner (darn it). On the other hand, the second thing revealed by our data is that there's clearly room for improvement on both platforms. Although 86-88ms represents good latency, worthy of our Smooth badge, it might not suffice for highly competitive gamers who want lag to be consistently lower than 5 frames in order to match what they'd get on PC or console games:  

Arena of Valor Results

This game gives a slight time advantage to the iPhone, but in terms of whole frames of lag there's no difference. Both devices get our Smooth rating for Arena of Valor. There's one caveat though: we did get some freakishly high results of around 200ms while testing this game, and we suspect this could be due to frame drops -- something we know to be an issue, especially on the Galaxy Note 10. In other words, if there's a performance bottleneck that prevents the rendering of some frames, then this will greatly increase our latency results. The only other possibility is that some taps in Arena of Valor incur a network-related time cost, rather than taking place purely within the local UI -- this is something we can't rule out and want to investigate further. In any case, we've excluded the outlier results from our averages below: 

PUBG Mobile Results 

Again, no significant difference between the two phones playing PUBG Mobile. Both produced Ultra-grade touch responsiveness on average, although there were quite a few 5-frame lags in our results (which the averaging has removed): 

Game for Peace Results

The Chinese version of PUBG, known as Game for Peace, is interesting because it allows you to unlock much higher graphics settings than the Western version on certain devices. It won't be a common scenario for the owner of the UK-version of the Note 10 to play the Chinese version of a game, but we really wanted to see if playing at a higher quality setting (HDR) with maximum frame rate (60fps) and with anti-aliasing turned on would increase lag. By the looks of it, both devices do suffer slightly on the heavier version of the game, taking 12-15ms longer to process touches on average, resulting in Smooth ratings rather than the Ultra badges we got on PUBG: 

QQ Speed Results

Our final game, the most popular racing game in China, is the only one to show a significant difference between the two test devices: Note 10 comes out on top by a significant margin, achieving an Ultra rating while the iPhone XS Max only gets Smooth. Our testers also reported feeling a difference in this game, so it's possible that this game has an optimisation issue on iOS:  

Methodology: Additional notes for the very curious 

Our philosophy at GameBench is to measure real-world gaming experiences. We're not in the business of creating artificial or synthetic benchmarks. For this reason, we've designed our touch latency rig to work while a player is in the middle of a real gameplay session.  

A potential criticism of our methodology, however, is that it measures lag as a whole, rather than dissecting it into specific components. What we're describing as "touch latency" can actually stem from multiple sources, including:

• Display lag, which tends to separated out in PC and Console tests because the display is a separate part of the system. Our results bundle display lag into the whole, because most gamers will play on the integrated mobile display; 
  
  
• Missed Vsyncs, aka "drop frames," which is when frames fail to render on time. Dropped frames can greatly increase total perceived lag, and although we've matched median frame rates across our test sample, we cannot guarantee that all games ran at a perfectly stable 60fps. On the contrary, we're pretty sure that dropped frames have increased some of our lag measurements, even after we excluded and replaced some outlier results. Where unexpected frame rate drops are common and serious enough to consistently increase latency, however, we think this should be captured in our results rather than glossed over. 
  
  
• Vsync position, which introduces a small measurement error. This measurement error derives from the fact that mobile displays don't update all in one go. Instead, a bar swipes across the screen from top-to-bottom (or left to right if in landscape mode), rather like a windscreen wiper, gradually deleting the old frame and bringing in the new one over a period of 16.67ms. For example, if the most recent frame that coincides with the user's touch (which we call the "frame zero") is very young, such that only a small part of it is shown onscreen, then the device has the luxury of slightly more time to process the touch before the next frame ("frame 1") comes into view. Alternatively, depending on the Vsync position, a device might have to wait slightly longer to display an animation that it had rendered nice and early, thereby making lag seem slightly longer. We've tackled this in the simplest possible way: by taking multiple measurements and averaging them out.
  
  
• Top / Bottom bias, which can also introduce a measurement error. This also relates to the top-to-bottom movement of the Vsync, which can cause animations on the top part of the screen to be displayed before animations that take place on the bottom. To remove this bias, we measured the time of the animation frame as the moment when this frame reached the centre of the display, rather than the moment when the animation became visible to the user. 
  
  
• Network lag, which could potentially increase the time gap between a touch and a response, if the response first relies on communication with the game server. We haven't tried to factor this out, but we have played all the games on the same enterprise-grade WiFi network. Again, if network lag increases touch lag, we think this is interesting to the gamer and not something that should be overlooked. 
  
  
• Different kinds of touch gestures, which have different degrees of lag. Our initial test data shows that "drag" touches take longer to be processed than simpler "tap" gestures. Over time, we hope to build up our data about multiple gestures, rather than solely tap. After all, there's no point being able to shoot quickly if you can't aim quickly! But considering this is a new benchmark, starting with a single, simple gesture like tap makes sense. 
  
  
• Semantics, as different people describe latency phenomena in different ways. Since there's no common standard, we've just used our own way to describe what we've measured. This includes discounting the animation frame from our assessment of the number of frames of lag -- i.e. it's only those frames between the touch frame and the animation frame that we consider to be frames of lag, even though we know that some engineers choose to include it the animation frame in their number.  

What next? 

We will publish additional touch latency data very soon, to show what happens when you buy a cheaper phone (with a mid-tier chipset), or a gaming phone (such as the Black Shark 2), and also show how rival games in the same genre after different degrees of lag. If you'd like to be the first to see these benchmarks when they're published, please sign up below!