A user test should always be as realistic as possible. The obvious solution to test mobile interfaces is to use a mobile device. But this comes with a few things to consider.

What makes testing on mobile realistic

There are 2 main things, why we would like to test a mobile interface on a mobile device:

  1. Touch input, especially for scrolling is realistic
  2. Keyboard input is realistic

What makes testing on mobile unrealistic

There are many downsides when testing mobile interfaces on mobile devices, which are:

  1. Limited screen space on mobile devices results in an unrealistic test setup. The Solid interface (to show instructions and controls) takes screen of real estate that would be available in reality.
  2. Browser-Tabs are well understood on desktop devices, but not as commonly used on mobile devices. When you can’t open the website in the same window, this can lead to confusion for less tech-savvy testers.
  3. Think-aloud reminders are only available for links in the same window on mobile.
  4. Screen recording can not be started from the browser on mobile. Anna helps start the native screen recorder, which is understood by most testers. Solid can not detect, when the native screen recorder was started, so you have to sync the audio/camera recording with the screen recording afterward.
  5. Only iOS devices can be used, due to technical limitations on Android.

Result

On mobile devices, you can expect a 10-30% lower completion rate, compared to desktop test setups. The results vary, depending on your tester's tech-savviness.

Successful mobile tests

We recommend testing mobile devices whenever possible on desktop devices. The downside of not having touch input is usually overestimated. Testers can very well handle that setup. The upsides are:

mobileTesting.png