Mobile email testing rig
Since 09, the progression of tools I use to test mobile email looks like this: free online tool like iBBDemo > DeviceAnywhere > iPod touch > one device from each major platform > device lab.
DeviceAnywhere quickly became a lifesaver, but paying by the minute and clunky interactions drove me to buy an iPod touch. It was a relief to finally test the speed and ease with which users interact, ergonomics, reading postures, and usage patterns.
Then as so much of our work became mobile, adding more devices seemed like a legitimate investment. Not just in QA, but in our design skills as it allowed us to study each platform. Today we’ve 34 (I’ll keep updating) testing devices - mostly purchased new and unlocked from Amazon - ranging in price from $55 - $600. There’s more affordable ways, like buying from eBay or places like Gamestop which sell used devices.
I’ve had some emails asking about our mobile testing set up. So here’s what we currently test on, how I’ve rigged it up, why we chose these devices, and which I think are essential or an indulgence:
iOS
Most of our clients stats show high iOS numbers, so these were the first devices we purchased. An iPod touch is a great buy, it runs the same email client as the iPhone and costs $199 without the data plan hassles. If the iPhone 5 has a larger screen and there is an iPad mini we’ll have to add both (yep added these).
1. iPod Touch (taller 4″ screen) 640 x 1136 OS 7.1
2. iPod Touch (taller 4″ screen) 640 x 1136 OS 6.1.3
3. iPod Touch 3.5″ 640 x 960 OS 5.1.1
4. iPod Touch 3.5″ 640 x 960 4.3.5
5. iPad 2 1024 x 768 OS 4.3.5
6. iPad 3 (2x resolution) 2048 x 1536 OS 7.0
7. iPad Mini 1024 x 768 OS 8.0.2
8. MacBook Air for testing MacMail
Android
We’ve seen some quirks across different Android versions relating to responsive email design. CSS background images not supported, some issues around URLs coupled with progressive disclosure and other weird happenings.
If you buy one Android, run Gingerbread as it accounts for ~60% marketshare. You can pick up an older model like our Galaxy Ace new for $199, which came with GB installed rather than the advertised Froyo. Though ideally you want to own a few running different platform versions and screen sizes ect.
9. LG GT540 - 2.1 Eclair, 320 x 480
10. Motorola Defy - 2.2.2 Froyo, 480 x 854
11. Samsung Galaxy Ace GT S5830M - 2.3.6 Gingerbread, 320 x 480
12. Galaxy Note - 2.3.6 Gingerbread, 800 x 1280 “Smablet”
13. Samsung Galaxy Nexus - 4.0.4 Ice Cream Sandwich, 720 x 1280
14. S2 - 4.1.2 Jelly bean, 480 x 800
15. S4 4.3 Jelly bean, 1080 x 1920
16. Nexus 7 - 4.4.2 KitKat, 800 x 1280 7″ tablet
17. Android mini PC - runs 4.0. Hooks up to any screen like your TV, comes with Gmail app…nuts for $55.
Windows Phone
Originally I wanted the Lumia just to play around with metro. Then we realized that in order to take screenshots you have to become a developer (don’t ask total nightmare). So as we had it set up, Graeme started tinkering with Windows dev and bought a couple more phones. Hopefully their numbers will pick up with WP8 and Surface as its a lovely UI. Though not essential to the US market just yet.
18. Nokia Lumia 800 running WP7.5 “Mango” screen is 480 x 800px
19. LG-E900 WP7 screen is 480 x 800px
20. Dell WP7 screen is 480 x 800px (this one’s wacky has a slide out keyboard)
21. Surface tablet, screen is 1366 x 768
BlackBerry
I admit I was late getting into BB, though I’m really interested to see where they’re heading with BlackBerry 10 (update: just added slick device!). OS 6 is webkit and supports media queries, so its not as bad as you might think. Going by this chart ~53% of BB users are on 6 and above. Go to any fancy lunch spot in LA and it’s all iPhone’ with a decent smattering of BlackBerry’ and hardly an Android in sight. If you’re targeting certain demographics or enterprise level b2b, then there’s still a lot of BB usage out there.
22. Blackberry Bold 9900 - 640 x 480 screen running OS 7.1
23. Blackberry Bold 9780 - 480 x 360 screen running OS 6
24. Blackberry Curve 9320 - 480 x 360 screen running OS 7.1
25. Blackberry Tour 9630 - 480 x 360 screen running OS 5
26. Blackberry Z10 - 768 x 1280 screen running OS 10
Kindle
I love my original Kindle, for $139 you got free 3G. I was surprised when the e-reader was first released to see a number of comments in the forums about accessing email. You can and many people do in a pinch via the experimental browser. It’s definitely worth getting a 7″ tablet like the Nexus 7 or Fire which has been a big seller and costs $159-$199. We’ve also pre-ordered the Paperwhite and Fire HD which has a new email client.
27. Original Kindle 3G - 600 x 800 screen - can browse webmail via the internet - greyscale
28. Kindle Touch 3G - touch screen webmail - greyscale (bit unresponsive rarely use it)
29. Kindle Paperwhite 3G - experimental WebKit-based browser
30. Kindle Fire 7″ tablet - Silk (based on Android vr2), apps like Gmail 600 x 1024
31. Kindle Fire HD tablet 7″ - New version of Silk (Android). Default client, Gmail, Hotmail and Yahoo 1280 x 800
32. Kindle Fire HD tablet 8.9″ - New version of Silk (Android). Default client, Gmail, Hotmail and Yahoo 1280 x 1920px
Palm/WebOS
The Pixi is a great little device, it’s a shame about webOS as it’s a nice UI and renders email well. The Treo runs Windows Mobile 6 which is a precursor to WP7. It uses touchscreen and a little 5-way button to navigate. Actually between the BB qwerty keyboards, trackball navigation, slide out keyboard on the WP7 and the Galaxy Note stylus it’s a reminder that touchscreen isn’t the only input mode.
33. Palm Pixi Plus – 320 x400 screen - Palm web OS v1.4.5 (our cheapest at $85)
34. Palm Treo Pro – 320 x 320 screen – Windows mobile 6.1 (Precursor to WP7)
Set up & peripherals
Tired of devices taking over my desk and low batteries, I recently got organized. I’m calling it a “device lab” because it sounds good…but it’s just a table with a bunch of phones/tablets hooked up to USB hubs:
Sofa table, great find from Ikea. Long and narrow with 2 shelves, takes ~12 devices on each level. $59.
Tons of USB cables to keep the everything charged
Three 5-port USB hubs
16 of these stands
Blat, a command line email program we use to send tests
Extra flash memory for some older devices to store screenshots
For screenshots you’ll need to install apps, desktop software like visual studio and dev kits for some devices.
Acrobat - for organizing and sending those screenshots to clients as a PDF.
Install a range of clients on each device: native, webmail apps, Outlook exc. and popular apps like Sparrow.
Couple of data plans to test connection speeds. You can dodge for a while using wifi, except for BB.
Your call
Whether you go the real device route or stick with preview tools is your call. Some of our coders work remotely and use preview tools before sending tests to us, so a mixture works. Owning just 2-3 devices can make a difference; besides testing you accumulate everyday observations like how I don’t like to rest the iPad on my belly after dinner…little things.
I’ve definitely noticed we hammer the QA in a way we didn’t used to, sending multiple tests in quick succession without having to consider the cost or wait for previews. Debugging and mobile R&D is a lot easier and more important than ever with responsive design techniques to figure out.
I remember when Graeme used to do mobile game dev for feature phones back in 2005. The company he worked for had ~60 real devices, plus a database with hundreds of device specs listed. Compared to that level of fragmentation I have it relatively easy. As for emulators, Graeme’s used them for 10+yrs and he’s never found one that accurately mimics the device.
Sure we’ll have to add new devices occasionally, say five a year at ~$2,500. Though considering mobile email is our core business, it’s a reasonable annual investment given the learning’s and benefits. Ultimately we just feel we can get a better handle on the craziness that’s mobile email with the actual devices in our hands.