Managing a device lab
What are the benefits that come from having a physical lab to test mobile emails, as opposed to using emulators such as BrowserStack?
Let's see, I'll try and list it:
- Ergonomics: if designing for touch it helps to physically interact with the device.
- Performance: can test download speeds.
- Input modes and form factor.
- Use cases & context: get a more nuanced feel, when do I reach for a 7" tablet vs. iPad.
- Cost: cheaper to buy a device if you test on it frequently ( DeviceAnywhere was costly ).
- Testing and debugging is fast and accurate. Some stuff is hard to debug via a preview tool.
- There's more interaction & animation in email nowadays with CSS, SVG and HTML5.
- Can install a range of email apps like Mailbox, Gmail, Yahoo, Outlook.com, Evomail.
- Landing page testing: we need to test on mobile web browsers as well as email clients.
- Choice: email preview tools tend to lag a little behind, and some devices they never get.
- Client confidence: clients like it because of all of the above.
- Designer confidence: see our work running on real devices; makes us sleep better.
I noticed you had some older models in your rig, why keep older phones, and how do you know when to phase them out?
Yes we have some older models, some we bought just because they were cheap and wanted to play with. Even though their marketshare has reduced, a few like Android 2.3 and BB 7.1 are still in play. Android 2.2 and BB 5 have dropped to such tiny numbers, we've mostly phased them out. Though it may be that some of the older models have greater mileage outside the US.
It's useful having old and new Android devices running the same OS, as we saw inconsistencies when testing HTML5 video ( and recently SVGs ) on 2.3. I include the older devices if I'm looking at support, there may only be a few people on Android 2.2. but I don't want them getting a blank screen.
It can't hurt to test on older devices, though we won't jump through hoops if something isn't 100% on an old WP. Mostly you'll want to purchase newer devices, but it just depends on the platform. Android users are slower to upgrade than iOS.
It's been my experience that building a device lab isn't all about what makes the most business sense. Sometimes you buy devices just because the platform interests you – WP – or you want one for your personal use – Kindle – or it was cheap – Palm.
I'm a freelancer, which devices would you prioritize and is it worth only getting a few?
Before spending money, it might be worth seeing if there are any Openlabs in your area. Also try pooling together with other local freelancers with what's in your pockets. If you're in the UK, Labcase drop off 10 devices to your office for 50 pounds a week, nice idea.
I started out with just one iPod and one Android, they made a big difference as before that I'd been using DeviceAnywhere which was a bit clunky. The iPod touch has an identical email client to the iPhone — though cheaper and without the data plan — and makes up the majority of mobile email opens. From $229 new, but you can pick up iOS devices secondhand from eBay and Gamestop.
I currently start most QA sessions with: iPod touch running the latest OS 7.1, iPad retina OS 7, Android Galaxy Note running Gingerbread 2.3, Android Samsung Galaxy Nexus running Ice Creme Sandwich 4.0, Android S2 running Jellybean 4.1 and an S4 running Jellybean 4.3. After that I move onto second tier testing like the Nexus 7 ( 4.4 ), Kindle Fire and BBZ10, useful but not as critical.
Samsung S2 – S5 are important test devices but quite pricey, you can get an older model like an S2 and upgrade the OS. A Nexus 7 tablet is cheaper than a phone at $199, it also allows you to test on a different form factor. I also really like my Samsung Galaxy Ace ( see it in a lot of mobile test labs ), I have it running GB and its dropped to $119 on Amazon. Android usage is mainly spread over three versions: Gingerbread, Ice Creme Sandwich and Jellybean which makes it a pain for developers.
If you continuously power up mobile devices the batteries will degrade, are you manually monitoring this or something smarter?
No nothing smart. We use the lab so often, half the devices are spread across desks unplugged a lot of the time. Though that tends to be the devices we test on most frequently. Some lesser used devices are kept plugged in for longer periods. Often not charging can be more of a problem, batteries can die if left to drain completely. We've had more dead devices this way than with over-charging.
Etsy uses, 'A few USB hubs and a power strip on a timer', I'm going to look into this myself:
We’re measuring the amount of power we use, and we've also set the devices up on a power strip with a timer so they only charge for a few hours each day. We’re still finding the right balance – Etsy. This whole post is worth a read, also see their Flickr device lab photos.
On Smashing magazine it mentions:
Jeremy Keith also told me that they have all the devices running through a wall socket that’s on a timer which switches the power off in the evening and nighttime and back on again in the morning. That might be useful for saving some energy and also to keep the batteries healthy.
Later on, when there are many more devices in the lab, you may want to start considering getting a better wireless router which can handle all the devices. Andre Jay Meissner told me that Apple’s Airport Extreme can handle up to around 30 devices, but not much more ( it claims to support 50! ).
We have four Wireless-N access points around the office, all connected to a gigabit LAN with devices distributed evenly to ensure maximum bandwidth. This should give us availability for over 120 devices, but with 30+ devices synchronizing with mail servers there can still be bottlenecks. So regardless of manufacturer claims I would try to keep the device to access point ratio as low as possible and minimize the number of devices on the network at any given time. It helps to completely power down devices that are not in use, this also helps battery life.
How do you manage and share devices, is it portable or always in one place?
Our device lab is stationary for the most part, kept in one room. It's a simple set up, we've two of these IKEA benches, 34 devices, 3 USB hubs, 16 stands, comfy chair and a computer so you can test desktop clients and access your code.
Though you can also just grab a few devices and take them to your desk. As a small team we've found we don't need a check out system, devices change hands too quickly. With a larger team, an online booking system or even Esty's library card approach might work.
One issue is that you'd need multiples with large teams, you can't have just one person doing QA at a time. iOS are most in demand, so we have seven. We've labelled everything with this handy printer so you know OS versions and screenshot shortcuts.
I read somewhere about putting devices in a cart, and wheeling it around to different departments. Also ‘developer stations' dotted around an office, with a handful of devices allocated to each station. Most labs I've seen are either a device wall in an open office or a room with some check out system in place so you know who's got what. Offices in different states could be difficult, you'd need to invest in more than one lab or have a 'QA department' that routes everyone's tests.
Some of our team are remote, they test as much as they can with what's in their pocket and preview tools like Litmus. Then they send us the code for a more thorough QA, and we report back any issues or sometimes make the fixes on our end. So we are the QA hub for remote folks and take turns on QA duty.
Reading your mobile testing rig post, are those stands any good?
They're OK, sturdy enough to hold heavier devices like the iPad and Surface tablet. For the odd device it can get a bit awkward with the cable, though I just turn them sideways. It might be worth testing a few different stands before you commit in bulk. Price wise these were $9 each from Amazon.
ETSY use a variety of stands, the one's on the left are $6.90 each and on the right $15.
Clearleft purchased their stands from The Iron Mill, at 14.99 pounds each (free UK shipping). They have have an openlab in Brighton if you are local.
For a smaller desktop set up I've seen these, though a bit more pricey at $149.00.
A lot of folks come up with custom shelving, Viljami Salminen built this device stand for the Helsinki Open Device Lab. Which was inspired by 64digital's ‘super dooper' solution.
Etsy's device lab, 'Designed to have different-height openings for our range of device sizes'.
Lego and @sugru powered device wall by the Open Device Lab in Hamburg @odl_hh
Finally, 'The Batstand'. One thing I like about these shelving units, is the wires are out of the way.
Do you test on more than one version of iOS?
iOS users upgrade fast, as of April 87% of devices run on 7.0 ( compared to 5.3% on Android' latest KitKat ). Audiobooks puts it at 87% iPhone, 78% iPad and 79% all platforms. So I think you can get away with testing just on the latest version. To be honest it's rare that I see major rendering issues across iOS. I noticed when doing SVG testing that the blur filter wasn't supported below 5.1.1, but that's about all that's come up recently.
We have seven iOS devices, the iPod Touch runs the same email client as the iPhone:
- iPod Touch 5 (taller screen) running 7.1
- iPod Touch 5 (taller screen) running 6.1.3
- iPod Touch (original screen) running 5.1.1
- iPod Touch (original screen) running 4.3.5
- iPad Retina running 7.0
- iPad non-retina running 4.3.5
- iPad Mini running 6.1.2
According to this the split between iPhone screen sizes is about 50 / 50. If you want to factor in how much of your email creative will be visible on the first screen, it's something to make note off. Also iOS7 added a margin and some layouts show up as off center ( fix in this source to ditch the margin ). Though apps like Gmail have more usage on Android, I've installed a number on iOS though many are edge cases:
- Gmail
- Mailbox
- Evomail
- Sparrow
- Boxer
- Birdseye
- Molto (renamed from Incredimail)
- Cannonball
Which versions are most installed e.g. Blackberry, Android, iOS and Windows phone.
Android: Sure here's the page I check often for Android version stats. It's currently Jelly Bean, then Gingerbread followed by Ice Cream Sandwich. There's three versions of Jelly Bean, with render quirks on Samsung 4.3 ( and KitKat 4.4 from what I've heard ):
iOS: as of April 2014, 87% of devices are using iOS 7. Also see this post for iPhone vs. iPad.
WP: Windows Phone 8 accounts for 80.9% of all Windows Phone handsets in use, with 47% of WP8 devices on update 3.
BB: I can't find stats that show the split between BB7 and BB10 users, though this page shows that BB10.1 is in the majority.
Do you use QA docs?
Yes, it lists all the different email client combos I just mark it, 'good' or 'bad'. For a full QA it's around 70 clients, for a ‘stop check' 20. I try not to annoy coders with half-assed bug reports. I'll add notes, screenshots and photos which can give a greater sense of scale e.g. button size in relation to thumbs.
You can see an internal QA doc here, ( we've since moved to Google docs for this ). Our client QA doc is more of a summary, with final screenshots. I wish there was a way to automate taking email client screenshots from multiple devices. I played around with Edge Inspect but it doesn't work for what I had in mind. Right now its manual and I include a cross section in my report. Clients gain confidence seeing screens from real devices that some don't feel with previews tools, unfortunately it's just not practical to screenshot 70+.
Do you mind sharing your current full device list, and I know it's a bit cheeky but the last three you purchased and the next three on your to-buy list?
Not cheeky at all...we currently have 34 devices. The last three we purchased were: S4 running 4.3, latest iPod Touch to run 7.0 and I think it was the BBZ10 or MacBook Air ( we work on PCs ). We were implementing an HTML5 video project and it was a ton of QA. I got away with fewer Android purchases last year, we had multiples running GB and as its marketshare decreased we upgraded a few.
Next on my list is another Samsung running KitKat 4.4 and Graeme mentioned buying a WP8, though its hard not to give up on that platform at least in the US. I'm not sure after that, maybe one of those big Android tablets or a Samsung Gear. We've been doing translations for 18 different markets, so I've been wondering how to take the device lab global. The US is one thing but what devices are popular in Japan, China and Russia? That's an area I'm researching now.