|
|
|
|
Quality Assurance (QA) hails from factories and manufacturing plants, where QA personnel checked physical parts to ensure that each piece met a rigid specification and prevented improperly-made components from reaching customers. Software development borrowed both the term and the strategies to create a type of QA that stands between a software’s code base and its users. While the methods and pace have changed since the heyday of manufacturing, the mindset remains intact. At a company like MailChimp—where our motto is “Listen Hard, Change Fast”—QA can’t afford to be a gatekeeper or slow down the process.
MailChimp QA is successful when people notice the app’s ease of use and powerful tools. To meet this goal, we’ve borrowed from a variety of disciplines to form a testing process that relies on human analysis, compassion, and speed. Ultimately, however, my QA philosophy centers on three characteristics: flexible, inquisitive, and nimble.
Flexible
MailChimp’s code development occurs in a five week cycle, with the final week devoted to testing. Our development cycle is light on process so we can quickly make updates. Features evolve quite a bit in five weeks, as we learn about user wants and needs, which means QA operates without a lot of lead time. Because of this process-light cycle, QA Analysts keep testing plans fluid enough to accommodate any last minute app changes that might come our way in the days leading up to the next release.
During the four weeks when QA isn’t exclusively focused on the MailChimp app, we turn our attention to mobile apps, integrations, and the MailChimp site. Frequent movement between projects means we can’t get bogged down by one activity. We’ve got to balance multiple release deadlines at a time, so testers move on and off of projects quickly.
Inquisitive
When I set out to build our QA team, I wanted to balance the pace of development with our ability to assess the user experience. We want to know if a feature helps customers get their jobs done—not just if it functions. If our customers can’t accomplish their goals, we want to understand why. We want to test with the mindset of a user. And since the QA team isn’t typically able to interact with many users, we ask for input from our colleagues who do.
We have an excellent customer research team in the office that pulls together user profiles, transcripts from interviews, and videos of users interacting with our application. It helps us understand where our customers are coming from and how different customers use varying combinations of our features. QA pours over this raw data to glean new ideas for tests. This granular perspective of customers helps us create user-centric testing scenarios that specifically address actual behaviors. For example, if QA is testing changes to something like MailChimp sign-up forms, we might try to create sign-up forms that use some of our customers’ brand colors and aesthetics—to see if it’s possible and something our customers could accomplish.
Research reports and videos aren’t the only way QA learns about the people who use our products. Many of the QA team members are former MailChimp support agents (including myself), so we visit our old stomping grounds to watch as customer support chats and emails are answered in real time. Chats give us immediate feedback about areas in our app that customers find difficult and confusing. Hearing from customers in the moment they need help gives us ideas for QA tests modeled after actual customer behaviors and struggles. We also log what we observe on these visits, because reoccurring patterns in feedback may lead to future developments in the app.
Following along in Support helps QA act both as software testers and user advocates. On a short release cycle, prioritization is key, and user feedback shapes those priorities. For example, with the recent release of our Automation workflows, we needed to make sure users could move content from autoresponders—our previous automation tool—into automation workflows. QA filtered through support emails and recreated users’ workflows as best we could. Testing realistic workflows from the perspective of a user makes it much easier to identify bugs that could be potential showstoppers.
Making a habit of talking to our colleagues about our customers sparks conversations across our Product, Support, and QA teams, getting everyone involved in the development process. These conversations help the QA team challenge our own assumptions about our users, which improves the quality of our testing. Cross-department communication also encourages design and development teams to think through features thoroughly, so we can quickly make refinements to the app.
Nimble
So how do you move fast when you’ve got piles of user research to read and teams of people to talk to? We’ve found that we needed to do away with some traditional QA processes in order to test quickly. Long, prescriptive test cases just don’t work for us. We capture test ideas in Asana, our task manager, as we think of them. Most test ideas are snippets of thoughts, like “Check for international workflows” or “Will this work with ecommerce data?” Other notes are snippets of emails our users sent to support.
We also don’t want to be too prescriptive with testing because it kills the creative process. I realized QA can get through much more when we aren’t bogged down by unnecessary documentation requirements or blinded by detailed specifications. Although there is usually a small list of “do not miss” items that we prioritize for each release, test plans are designed to give our testers high-level perspective, rather than boss them around. QA is empowered to come up with creative tests that cover multiple parts of MailChimp in one pass.
For example, to test updates to the MailChimp Email Beamer, a tool for creating campaigns without logging into MailChimp, we built a campaign using several languages, image types, and emojis. This kind of testing helps us consider multiple use cases at one fell swoop. While giving testers specific guidelines to follow might appear to streamline the process, once a tester reads a structured testing specification, it’s difficult to think outside that specification and hampers the ability to test creatively.
Over the past two years, we’ve established a flexible testing method tailored to our unique development process and needs. By gathering information about customers from our researchers and support agents, we maintain a sharp focus on user needs in the testing process while keeping up with our fast-moving production cycle. We sweat the small stuff—quickly.
|
|
A Pause to Reflect, an Expanded Focus and a Secret Project
by Jason Beaird
|
|
|
We started this newsletter for a few reasons: as a way to share our work, to contribute to the UX community, and to use our own products with purpose. Over the last 30 issues, we've featured a host of MailChimp developers, designers, and researchers, among others. Some of our past authors were from traditional UX roles, while others came from elsewhere in the MailChimp universe. What becomes apparent, in hindsight, is that The UX Newsletter would not exist without these other contributors—and especially not without shared vision and extensive collaboration.
The Path Forward
It's been nearly 2 months since Issue 30, a little longer than usual. During this break, we thought about the ground we want to cover here. Don't worry, our love for UX lives on, as will the UX newsletter. We’ll continue to write about design flows, development tips, and high-fiving chimps. But we’re also going to feature more authors from other departments at MailChimp. We may even include some invited guests from the greater UX community. If you have any suggestions, we'd love to hear 'em.
One More Thing
In addition to thinking about where we want to take the newsletter, we've been quietly working together with the DesignLab on something special for all you loyal subscribers out there. We're planning to announce that early next week, so stay tuned and as always, thanks for reading!
|
|
Ask Us Anything
We want this newsletter to be a dialogue. If you've got questions about our must-have productivity apps, how many miles our research team traveled this year (spoiler alert: 96,341!), or our favorite Thanksgiving dish, send them in! Seriously: hit reply and ask us anything. We'll try to answer every email and maybe even share our conversation in future newsletters.
|
|
|
|
|
|
|
