Last week I wrote about the huge gaps that can (and probably do) exist between the power users and the mainstream users of your product.
But what do you DO about it? Power users are your evangelists, your backup QA testers, and your visionaries. They’re also by far the easiest to get in touch with (just try to avoid them — they’ll find you.) It would be stupid to blindly ignore their feedback. It’s also really hard to tell sometimes: is this an issue that only affects the power users? Or is it a universal thing, and they were just the first to detect it?
Here’s how I try to evaluate feedback, bug reports, suggestions:
Understand When, How Often, Why
- When is X happening? (in what circumstances? if there was anything numerical involved like # of messages, file size, # of users, what was the number?)
- How often is X happening? (every single time/occasionally/just once but it was really catastrophic?)
- Why is X happening? (what were you doing right before it happened?)
The fastest way to ask these questions is just to shoot the person an email asking followup questions — most people will be able to answer. I often also add “if it’s easier to explain over the phone or chat, here’s my contact info”).
You’ll hear things like “…it happens when I upload a file. What? Oh, the file is 700MB, does that make a difference?” or “well, I wanted to match our obscure data format, so I changed every single one of the settings…”
Look at Non-Average Data
Averages are really easy to measure, and lots of tools default to “average” values. But the “average” value hides a ton of variability. Let’s say you have 10 users who sent 1, 1, 1, 1, 2, 4, 4, 4, 800 messages.
The average number of messages sent is 81.8. The median number of messages sent is 2.
That’s a 40x jump. That’s a good multiplier to keep in your head when you read feature requests. Just by asking yourself, “is this a situation that only affects the 40x user?”, you’ll more effectively prioritize feature requests, complaints (and compliments).
Go Find Some Normal Users
Want to know if this problem is affecting mainstream users? Ask them. You don’t need a lot of data — just a couple of anecdotes can provide a very useful sanity check on your assumptions.
I find mainstream users to talk to by:
- Responding to people who submitted support tickets. (Even if it was an unrelated issue, if you email them personally, and it’s clearly not an automated/marketing email, and you’re nice, a decent percentage of them will answer a question or two for you. Surprising but true!)
- Using Qualaroo to pop up a survey question while people are on-site, then follow up to their response via email.
- Asking your power users if they have “less tech-savvy coworkers” who also use your product, and can you get their email address? You’ve got to be tactful with this one: you don’t want their coworker to feel dumb.
Usually the “conversation” can be as simple as: “One of our other customers described this problem. Does this sound familiar to you? / Has this ever happened to you?”
If they have no idea what you’re talking about, this may be an ignorable power-user problem.
If their response is a “yes”, a “well, sorta…”, or a related followup question — you probably have an issue that affects a significant percentage of users. Now it’s time to dig deeper into the when/how often/why.
Think about None, A Few, and A Lot Use Cases
Settings, features, preferences, limits, and displays are often very different based on ‘amount’ use cases. We tend to spec out or mock up “optimal” use cases — here’s how it’ll work with an average number of users, an average number of items, an average file size. Too bad our users aren’t so average.
Instead, think about how this change/solution will work:
- When there is 0 activity
- When there are very few users
- When there are no messages/projects/photos/items to act upon
- When the display looks sparse
- When the files are enormous
- When there are 1000s of users
- When there are 1000s of messages
- When the display is overwhelmingly cluttered
If you’re lucky, you have an admin tool where you can emulate these situations. If not, it’s worthwhile to take some extra time and mock up options.
And finally: be prepared for complaints
The corollary of “if you’re not embarrassed by the MVP of your product, you waited too long to launch it” is “if your power users aren’t angry at you, you’re not going to succeed in the mainstream.”
Sorry everyone: I wish we could build that crazy feature too. But we’re not going to.