We all know how important testing is. Some of our marketing works, some of it doesn’t. Some customer love us, others don’t. Testing helps us cut through that endless fog of uncertainty.
If you spend any time in the startup or online marketing circles, you’ll hear how wonderful A/B testing is. Let’s say you want to test your home page. Throw up two different versions (usually the current version and your new one), compare the results, and BAM, you know which is better.
Sounds easy right?
Well it isn’t. Here’s the thing, we need traffic to make those tests valid. If you only have 10 people go through your test, the numbers are just too small. You won’t be able to learn anything. At that volume, our results are just random coincidence. We mine as well just flip a coin. So we need dozens, hundreds, and preferably thousands of people to run our test on.
Well that’s just swell. What do we do if we don’t have hundreds of customers or visitors? What if we have like 2?
Even when you only have a few customers or visitors, you still need to be testing. And you’ll need to take a completely different approach. And it just so happens that we use this approach every day at KISSmetrics (for slightly different reasons).
Instead of obsessing over data, we need to reach out and talk to people directly.
Let’s run through the nitty gritty with a real example.
How I Get Test Results From Small Groups of People
Right now, I’m helping redesign the People Search report in KISSmetrics. The report helps you find different groups of people within your data. And we’re revamping it to provide better insights and making it easier to use.
If we followed the conventional approach, we would build a second version of the report, enable it for a small portion of our customers, and compare engagement rates between the two groups.
But we’ve gone a different route.
Before we drafted a single new design, we started talking to our customers. In fact, we scheduled meetings with 5 of them to get a DEEP understanding of how they use the report. We learned why they use the report, what frustrates them, and how the report helps them build a better business.
And we didn’t stop there.
We started putting together some mock-ups (really rough drawings) of ideas we had on how to make it better. Then we scheduled some more meetings. This time, we showed the drawings alongside the current version. First, we asked them to walk us through the current report and how they use it. Then we showed them the drawings of the new version and asked them what they liked and didn’t like. We ran our own little A/B test right on Skype.
The feedback we got was amazing. It saved us hours of work and kept us from building something that would have been much worse. Right away, we learned that one of our ideas was straight up terrible. Not kinda sorta bad. Completely terrible.
With a few meetings and a drawing, we ran our A/B test with limited data.
In our case, we’re looking for speed. Will 5 people give us a complete picture of what all of our customers want? No. But it will allow us to move a LOT faster. If we wanted to do a legit A/B test, we’d have to make a bunch of guesses on what we think are improvements, build and design the whole thing, then launch it. This takes weeks of effort.
Instead, we can get quick feedback on our overall direction and change course without having to build the whole thing.
We also gain a depth of understanding that a simple A/B test would never give us. By talking through the report with someone, we understand what they’re biggest frustrations are and how they incorporate it into their business. Then we’re able to build a new version that goes far beyond the first.
And this process works just as well when you don’t have enough data to run a full A/B test.
What if you only have 2 customers? Reach out to them and talk through your different ideas. What if you don’t have ANY customers? Get in touch with the types of people you think would be your customers. By listening for the passion in people’s voices, you’ll quickly know if you’re on track.
When multiple people start getting excited about the same idea, that’s when it’s time to turn that idea into a reality.
5 Best Practices When Talking to Customers and Testing Ideas
1. Ask open-ended questions without leading the customer.
Don’t show someone a drawing of a new design or product, explain the entire thing to them, and then ask for feedback. You won’t be there to explain the concept to every customer. You want that initial reaction to better understand what customers will be going through when they see it for the first time on their own.
2. This is not a sales call.
Don’t sell anything, not even yourself. In fact, you should be listening a lot more than you’re talking. Spend your energy focusing on understanding the perspective of who you’re talking to and ask them great questions.
3. Take notes.
Once you start doing several of these, it’s going to be really hard to keep track of who said what. Taking notes will make sure you’re able to keep everything straight. Feel free to record the meeting if you don’t want to take notes at the same time. You can even have your recordings transcribed for you and save some time.
4. Look for passion.
You’re on the right track when people start to get excited over your ideas. If they start asking “Where can I get this?” or “When will it be available?”, you’re definitely on to something. Listen for the passion in their voice, it’ll tell you what’s most important.
5. Look for trends.
We still want to be careful with not getting enough data. Remember that one idea I had to completely throw out because no one liked it? Well, one guy liked it. If I had only talked to him, I would have recommended changes that the rest of our customer base would have HATED. Avoid making changes on a single comment. Look for things that come up over and over again. That’s where you want to focus.
The Bonus Points of Awesome
As you start doing these interviews on a regular basis, you’ll enjoy another serious benefit. You’ll start building relationship with all the customers you’re talking to. They’ll go from happy customers to loyal evangelists without you having to do any extra work. Simply by reaching out to people and asking them for help, they’ll become more committed to your cause. They won’t be able to stop talking about how awesome you are when they see how their feedback helps you build a better product.
This alone is worth the effort of all these interviews. Combined with learning which of your ideas will work the best, this is one of the easiest ways to grow your business.