Growth graphs like this don’t come around too often:

slack-1year-Feb12-2015-dau

I’ve gotta hand it to Slack, they’re playing the PR game pretty well too. Headline after headline on Techcrunch and Techmeme keep telling us all how crazy the growth is over there. They’ve already built the brand to go with it.

But here’s the thing about rocketships: they either go further than you ever thought possible or they blow up in your face.

It all comes down to momentum. Keep it up and you’ll quickly dominate your market.

But once you start to lose momentum, the rocketship rips itself apart. Competitors catch up, market opportunities slip away, talent starts to leave, and growth stalls. It’s a nasty feedback loop that’ll do irrecoverable damage. Once that rocketship lifts off, you either keep accelerating growth or momentum slips away as you come crashing back down. Grow or die.

There’s no room for mistakes. But when driving growth, there’s 2 forces that will consistently try to bring you crashing back down.

1) The Counter-Intuitive Nature of Growth

At KISSmetrics, I launched a bunch of tests that didn’t make a single bit of sense when you first looked at them. And many of them were our biggest winners.

Here’s a good example. Spend any time learning about conversion optimization and you’ll come across advice telling you to simplify your funnel. Get rid of steps, get rid of form fields, make it easier. Makes sense right? Less effort means more people get to the end. In some cases, this is exactly what happens.

In other cases, you’ll grow faster by adding extra steps. Yup, make it harder to get to the end and more people finish the funnel. This is exactly what happened during one of our homepage tests. We added an entire extra step to our signup flow and instantly bumped our conversions to signups by 59.4%.

Here’s the control:

Home Page Ouath Test Control

Here’s the variant:

Home Page Oauth Test Variant

In this case, the variant dropped people straight into a Google Oauth flow. But we didn’t get rid of our signup form since we still needed people to fill out the lead info for our Sales team.

Number of steps on the control:

  1. Homepage
  2. First part of signup form
  3. Second part of signup form (appeared as soon as you finished the first 3 fields)

Number of steps on the variant:

  1. Homepage
  2. Google account select
  3. Google oauth verification
  4. Signup form completion

You could say the minimalist design on that variant helped give us a winner which is true. But we saw this “add extra steps to get more conversions” across multiple tests. Works like magic on homepages, sign up forms, and webinar registrations. It’s one of my go-to growth hacks at this point.

Counter-intuitive results come up so often that it’s pretty difficult to call winners before you launch a test. At KISSmetrics, we had some of the best conversion experts in the industry like Hiten Shah and Neil Patel. Even with this world-class talent, we STILL only found winners 33% of the time. That’s right, the majority of our tests FAILED.

We ran tests that I would have put good money on. And guess what? They didn’t move our metrics even a smidge. The majority of our tests made absolutely zero impact on our growth.

It takes a LOT of testing to find wins like this. So accelerating growth isn’t a straight-forward process. You’ll run into plenty of dead-ends and rabbit holes as you learn which levers truly drive growth for your company.

2) You’ll Get Blind-Sided by False Positives

Fair enough, growth is counter-intuitive. Let’s just A/B test a bunch of stuff, wait for 95% statistical significance, and then launch the winners. Problem solved!

Not so fast…

That 95% statistical significance that you’ve placed so much faith in? It’s got a major flaw.

A/B tests will lead you astray if you’re not careful. In fact, they’re a lot riskier than most people realize and are riddled with false positives. Unless you do it right, your conversions will start bouncing up and down as bad variants get launched accidentally.

Too much variance in the system and too many false positives means you’re putting a magical growth rate at serious risk. Slack wants to get as much risk off the table while still chasing big wins. And the normal approach to statistical significance doesn’t cut it. 95% statistical significance launches too many false positives that will drag down conversions and slow momentum.

Let’s take a step back. 95% statistical significance comes from scientific research and is widely accepted as the standard for determining whether or not two different data sets are just random noise. But here’s what gets missed: 95% statistical significance only works if you’ve done several other key steps ahead of time. First you need to determine the minimum percentage improvement that you want to detect. Like a 10% change for example. THEN you need to calculate the sample size that you need for your experiment. The results don’t mean anything until you hit that minimum sample size.

Want to know how many acquisition folks calculate the sample size based on the minimum difference in conversions that they want to detect? Zero. I’ve never heard of a single person doing this. Let me know if you do, I’ll give you a gold star.

But I don’t really blame anyone for not doing all this extra work. It’s a pain. There’s already a ton of steps needed in order to launch any given A/B test. Hypothesis prioritization, estimating impact on your funnel, copy, wireframes, design, front-end and back-end engineering, tracking, making the final call, and documentation. No one’s particularly excited about adding a bunch of hard-core statical steps to the workflow. This also bumps the required sample sizes for conversions into the thousands. Probably not a major problem for Slack at this point but it will dramatically slow the number of tests that they can launch. Finding those big wins is a quantity game. If you want higher conversions and faster viral loops, it’s all about finding ways to run more tests.

When you’re in Slack’s position, the absolute last thing you want to do is expose yourself to any unnecessary variance in your funnel and viral loops. Every single change needs to accelerate the funnel like clockwork. There’s too much at stake if any momentum is lost at this point. So is there another option other than doing all that heavy duty stats work for each A/B test? Yes there is.

The Key to Keeping Rocketships Flying

Right now, the team at Slack needs to be focusing on one thing: how not to lose.

Co-founder Stewart Butterfield mentioned that he’s not sure where the growth is coming from. This is a dangerous spot to be in. As they start to dive into their funnel, there’s a serious risk of launching bad winners from false positives. They’ll need every last bit of momentum if they want to avoid plateauing early.

As it turns out, there is a growth strategy that takes these A/B testing risks off the table. It’s disciplined, it’s methodical, and it finds the big wins without exposing you to the normal volatility of A/B testing. I used it at KISSmetrics to grow our monthly signups by over 267% in one year.

Here’s the key: bump your A/B decision requirement to 99% statistical significance. Don’t launch a variant unless you hit 99%. If you’re at 98.9% or less, keep the control. And run everything you can through an A/B test.

Dead serious, the control reigns unless you hit 99% statistical significance. You’ll be able to keep chasing counter-intuitive big wins while protecting your momentum.

At KISSmetrics, we actually did a bunch of Monte Carlo simulations to compare different A/B Testing strategies over time.

I’ve posted the results from 3 different strategies below. Basically, the more area under the curve means the more conversions you earned. Each dot represents a test that looked like a winner. You’ll notice that many dots actually bring the conversions down. This comes from false positives and not being rigorous enough from your A/B testing.

Here’s what you get if you use the scientific researcher strategy:

conversion-rate-vs-observations-600

Not much variance in this system. Winners are almost always real winners.

Here’s your regular sloppy 95% statistical significance strategy that makes changes as early as 500 people in the test:

the-impatient-marketer-600

Conversions bounce around quite a bit. False wins come up often which means that if you sit on particular variation for long, it will drag those conversions down and slow growth. There goes your momentum.

Now let’s look at the 99% strategy that waits for at least 2000 people in the test for a decent sample size:

the-realist-conversion-rates-600

Still a chance to pick up false winners here but a lot less variance than 95%. Let’s quantify all 3 strategies real quick by calculating the area under the curve. Then we’ll be able to compare them instead of just eye-balling the simulations.

  • Statistical researcher = 67759
  • 95% statistical significance = 60532
  • 99% statistical significance = 67896

Bottom line: the 99% strategy performs just as well as the scientific researcher and a lot better than the sloppy 95%. It’s also easy enough for any team to implement without having to do the extra stats work.

The 99% rule is my main A/B testing rule but here are all of them:

  • Control stands unless the variant hits a lift at 99% statistical significance.
  • Run the test for at least a week to get a full business cycle.
  • Get 2,000 people through the test so you have at least a halfway decent sample size.
  • If the test looks like a loser or has an expected lift of less than 10%, kill it and move on to the next test. Only chase single digit wins if you have a funnel with crazy volume.

I used these rules to double and triple multiple steps of the KISSmetrics funnel. They reduce the risk of damaging the funnel to the bare minimum, accelerate the learning of your team, and uncover the biggest wins. That’s how you keep your growth momentum.

Embedding A/B Tests Into the Slack Culture

I can give you the rules for how to run a growth program. But you know what? It won’t get you very far unless you instill A/B tests into the fabric of the company. The Slack team needs to pulse with A/B testing. Even the recruiters and support folks need to get excited about this stuff.

This is actually where I failed at KISSmetrics. Our Growth and Marketing teams understood A/B testing and our entire philosophy behind it. We cranked day in and day out. It was our magic sauce.

But the rest of the team? Did Sales or Support ever get it? Nope. Which meant I spent too much time fighting for the methodology instead of working on new tests. If I had spent more time bringing other teams into the fold from the beginning, who knows how much further we could have gone.

If I was at Slack, one of my main priorities would be to instill A/B testing into every single person at the company. Here’s a few ideas on how I’d pull that off:

  • Before each test, I’d show the entire company what we’re about to test. Then have everyone vote or bet on the winners. Get the whole company to put some skin in the game. Everyone will get a feel for how to accelerate growth consistently.
  • Weekly A/B testing review. Make sure at least one person from each team is there. Go through all the active A/B tests, current results, which one’s finished, final decisions, and what you learned from them. The real magic of A/B testing comes from what you’re learning on each test so spread these lessons far and wide.
  • Do monthly A/B testing growth talks internally. Include the rules for testing, why you A/B test, the biggest wins, and have people predict old tests so they get a feel for how hard it is to predict ahead of time. Get all new hires into these. Very few people have been through the A/B test grind, you need to get everyone up to speed quickly.
  • Monthly brainstorm and review of all the current testing ideas in the pipe. Invite the whole company to these things. Always remember how hard it is to predict the big winners ahead of time, you want testing ideas coming at you from as many sources as possible.

Keep Driving The Momentum of That Slack Rocketship

I’m really hoping the team at Slack has already found ways to avoid all the pitfalls above. They’ve got something magical and it would be a shame to lose it.

To the women and gents at Slack:

  • Follow the data.
  • Get the launch tempo as high as possible for growth, you’ll need to run through an awful lots of ideas before you find the ones that truly make a difference.
  • Only make changes at 99% statistical significance.
  • Spread the A/B testing Koolaid far and wide.
  • Don’t settle. You’ve got the magic, do something amazing with it.

{ 8 comments }

In The Lean Startup, Eric Ries describes 3 engines of growth:

  • The Sticky Engine
  • The Viral Engine
  • The Paid Engine

In a post of mine, I claimed that there’s really only 2 engines. Short and sweet summary:

There aren’t 3 engines of growth, there’s only 2: organic and paid. I lumped the word-of-mouth into the viral engine and explained how retention doesn’t drive growth which was the main focus of the sticky engine. This is because churn scales with your acquisition so if you only focus on retention, your growth will stall regardless of how low you get it. So there’s no reason to have a sticky engine of growth.

But I made two mistakes.

One was an outright error on my part. The other as an omission which adds a bit of nuance.

Eric Ries even responded to the post (which was awesome) with 4 tweets:

Error #1: Blending Word of Mouth and Viral Engines

Eric Ries clearly separates viral from word-of-mouth (organic) engines of growth. I incorrectly lumped them together. In the first paragraph of his section on viral engines of growth, Ries states:

“This is distinct from the simple word-of-mouth growth discussed above [the sticky engine of growth]. Instead, products that exhibit viral growth depend on person-to-person transmission as a necessary consequence of normal product use. Customers are not intentionally acting as evangelists; they are not necessarily trying to spread the word about the product. Growth happens automatically as a side effect of customers using the product.”

Page 212 of The Lean Startup if you’re curious.

Fair enough, viral and word-of-mouth engines aren’t the same. One depends on delighting customers to the point where they voluntarily tell others about you. Viral engines depend on making the product visible to others as each customer uses it.

Keeping viral and word-of-mouth engines separate makes a lot of sense.

That’ll teach me to build off of frameworks while skimming them instead of reading the entire chapter again.

My bad.

Error #2: Ommiting that the Sticky Engine Scales When Word-of-Mouth Exceeds Churn

Eric Ries focuses pretty heavily on getting retention as low as possible for the sticky engine of growth. Page 212 in the Lean Startup:

“[For an engagement business] its focus needs to be on improving customer retention. This goes against the standard intuition in that if a company lacks growth, it should invest more in sales and marketing.”

In my post, I showed that growth hits a ceiling no matter how low you get your churn. This is because churn will eventually match your current acquisition rate. Even if you lower churn, your growth looks like this:

10_Percent_Monthly_Churn_Reduced_to_7_Small

There’s one main exception to this.

When you get a word-of-mouth growth rate to exceed your churn rate, you’ll grow exponentially. Even though your churn grows each month, so does your word-of-mouth. Then you get a nice compounding growth rate that accelerates over time. Ries points this out on page 211:

“The rules that govern the sticky engine of growth are pretty simple: if the rate of new customer acquisition exceeds the churn rate, the product will grow.”

But this doesn’t change my primary point: churn is not the key to growth for the sticky engine. Accelerating word-of-mouth is the key instead of churn. Getting customers to keep using your product is one thing. Getting them to put their own reputation on the line by recommending you is another hurdle entirely. You’ll still hit low churn long before you see any substantial word-of-mouth.

Ries does bring up an example of a business that has 40% churn and 40% acquisition at the same time. And when your churn matches your acquisition, you stall. He focuses on lowering churn to get the sticky engine running. But I’m skeptical that the acquisition is coming from actual word-of-mouth. With churn that high, I’d expect the acquisition to be from conventional sales and marketing channels that don’t scale with churn. And if that’s the case, lowering churn is only the first step. You’ll hit a new ceiling since your acquisition won’t scale as easily as churn does.

If you have worked with a business that achieved high rates of growth from word-of-mouth but also had high rates of churn, I’d love to hear about it. Be sure to let me know in the comments.

Once again, churn is just one piece of the puzzle. You’ll still need to keep refining your product and improving your customer support long after you achieve low churn. Word-of-mouth requires delighting customers at an entirely different level than what it takes to keep them around. In other words, low churn is the first step to word-of-mouth growth. It grows your average customer value and extends your runway. But you’ll need to keep pushing in order to get your word-of-mouth high enough that it outpaces churn. Then, and only then, will you have a sticky growth engine.

If you focus on delighting customers to the point where you get a sizable amount of word-of-mouth growth, you’ll hit low churn along the way.

To recap, you have two options when your growth stalls:

  1. Find a way to accelerate your current acquisition with paid or viral engines (you’ll eventually hit another plateau unless you keep accelerating it)
  2. Focus on your product and customer support to increase word-of-mouth (and lower churn along the way).

Lowing your churn will make either strategy more viable. You’ll either start growing exponentially at a lower rate of word-of-mouth or you’ll lower the demands on your acquisition which makes it easier to outpace churn.

{ 4 comments }

UPDATE: I actually made 2 errors with this post. I decided to correct them with a new post so you can see exactly what happened. See the mistakes I made in this post over here.

One of my all-time favorite books for startups is Eric Ries’ The Lean Startup.

In it, Eric Ries breaks down three engines of growth:

1. The Viral Engine of Growth – Word of mouth or viral invite system

2. The Paid Engine of Growth – Pay for each customer through ads or marketing systems

I’m mostly in agreement with Ries on these two. You’re either going to have to pay for your customers or you’ll need to grow from word-of-mouth/viral invite systems.

But the third system of growth really isn’t a system of growth:

3. The Sticky Engine of Growth – Keep customers engaged over the long term and reduce churn

This applies to two types of businesses, subscriptions and user engagement. Software-as-a-Service companies use subscriptions so the longer people stay subscribed, the more money they make. For consumer tech companies like Twitter, Instagram, or Facebook, they rely on user engagement so they can monetize their users with ads. In both cases, the business benefits as users keep using the product over the long term.

The strategy for this growth engine is pretty straightforward: reduce your churn to increase the value of your customers. You do this by keeping customers engaged and lowering the percentage that leave in any given month (your churn rate).

But Ries’ Sticky Engine of Growth doesn’t actually produce growth that scales.

You Can’t Get Hockey Stick Growth By Only Attacking Churn

Churn is not a path to growth. It simply raises your growth limit. It’s the ceiling that lets you keep playing. It buys you time and gives you more breathing room.

But if you want hockey-stick growth, you’ll need to build another engine of growth WHILE attacking your churn rate.

Here’s the problem: when you have a “sticky” business and need long-term customer engagement, churn puts your business into a constant rate of decay.

Churn nips at your heels, rots your customer base, and will deadweight your company if you’re not careful.

Let’s do a quick example.

Say you have a 10% churn rate for a SaaS app. Let’s also say that you’ve found a way to acquire 100 customers per month. Here’s what happens to your growth if you keep your acquisition constant:

10_Percent_Monthly_Churn_Small

Early on, the 10% churn doesn’t really matter. Your 100 new customers easily make up for it. But once you get to 1000 customers, you churn rate equals your acquisition rate. Within 2 years, your business has stalled.

In order to beat churn, you have to keep accelerating your growth. Even if you have 1-2% churn (the goal for SaaS companies), your growth will consistently slow down unless you build another engine to accelerate it. Churn doesn’t get you to the next level, it simply let’s you take another shot.

Now let’s say that we reduce the churn rate from 10% to 7.5% after 6 months. Here’s how your growth differs from the first example:

10_Percent_Monthly_Churn_Reduced_to_7_Small

See how you hit that next ceiling after an small spike? When people talk about growth from lower churn, it’s that initial spike since the growth rate now exceeds the churn rate. But it doesn’t take long for the new churn rate to catch up and stall the business again.

No matter how low you get your churn, you’ll hit a cap sooner or later. Your growth will keep slowing down as every month goes by. The only way to accelerate growth is to build one of the primary growth engines: organic or paid.

The reason that churn is so nasty is that it quickly scales to the size of your business. 10% churn with 100 customers means that 10 customers left this month. If you somehow manage to get to 100,000 customers without addressing your churn, you’ll now be losing 10,000 customers each month. It’s fairly consistent all the way up. But marketing, sales, and growth systems don’t scale so easily. Paying for 10,000 new customers each month is an entirely different game than 10 new customers. Even viral systems don’t scale forever, they’ll start to slow and churn will catch up in a hurry.

And don’t convince yourself that you can achieve some absorb churn rate like 0.1%. Top-tier SaaS businesses are in the 1-2% range, maybe as low as 0.75%. There are hard limits on how low you can go.

Considering that most VC’s are looking for at least 100% year-over-year revenue growth rates for SaaS (consumer tech has even more absurd growth benchmarks), you need to build a growth engine that doesn’t mess around.

I’ve spent 2 years understanding the growth model of a SaaS business at KISSmetrics. While churn is one of our top priorities, we wouldn’t get very far unless we committed to building an additional growth engine. That’s why we built out our marketing and sales teams.

Maybe you double-down on product and customer service to accelerate word-of-mouth. If you’re in consumer tech, a viral invite system might work if it adds to the core value of your product. Or maybe you build a paid engine with content marketing and ad buys. Either engine can work. But you need to remember that growth won’t come just from lower churn.

Why does this matter?

If you’re building a business that relies on keeping customers engaged over time, you cannot expect to grow your company from just a low churn rate.

Churn is absolutely CRITICAL to the success of your business. But it’s only one piece of the puzzle.

Look at any SaaS business that has IPO’d recently like Marketo or Box. They all have massive marketing/sales budgets. They’re even hemorrhaging cash to keep accelerating their growth rates.

That being said, I DO agree with Ries that the primary goal of a sticky business model is to focus on customer retention. No subscription or engagement business is going to get very far unless they control their churn. You’ll hit a ceiling that won’t budge until you do. Before you can think about growth, you need to get your churn to acceptable levels. Or all your customers will leave just as fast as you acquired them.

But once you have a low churn rate, growth isn’t going to magically appear. And a business looking for high rates of growth will need to acquire customers at scale. Raising engagement will increase the value of your current customers but it won’t necessarily bring you new customers. You’ll need to delight customers to the point that word-of-mouth and virality start working in your favor. Or you’ll need to start paying for customers.

Sticky Engines Don’t Acquire Customers, They Grow Customer Value

The primary benefit of sticky engines isn’t growth, it’s an increase in customer value.

Any subscription or engagement business attempts to spread customer payments out over a long period of time. For many SaaS businesses, the goal is to keep customers subscribed for 24-36 months. By spreading payments out, you’re able to increase the value of your average customer. This is one of the main reasons that tech companies have moved to subscription payments instead of up-front software licenses. And consumer tech companies can monetize long-term, active users a lot easier with ad revenue.

In fact, a well-executed upsell and churn reduction system can give you negative churn. This means the value of your current customers is increasing faster than the value lost from customers leaving. Your total customer count drops while your revenue increases slightly. Even if you don’t acquire any more customers, your revenue will still grow. At least in the short-term.

But this isn’t considered a primary growth engine. It’s mainly a strategy to mitigate the impact of churn so you get the full benefit of your real growth engine. The revenue growth from negative churn pales in comparison to any half-decent growth engine. Negative churn will only give you marginal gains.

Won’t Better Engagement Lead to More Word-of-Mouth and Virality?

Possibly.

Word-of-mouth growth requires a level of engagement well beyond what it takes just to keep customers engaged each month. Providing enough value to keep customers interested is one thing, providing enough for them to drag their friends into the product is something else altogether.

If you’re pursuing organic growth instead of paid, many of the tactics you employ will be very similar to the tactics that you’d use to reduce churn:

  • Improve value of product
  • Reduce friction across all customer touch-points
  • Focus on a single market
  • Provide fast and helpful customer service

But you’ll need to perfect these tactics and delight your customers at a level well beyond what it takes just to reduce your churn. At that point, you’re deliberately pursuing an organic engine of growth.

That being said, I’m a huge fan of Eric Ries’ book The Lean Startup. I definitely consider it one of the classics for startups. It’s a huge inspiration for my own work, I highly recommend it.

{ 0 comments }