You’re being lied to.

Well, not intentionally.

We’re constantly being pinged with stories of companies that have rocketed to success. Especially in tech, there’s always another $1 billion unicorn around the corner. Uber, Facebook, Airbnb, Slack, Zenefits, Box, Shopify, yadda yadda yadda.

At the same time, rock-solid companies seem to lose their way and crater.

We’re all desperate to know why.

Makes sense. We want to replicate the crazy success and avoid failure.

This is where we all get sucked in the nonsense narratives. They’ll give you false hope on how to produce success.

Here’s a good example: should a company expand into different products, industries, or markets? We’ll answer this question in a minute.

But first, who loves LEGO? I DO. My favorite childhood toy by far. You know how people will buy huge mansions and a dozen sports cars if they ever hit it big? I’ll just buy every LEGO set and fill an entire room with them. These days, they even have a Batwing with Joker steamroller set. How cool is THAT?

LEGO_Batwing_and_Joker_Steamroller

As it turns out, LEGO is a great case study for how delusional we can be about what produces successful companies.

Go check out LEGO’s 2014 annual report. In 2014, their net profit increased by about 15% to over 7 billion Danish krone (DKK). At current exchange rates, that’s about USD$1 billion. Back in 2011, they pulled DKK 4 billion in net profit. So they’ve had similar growth rates since 2011 and have nearly doubled their net profit. Not shabby at all.

Why is Lego doing so well? Management gives the credit to expansions beyond it’s core business, they crushed it with the LEGO Movie and the new line of LEGO sets that released with it. Right on page 5 of the annual report: “new products make up approximately 60% of the total sales each year.” They’ve also seen a lot of growth from the toy market in Asia.

So we’ve answered our question right? If we want to keep growing, we’ll want to expand beyond our core product and market base at a certain point, right?

Well, wait a minute. Our story isn’t that simple.

Go back to 2004 when LEGO nearly went bankrupt. Their COO, Poul Plougmann, got sacked and the business press lambasted the company for poor results. They caught a ton of flack for releasing a LEGO Harry Potter line (apparently, sales slowed when there was a gap between some of the Harry Potter movie releases), experimenting with new toy products, jumping into video games, launching a failed TV show, and trying to go beyond it’s core brand. The consensus was that they should get back to their core base and stop messing around by trying to innovate into new products.

Wait, which is it? In 2014, product expansion from the LEGO Movie helps push the company to new heights. In 2004, the LEGO Harry Potter line, TV shows, and the first attempt at video games nearly pushes it to bankruptcy. During each period, we push narratives and recommendations that contradict themselves. Go back to your core base! Wait, never mind! Expand into new products!

I can’t take credit for this insight or finding the LEGO story. It’s one of the case studies used in The Halo Effect by Phil Rosenzweig.

Rosenzweig shows how narratives are twisted to explain results after they occur. He wrote the original version of his book back in 2007 (there’s a new 2014 copy that you should grab if you haven’t yet). Then after the book is published, LEGO turns around and we start attributing their success to LEGO’s constrained innovation:

LEGO went back to its base. Innovation trashed the company in 2004 because it was highly unprofitable and expanded beyond its core strengths. Now LEGO has entered another golden era by constraining innovation.

But LEGO just had another huge year by expanding into its first movie. Hard to get further from its product base than that. A decade ago, the LEGO TV show got part of the credit when LEGO struggled. Now the LEGO Movie gets the credit when profits have turned around.

Again, which is it? Innovation? Constrained innovation? Innovation as long as you do these 7 simple steps? Maybe all of the above? Reducing a business to a simple narrative for a blog post or interview is incredibly difficult. And you’ll want to be careful of any source that attempts to do so.

To be fair, David Robertson and Bill Breen wrote a book that dives into the Lego story. I’m hoping they capture the nuance of what went into LEGO’s turn-around. I haven’t read the book myself but it’s on my to-read list.

We’re all exceptionally good at rationalizing any argument. If things go well, we’ll cherry pick some attributes and credit them for the company’s success. Then when things go sideways, we take the same attributes to explain the failure. It all sounds nice and tidy. Too bad it’s a poor reflection of reality.

Phil Rosenzweig calls this habit of ours the Halo Effect. When things go well, we attribute success to whatever attributes stand out at the company. When things go poorly, we attribute bad results to those exact same attributes. It’s one of the 9 delusions that he covers in his book. Let’s go through each of them.

The Halo Effect

The tendency to look at a company’s overall performance and make attributions about its culture, leadership, values, and more. In fact, many things we commonly claim drive company performance are simply attributions based on prior performance.

This is what happened to Lego. In 2004, they’re skewered by the press for trying to expand beyond it’s core business. Now it can’t get enough praise as it drives growth into new markets and product lines.

This happens to companies, teams, and you. When things go well, the quirks get credit for success. When things go poorly, those same quirks get the blame. Our stories search for what’s convenient, not what’s true.

Remember this when you’re in your next team meeting. Someone will float a story for how you got to this point. If it sounds good, the story will spread and your whole organization will start shifting in response to it. And a nonsense story means nonsense changes. There are two things you can do to limit these non-sense stories:

  • Chase causality as often as you can (more on this in a moment). The better your team understands how your systems really work, the closer your stories will be to the truth.
  • Realize that your stories are typically nonsense. It’s your goal to test the validity of that story as fast as you can.

The Delusion of Correlation and Causality

Two things may be correlated, but we may not know which one causes which. Does employee satisfaction lead to high performance? The evidence suggests it’s mainly the other way around — company success has a stronger impact on employee satisfaction.

We’ve all heard the adage “correlation, not causation.” But when you’re about to come up short on a monthly goal, how easy is it to remember correlation versus causation? It’s not. We all break and reach for the closest story we can. Even if we avoid throwing blame around, we still grasp for any story that will guide our way through the madness.

Proving causality is one of the most difficult bars to reach. Very few variables truly impact our goals in a meaningful way. How do we deal with this?

If you only rely on after-the fact data, you never move beyond correlation. Every insight and every bump in a metric is, at best, a correlation. The only way to establish any degree of casualty (and we’re never 100% sure) is to run a controlled experiment of some kind. You’ve got to split your market into two groups and see what happens when you isolate variables.

This is why I push so hard for A/B tests and get really strict with data quality. They allow us to break past the constraints of correlation and gain a glimpse of causation.

If you limit your learning to just correlation, you’ll get crushed by those chasing casualty. They’ll have a much deeper understanding of your environment than you do. You won’t be able to keep up.

And remember, the business myths, stories, best practices, and press rarely look at correlation versus causation. It’s all just correlation.

The Delusion of Single Explanations

Many studies show that a particular factor — strong company culture or customer focus or great leadership — leads to improved performance. But since many of these factors are highly correlated, the effect of each one is usually less than suggested.

Data is messy, markets are messy, customers are messy. The complexities of these systems vastly exceed our ability to understand or adequately measure them. Variables interact and compound in limitless ways.

Whenever someone gives you a nice, tidy explanation for why a business succeeded or failed, assume it’s nonsense.

You can’t depend on a single variable to drive your business forward. World-class teams have mastered countless business functions, everything from employee benefits to market research. The hottest New York Times bestseller may give you a 5 step process on how to conquer the world with nothing other than whatever flavor-of-the-month strategy everyone loves at the moment. But that’s a single variable among many.

Remember that your business moves within an endlessly complex system. Not only are you trying to change this system, you’ll be pushed around by it.

The Delusion of Connecting the Winning Dots

If we pick a number of successful companies and search for what they have in common, we’ll never isolate the reasons for their success, because we have no way of comparing them with less successful companies.

Good ol’ survivorship bias. We can’t just look at winners. We need to find a batch of losers and look for the differences between the two groups. Otherwise, we’re just pulling out commonalities that don’t mean anything.

The tech “unicorn” fad has succumbed to this delusion. Everyone’s looking for patterns among the recent $1 billion tech startups, trying to find the patterns so they can build their own unicorn. But they’re doing many things in exactly the same way as all the startups that blow up or stall out. We just don’t hear about those failures. And if we do, those stories aren’t deconstructed in the same level of detail as the unicorns. So we get a picture of what amazing companies look like but a very limited view on how they differ from their failed counterparts.

Study the failures just as deeply as the successes.

The Delusion of Rigorous Research

If the data aren’t of good quality, it doesn’t matter how much we have gathered or how sophisticated our research methods appear to be.

Rosenzweig takes a shot at Jim Collins with this one. Jim Collins has written several well-renowned books like Good to Great, Built to Last, and Great by Choice. Collins and his team do a ton of historical research to figure out which attributes separate great companies from average companies. As Rosenzweig points out, most of this research is based on flawed business journalism that suffers from the Halo Effect. So the raw data for Collins’ research is horribly flawed which then means his books aren’t as solid as many people think.

Regardless of how you feel about Collins’ books, this is still a critical delusion to remember. It doesn’t really matter how sophisticated you are with modeling, data science, research, or analytics if your data sucks. Fix your data first before trying anything fancy.

This is where I start with every business I work with. Before jumping into growth experiments, A/B testing, or building out channels, I always make sure I can trust my data. Data’s never 100% perfect but there needs to be a low margin of error. The quality of your insights depends on the quality of your data.

The Delusion of Lasting Success

Almost all high-performing companies regress over time. The promise of a blueprint for lasting success is attractive but not realistic.

You will regress to the mean. Crazy success is an outlier by default. Sooner or later, results come back down to typical averages.

Mutual funds prove this point perfectly. In any 2 year period, you can find mutual funds that crush the S&P 500. Wait another 5-10 years and those same mutual funds have fallen back to earth. Your company is in the same boat. If things go crazy well, it’s a matter of time before you come back down. Take advantage of your outlier while it lasts.

This is particularly dangerous with individual or team performance. Is it really talent or are you just an outlier? Sooner or later, you’ll have some campaign or project that takes off. Well… if you launch enough stuff, you’re bound to get lucky. The real question is how long can you sustain it? Can you repeat that success? And since we all regress to the mean eventually, how can you use you current success to get through the eventual decline?

All channels decline, all products decline, all markets decline, all businesses decline. You will decline. What are you doing now to plan for it?

The Delusion of Absolute Performance

Company performance is relative, not absolute. A company can improve and fall further behind its rivals at the same time.

You’re graded on a curve whether you like it or not. Even if you’re improving, customers won’t care if your competitor is improving faster than you are. You’ll need to stay ahead of the pack no matter how fast the pack is already moving.

Otherwise, it’s a matter of time before you’ve lost the market. Your success isn’t determined in isolation. Just because you did a great job doesn’t mean you’ll achieve greatness.

This stems from a basic psychological principle: as humans, we do a terrible job at perceiving absolute value. This applies to pricing, customer service, product value, and every trait around us. In order to gauge how good or bad something is, we always look for something to compare it to. It really doesn’t matter if you cut prices by 50% if your competitor found a way to cut them by 60%. You’re still considered too expensive.

Your work will always be judged in relation to the work of your peers.

The Delusion of the Wrong End of the Stick

It may be true that successful companies often pursued a highly focused strategy, but that doesn’t mean highly focused strategies often lead to success.

Another shot at Good to Great with this one.

One of the core concepts in Good to Great is hedgehog versus fox companies. Hedgehog companies focus relentlessly on one thing. Foxes dart from idea to idea. According to Collins, amazing companies are all hedgehogs with ruthless focus.

But we don’t have the full picture of the risk/reward trade-off. It’s a lot like gambling or investing. You COULD throw your entire life savings into a single stock (hedgehog) and if that stock takes off… you’ll make a fortune. But if it doesn’t? You’ve lost everything. Investors that diversify (foxes) won’t reap extreme gains but they also won’t expose themselves to extreme loses.

Companies might work very similarly. Yes, hugely successful companies could tend to be hedgehogs. They made big bets and won. But that might not be the best strategy for your company if it means taking on substantial amounts of risk. Most importantly, we can’t say for sure what the risk/reward trade-offs look like without a larger data set of companies. Even if great companies out-perform average companies when they’re hedgehogs, there could be just as many hedgehog companies that weren’t so lucky.

The Delusion of Organizational Physics

Company performance doesn’t obey immutable laws of nature and can’t be predicted with the accuracy of science — despite our desire for certainty and order.

Physics is beautiful and elegant. Business is not.

No matter what you do, you cannot remove uncertainty in business like you can with physics. Books, consultants, blog posts, and pithy tweets will all try to convince you that a simple step-by-step process will take your business to glory. As much as we’d all like to have simple rules to follow, that’s not how this game is played. Business cannot be reduced to fundamental laws or rules.

And sometimes, the outcome is completely outside your control. Even if you do everything right, follow all the right strategies, use the best frameworks, hire the best people, and build something amazing, the whole business can still go sideways on you. We can’t remove uncertainty from the system. All we can do is stack the odds in our favor. Fundamentally, business and careers are endless games of probability.

Recap Time! The 9 Delusions From the Halo Effect

Here are all 9 delusions in a nice list for you:

  • The Halo Effect
  • The Delusion of Correlation and Causality
  • The Delusion of Single Explanations
  • The Delusion of Connecting the Winning Dots
  • The Delusion of Rigorous Research
  • The Delusion of Lasting Success
  • The Delusion of Absolute Performance
  • The Delusion of the Wrong End of the Stick
  • The Delusion of Organizational Physics

Don’t get sucked into the delusional narratives of success. Embrace the uncertainty.

{ 2 comments }

I’ve been known to read a few books.

For the last few years, I’ve actually been keeping track of how many books I read. Here’s my annual totals:

  • 2011 = 39
  • 2012 = 68
  • 2013 = 50
  • 2014 = 70

With the 9 books that I’ve read so far in 2015, that brings my total to 236. This doesn’t even include all the books I read, just the one’s that relate to business. I even finished all 100 books on Josh Kaufman’s Personal MBA reading list along the way.

Before we get into how I read this much (and how you can too)… why even bother? Reading takes a ton of time, especially if you want to read 70 books a year.

I’ll be straight with you: I would not be where I am today if I didn’t read as much as I do.

I originally started as a contractor at KISSmetrics, working on blog posts and support help videos. Two years later, I was the Head of Marketing and reported directly to the CEO.

As soon as I figured out how to perform at a high level with my current level of responsibilities, I asked to take on bigger challenges. Basically, I kept asking to get thrown into the deep-end. Then when I learned to tread water, I found the next storm to get thrown into.

  • Once I got comfortable with blog posts and product videos, I jumped into webinars. We had only done one webinar before I started. My webinar system went on to become our second-largest source of leads.
  • Our email system was in shambles. So I rebuilt the entire thing on a marketing automation tool and integrated it into the workflows of our Sales team. This set the whole foundation for being about to scale our lead growth.
  • Then I took responsibility for our lead counts and conversion rates. I put together our A/B testing strategy and our lead gen campaigns. A year later, we had tripled our conversion rates and quadrupled our monthly lead count.
  • To top it off, I jumped into the Head of Marketing role. I started attending board and leadership meetings, set the Marketing strategy and budget for 2015, and doubled the size of the team. We kept hitting all of our monthly goals like clockwork.

Every 6-9 months, I was taking a huge jump in responbilities.

The only reason I survived that kind of pace is that I had done an immense amount of reading. Need to crank out webinars? I just finished 4 books on webinars and how to give great presentations, let’s do this! Now driving monthly lead counts and improving conversions? Sure thing, I’ve read every conversion optimization book on Amazon. Time to double the size of the Marketing team? Good thing I just finished 5 of the top books on hiring. When my responsibilities increased, I had already spent time studying from the best in the field.

Reading won’t turn you into a world-class expert. You’ll need years of in-the-trenches experience to be truly world-class. But reading at this volume will make sure you can hit the ground running. You’ll become a solid practitioner in weeks instead of years. That’s how you survive being thrown into the deep-end every couple of months.

I’ve got 8 principles that I use to keep my reading volume up. If you follow them, you’ll be able to take on huge jumps in responsibility and accelerate your career.

Anchor 30-60 Minutes of Daily Reading

This is the key to making all this work.

By carving out just 30-60 minutes every day, you’ll be surprised how many books you’ll start to go through.

If you’ve done any research on how to change habits (The Power of Habit and Self-Directed Behavior will get you up to speed), you know that building a new habit works best when you anchor it against some other trigger. When you hit that trigger, you’ll be prompted to do your new habit. Before long, it’ll be an automatic response that you don’t even think about.

My favorite anchor for this? Reading in bed as I fall to sleep. If I’ve had a crazy day and I’m completely exhausted, I might only get through a page or two before passing out. But if I have plenty of energy left over and have a great book in my hands, I’ll easily crush a third of a book in a couple of hours before finally falling to sleep. Most likely, 30 minutes and a chapter or two is more than enough to put me to sleep.

One important caveat here: don’t regularly sacrifice sleep for reading. Some people have a really hard time falling asleep while reading non-fiction. Too many ideas start jumping into your head and you get amped thinking about all the new possibilities. If this happens to you regularly, try some fiction before bed and find another time during the day to anchor your business reading.

Find your ideal trigger for 30-60 minutes of daily reading. If one doesn’t work after a week or so, try another. Here’s a few other options:

  • Right after breakfast before the rest of your day starts.
  • During your lunch break.
  • On the train or bus during your commute.
  • An audiobook while your drive to work.
  • First thing you do when you get home from work.
  • After brushing your teeth and before going to bed.
  • In bed as you fall asleep.

Buy Your Next Book Before You Finish Your Current Book

As soon as you get close to finishing your current book, make sure you’ve got another one sitting around for you to dive into next. There shouldn’t be any breaks in your new reading habit. Finishing a book and then binging on Netflix for a few weeks will really slow you down.

When you start to get to the end of your current book, grab one on Amazon or run by your local book store. It’s also a nice little incentive to keep you going and wrap up the current book. Finishing sooner also means starting the next one sooner.

Since I’m a compulsive reader, I’ve moved most of my reading to a Kindle which makes this a non-issue. As soon as I’m finished with my current book, I download the next book that I want instantly. Takes me 2 minutes tops.

What about buying a bunch at a time so you have a to-read pile?

If that works for you, go for it. Several years ago, I developed a habit of buying $200-300 worth of books at a time. Then I’d only read the first couple before I bought another batch, leaving me with a pile that I never seemed to get around to. So now I force myself to finish my current book before buying the next one.

Take Your Book With You Everywhere

That’s right, be that nerd that never goes anywhere without a book. Now when you have a few spare minutes waiting for your next meeting, the bus, or your friends to show up, you’ve got a book at your side. Just like how that daily 30-60 minutes adds up over a year, 10 minutes a few times throughout the day also adds up real fast.

I’ve always got my Kindle on me. Whenever I have some spare time, I grab it and start reading. It’s much more fulfilling that scrolling through my Facebook feed. Would you rather level up one of your business skills or see which of your friends just shared a BuzzFeed article? I don’t even have Facebook installed on my phone, I’m too busy reading.

Flights and commutes are also perfect for this. Don’t crank on that spreadsheet or deck, just relax and get some reading done. With nothing else to distract you, you’ll crush books. Everyone else frets about their flight getting delayed, I just use the extra time to finish a few more chapters. If my other work is important enough, I’ll find time to get it done. But by allocating travel time to reading, I’m consistently investing in my long-term instead of just running on the short-term project treadmill.

Keep a To-Read List

I have hundreds of books in my backlog, more than I could ever read. You should start your own to-read list which includes every book you’ve ever wanted to read. This way, you’ll never have to wonder what you should read next. As soon as you start to finish your current book, you’ve got a huge list of books you’ve already decided to read.

Some people will add books to their Amazon wishlist, I just use Evernote. Here’s a snippet of what mine looks like:

Lars Lofgren To-Read List

That goes on for hundreds, maybe thousands of books. When I’m looking for my next book, I scroll down and start popping titles into Amazon to jog my memory on what each book’s about. When I find the right one, I hit the purchase button.

But where do you get ideas for books in the first place?

There’s a number of people that I respect immensely in my field. If any of them recommends a book, I add it straight to my list. Even if it’s a topic I’m lukewarm on, any recommendation from them goes to the top.

I’ll also check out the reviews on Amazon whenever I see a book mentioned elsewhere. It might come up in some random blog post, a retweet scrolling through my feed, or a New York Times article. And if it looks solid, then it gets added.

Most importantly, pay close attention to books that get mentioned repeatedly in your field. For marketing, you’ll see Influence, Permission Marketing, and The 22 Immutable Laws of Marketing all the time. And deservedly so, they’re classics of the field. As soon as you start to get a vibe that a book is a classic in a field that you want to build mastery in, add it to the list.

Your list will start small but will quickly expand. Before you know it, you’ll have a longer to-read list than you could ever hope to finish.

Alternate Between Depth and Entertainment

Text books aren’t just for college kids. Some of the most valuable reads I’ve discovered were pretty hefty tombs. Even if you love to read, these things take some serious effort to get through. Diffusion of Innovations and The Social Animal are great examples. Full of insights and value but a slog to get through. There’s no quirky anecdotes, no fun tangents, and no narrative to speed things along. You’ll earn each and every page.

You’ll want to tackle these pillars of learning but don’t overdo it. After you’ve finished one, make sure to sprinkle in a few lighter reads. Something from Michael Lewis or Seth Godin will do the trick.

I also alternate between topics I love and topics I’m not ecstatic about but are still critical to rounding out my knowledge. For example, I’m not super passionate about data visualization but Show Me the Numbers gives a great foundation for building out graphs and tables in your work. Occasionally pick up a book that is critical to building out your expertise regardless of how excited you are to read it. Once you finish, jump back into some topics that you’re more excited about.

Keep switching up the book types and topics so that reading doesn’t feel too much like work. Don’t let yourself get burned out on it. A fiction or non-business book goes a long way to giving you some relief after conquering some musty tomb.

Cut Your Cable

I’ve never had cable and I never will.

Don’t get me wrong, I love movies and solid TV. But I demand control of my time and my schedule when it comes to entertainment. Having an all-you-can-eat stream of B-rate reality TV is just too much of a time suck. You’ll progress so much faster if you replace that time with reading.

Get a Netflix subscription, set aside $50 to buy anything else you want on Amazon or iTunes, and you’ll have more than enough to watch. You’ll even spend less than most cable subscriptions. And with HBO Now, there’s no reason to have cable.

Take the 2-3 hours that most people watch each day and replace it with reading. You’ll easily finish a book every week.

Don’t Speed Read

Could I double the number of books that I read each year by getting good at speed reading? Absolutely. Do I want to? Nope.

You see, I’m not reading for sheer volume. That misses the point. The goal here is mental preparation so I’m as qualified as possible even when I walk into completely novel situations. So my main priority is retention. The better I can incorporate what I’ve learned into my mental models, the more I can relay on instinct in any number of crazy situations.

This is why I read at a speed that’s about half of my max. A 50% pace means I can use the other 50% of my energy to think deeply about each concept as it comes up in the reading. Connecting the reading to my experiences and everything else I’ve already learned helps me retain each new principle.

Volume certainly helps as you continue to grow your own capabilities. But retention shouldn’t be scarified in the name of hitting some vanity metric on how many books you read. They’ll only help you if you actually integrate them into your own mental models.

Wrap Up

If you want to accelerate your career or your business, you’ll need to constantly be preparing for the next major challenge. Voracious reading is one of my trade secrets to stack the odds in my favor when a new opportunity comes along.

And here’s my 8 tips for reading 70 books a year:

  • Anchor 30-60 minutes of daily reading
  • Always have an unfinished book on hand
  • Buy your next book before you finish your current book
  • Take your book with you everywhere
  • Keep a to-read list
  • Alternate between depth and entertainment
  • Cut your cable
  • Don’t speed read

If you want to keep tabs on what I’m reading, I list every business book that I finish here.

{ 11 comments }

Growth graphs like this don’t come around too often:

slack-1year-Feb12-2015-dau

I’ve gotta hand it to Slack, they’re playing the PR game pretty well too. Headline after headline on Techcrunch and Techmeme keep telling us all how crazy the growth is over there. They’ve already built the brand to go with it.

But here’s the thing about rocketships: they either go further than you ever thought possible or they blow up in your face.

It all comes down to momentum. Keep it up and you’ll quickly dominate your market.

But once you start to lose momentum, the rocketship rips itself apart. Competitors catch up, market opportunities slip away, talent starts to leave, and growth stalls. It’s a nasty feedback loop that’ll do irrecoverable damage. Once that rocketship lifts off, you either keep accelerating growth or momentum slips away as you come crashing back down. Grow or die.

There’s no room for mistakes. But when driving growth, there’s 2 forces that will consistently try to bring you crashing back down.

1) The Counter-Intuitive Nature of Growth

At KISSmetrics, I launched a bunch of tests that didn’t make a single bit of sense when you first looked at them. And many of them were our biggest winners.

Here’s a good example. Spend any time learning about conversion optimization and you’ll come across advice telling you to simplify your funnel. Get rid of steps, get rid of form fields, make it easier. Makes sense right? Less effort means more people get to the end. In some cases, this is exactly what happens.

In other cases, you’ll grow faster by adding extra steps. Yup, make it harder to get to the end and more people finish the funnel. This is exactly what happened during one of our homepage tests. We added an entire extra step to our signup flow and instantly bumped our conversions to signups by 59.4%.

Here’s the control:

Home Page Ouath Test Control

Here’s the variant:

Home Page Oauth Test Variant

In this case, the variant dropped people straight into a Google Oauth flow. But we didn’t get rid of our signup form since we still needed people to fill out the lead info for our Sales team.

Number of steps on the control:

  1. Homepage
  2. First part of signup form
  3. Second part of signup form (appeared as soon as you finished the first 3 fields)

Number of steps on the variant:

  1. Homepage
  2. Google account select
  3. Google oauth verification
  4. Signup form completion

You could say the minimalist design on that variant helped give us a winner which is true. But we saw this “add extra steps to get more conversions” across multiple tests. Works like magic on homepages, sign up forms, and webinar registrations. It’s one of my go-to growth hacks at this point.

Counter-intuitive results come up so often that it’s pretty difficult to call winners before you launch a test. At KISSmetrics, we had some of the best conversion experts in the industry like Hiten Shah and Neil Patel. Even with this world-class talent, we STILL only found winners 33% of the time. That’s right, the majority of our tests FAILED.

We ran tests that I would have put good money on. And guess what? They didn’t move our metrics even a smidge. The majority of our tests made absolutely zero impact on our growth.

It takes a LOT of testing to find wins like this. So accelerating growth isn’t a straight-forward process. You’ll run into plenty of dead-ends and rabbit holes as you learn which levers truly drive growth for your company.

2) You’ll Get Blind-Sided by False Positives

Fair enough, growth is counter-intuitive. Let’s just A/B test a bunch of stuff, wait for 95% statistical significance, and then launch the winners. Problem solved!

Not so fast…

That 95% statistical significance that you’ve placed so much faith in? It’s got a major flaw.

A/B tests will lead you astray if you’re not careful. In fact, they’re a lot riskier than most people realize and are riddled with false positives. Unless you do it right, your conversions will start bouncing up and down as bad variants get launched accidentally.

Too much variance in the system and too many false positives means you’re putting a magical growth rate at serious risk. Slack wants to get as much risk off the table while still chasing big wins. And the normal approach to statistical significance doesn’t cut it. 95% statistical significance launches too many false positives that will drag down conversions and slow momentum.

Let’s take a step back. 95% statistical significance comes from scientific research and is widely accepted as the standard for determining whether or not two different data sets are just random noise. But here’s what gets missed: 95% statistical significance only works if you’ve done several other key steps ahead of time. First you need to determine the minimum percentage improvement that you want to detect. Like a 10% change for example. THEN you need to calculate the sample size that you need for your experiment. The results don’t mean anything until you hit that minimum sample size.

Want to know how many acquisition folks calculate the sample size based on the minimum difference in conversions that they want to detect? Zero. I’ve never heard of a single person doing this. Let me know if you do, I’ll give you a gold star.

But I don’t really blame anyone for not doing all this extra work. It’s a pain. There’s already a ton of steps needed in order to launch any given A/B test. Hypothesis prioritization, estimating impact on your funnel, copy, wireframes, design, front-end and back-end engineering, tracking, making the final call, and documentation. No one’s particularly excited about adding a bunch of hard-core statical steps to the workflow. This also bumps the required sample sizes for conversions into the thousands. Probably not a major problem for Slack at this point but it will dramatically slow the number of tests that they can launch. Finding those big wins is a quantity game. If you want higher conversions and faster viral loops, it’s all about finding ways to run more tests.

When you’re in Slack’s position, the absolute last thing you want to do is expose yourself to any unnecessary variance in your funnel and viral loops. Every single change needs to accelerate the funnel like clockwork. There’s too much at stake if any momentum is lost at this point. So is there another option other than doing all that heavy duty stats work for each A/B test? Yes there is.

The Key to Keeping Rocketships Flying

Right now, the team at Slack needs to be focusing on one thing: how not to lose.

Co-founder Stewart Butterfield mentioned that he’s not sure where the growth is coming from. This is a dangerous spot to be in. As they start to dive into their funnel, there’s a serious risk of launching bad winners from false positives. They’ll need every last bit of momentum if they want to avoid plateauing early.

As it turns out, there is a growth strategy that takes these A/B testing risks off the table. It’s disciplined, it’s methodical, and it finds the big wins without exposing you to the normal volatility of A/B testing. I used it at KISSmetrics to grow our monthly signups by over 267% in one year.

Here’s the key: bump your A/B decision requirement to 99% statistical significance. Don’t launch a variant unless you hit 99%. If you’re at 98.9% or less, keep the control. And run everything you can through an A/B test.

Dead serious, the control reigns unless you hit 99% statistical significance. You’ll be able to keep chasing counter-intuitive big wins while protecting your momentum.

At KISSmetrics, we actually did a bunch of Monte Carlo simulations to compare different A/B Testing strategies over time.

I’ve posted the results from 3 different strategies below. Basically, the more area under the curve means the more conversions you earned. Each dot represents a test that looked like a winner. You’ll notice that many dots actually bring the conversions down. This comes from false positives and not being rigorous enough from your A/B testing.

Here’s what you get if you use the scientific researcher strategy:

conversion-rate-vs-observations-600

Not much variance in this system. Winners are almost always real winners.

Here’s your regular sloppy 95% statistical significance strategy that makes changes as early as 500 people in the test:

the-impatient-marketer-600

Conversions bounce around quite a bit. False wins come up often which means that if you sit on particular variation for long, it will drag those conversions down and slow growth. There goes your momentum.

Now let’s look at the 99% strategy that waits for at least 2000 people in the test for a decent sample size:

the-realist-conversion-rates-600

Still a chance to pick up false winners here but a lot less variance than 95%. Let’s quantify all 3 strategies real quick by calculating the area under the curve. Then we’ll be able to compare them instead of just eye-balling the simulations.

  • Statistical researcher = 67759
  • 95% statistical significance = 60532
  • 99% statistical significance = 67896

Bottom line: the 99% strategy performs just as well as the scientific researcher and a lot better than the sloppy 95%. It’s also easy enough for any team to implement without having to do the extra stats work.

The 99% rule is my main A/B testing rule but here are all of them:

  • Control stands unless the variant hits a lift at 99% statistical significance.
  • Run the test for at least a week to get a full business cycle.
  • Get 2,000 people through the test so you have at least a halfway decent sample size.
  • If the test looks like a loser or has an expected lift of less than 10%, kill it and move on to the next test. Only chase single digit wins if you have a funnel with crazy volume.

I used these rules to double and triple multiple steps of the KISSmetrics funnel. They reduce the risk of damaging the funnel to the bare minimum, accelerate the learning of your team, and uncover the biggest wins. That’s how you keep your growth momentum.

Embedding A/B Tests Into the Slack Culture

I can give you the rules for how to run a growth program. But you know what? It won’t get you very far unless you instill A/B tests into the fabric of the company. The Slack team needs to pulse with A/B testing. Even the recruiters and support folks need to get excited about this stuff.

This is actually where I failed at KISSmetrics. Our Growth and Marketing teams understood A/B testing and our entire philosophy behind it. We cranked day in and day out. It was our magic sauce.

But the rest of the team? Did Sales or Support ever get it? Nope. Which meant I spent too much time fighting for the methodology instead of working on new tests. If I had spent more time bringing other teams into the fold from the beginning, who knows how much further we could have gone.

If I was at Slack, one of my main priorities would be to instill A/B testing into every single person at the company. Here’s a few ideas on how I’d pull that off:

  • Before each test, I’d show the entire company what we’re about to test. Then have everyone vote or bet on the winners. Get the whole company to put some skin in the game. Everyone will get a feel for how to accelerate growth consistently.
  • Weekly A/B testing review. Make sure at least one person from each team is there. Go through all the active A/B tests, current results, which one’s finished, final decisions, and what you learned from them. The real magic of A/B testing comes from what you’re learning on each test so spread these lessons far and wide.
  • Do monthly A/B testing growth talks internally. Include the rules for testing, why you A/B test, the biggest wins, and have people predict old tests so they get a feel for how hard it is to predict ahead of time. Get all new hires into these. Very few people have been through the A/B test grind, you need to get everyone up to speed quickly.
  • Monthly brainstorm and review of all the current testing ideas in the pipe. Invite the whole company to these things. Always remember how hard it is to predict the big winners ahead of time, you want testing ideas coming at you from as many sources as possible.

Keep Driving The Momentum of That Slack Rocketship

I’m really hoping the team at Slack has already found ways to avoid all the pitfalls above. They’ve got something magical and it would be a shame to lose it.

To the women and gents at Slack:

  • Follow the data.
  • Get the launch tempo as high as possible for growth, you’ll need to run through an awful lots of ideas before you find the ones that truly make a difference.
  • Only make changes at 99% statistical significance.
  • Spread the A/B testing Koolaid far and wide.
  • Don’t settle. You’ve got the magic, do something amazing with it.

{ 8 comments }