7 Test, Learn, Adjust
Once you’ve developed the parameters and objectives of your test, it’s time to implement it in a way that helps you acquire the learning and leverage required to go further. It’s not enough to just conduct that first test—it’s likely the first of many iterations, tested rapidly. You need to build momentum to propel you on the journey toward your longer-term vision. It’s time to get entrepreneurial.
Communicate your Compelling Vision for Membership
Even with a green light from leadership, you must educate key people organization-wide to understand the vision and their roles in achieving it. Team members, partners, and eventual customers must be persuaded that this complex path, significant investment, and effort is justified by the outcome. Success is most likely when everyone embraces the strategy.
Articulate the compelling vision. Add images to illustrate it and, if you have them, work with your data scientists, market research, finance, and operations teams to buttress the vision with numbers. The plan won’t be exactly right—it’s unlikely even if your business is wildly successful. But it’s critical to make nebulous ideas more visceral and verify that the logic holds when all the pieces are assembled. When your colleagues can see and understand the vision of where you’re headed, they are more likely to get on board.
Realistic expectations and effective communication are essential to excite the team about the mission and motivate them for the possibly arduous journey ahead. Brett Brewer, General Manager, Microsoft Office Growth, has actually developed his own courses and onboarding materials to educate his colleagues about the unique challenges and processes needed for success in the Office 365 subscription business.1 According to Brewer, a big part of why his team created a “Subscription 101” type of curriculum was a realization that functional departments like marketing, finance, and product development don’t operate the same way for software-as-a-service as they do for manufacturing. In other words, if you have subscription pricing, you need to think differently about all departments. In fact, Brewer points out that universities are now asking him and other business practitioners to guest lecture on these concepts. It seems many academic institutions have yet to catch up with modern business models because the world of subscriptions is evolving and growing so rapidly.
Case Study: FabFitFun
FabFitFun (FFF) launched in 2010 as a newsletter. Today, it’s a $200 million lifestyle brand complete with major financial backing and global strategy. It offers members a quarterly subscription box, access to a vibrant user community, and original content delivered by newsletter, magazine, blog, and its newest live TV channel, FFFTV.
FabFitFun launched with a consistent mission and built out the robust membership business it is today. The mission—FFF’s forever promise—is described on its magazine site: “FFF aims to motivate, inspire and excite readers with a slew of great, healthy resources for every kind of woman—from the new college student to the new mom.”
Begin by describing what you’d do if you could invest in your ideal subscription model immediately, without any constraints. For FabFitFun, that vision includes inspiring everyone from college student to new mom. What would it look like if it truly provided great healthy resources for every kind of women, including older moms (like myself) or women whose education ended with high school?
FFF might have envisioned these features at the outset:
• Content (text, image, video, audio)
• Education (classes, trainings)
• Live events
• Global community of women inspiring women with user-generated content
• Coaching by professionals and/or by peers—everything from personalized styling to fitness to relationship advice
Obviously, FFF didn’t implement this full vision initially, and you don’t have to either. Few companies know enough and have the resources and confidence to do everything at once. Remember, Amazon started as a bookseller, and Netflix only carried other people’s movies and TV shows.
Start Small and Keep Adding—Don’t Do It All at Once*
When FabFitFun launched in 2010, founder and chief editor Katie Kitchens had many questions. Her list of learning and leverage objectives was long.
She needed to see if her promise appealed to her target audience, to learn what it would cost to attract them, and what it would take to engage and retain them. She needed to prove that the audience would spend money with her brand, although it wasn’t yet clear whether that would be content or products. If she wanted to sell physical products, she had to figure out which products, how to procure them, and whether they’d be exclusively hers or a curated collection.
Despite FFF’s bold “forever promise,” its first offering was modest—a newsletter. Even today, with a global footprint, television, branded products, and nearly a million members on its Facebook page,2 the company has infinite room to continue traveling toward full realization of its lofty goal. Step by step, it layers in greater value, stronger infrastructure, and new functionality to actualize the vision first defined in 2010. It continues to add benefits for its members, augmenting “how” it delivers on its forever promise.
The Electronic Arts (EA) story from Chapter 6 provides another example of this progression. After conducting successful small tests to validate its minimual viable product (MVP), EA initiated a second phase of experimentation, which entailed testing hypotheses for future scenarios to answer pressing, long-term questions:
1. Would subscription pricing cannibalize their business? Would the subscribers be those who were already buying multiple games at $60 every year, or would the offer attract casual players and entice them to spend more time and money?
2. How would console manufacturers and retailers react to a strategic pivot?
3. How would game players react if they got access to new releases (“frontline games”) in the subscription?
4. Would different regions of the world respond differently to a subscription offer?3
Phase two took more than a year because the team needed to observe player behavior over time to evaluate the impact of the subscriptions on game play patterns. The phase two offer included most of the company’s games, new releases as well as the catalog titles, at a low price of $14.99 per month or $99.99 per year.
Again, the test was limited to players using PCs, a small percent of overall customers. The primary goal was to increase engagement among, and deepen the relationship with, existing EA customers. This effort was not oriented to bring in new customers, although the team believed that new players would try the service because of its overwhelming value. The company believed that the more deeply subscribers engaged, the more value they’d get from their game time, naturally resulting in happier subscribers and greater revenue.
Learn from Early Failures
Many organizations abandon their forever journey early on because of a perceived or real failure.
Most of these failures come from a poorly designed test. If you don’t define the goals of the test (revenue vs. signups vs. engagement, for example), how will you determine if the test “worked” or not?
I’ve seen companies “test” subscription by making their existing catalog of content available at a monthly price. When people binge the first month and then cancel (what I call a “smash and grab”), the company views the test as a failure. Is it a failure? Not if the question was “How will people consume our content if they have unlimited access and pay by the month?”
Some companies perceive general success when what they’ve proven is narrow. For example, you may prove that people will use something you give away. That’s engagement. It doesn’t prove willingness to pay. Those are two different things!
Even if your experiment is well defined, failure can illuminate remedial actions to get you on a right path, one you might not have expected. Try to pinpoint whether your failure is due to a communication problem, a product problem, or an execution problem.
TABLE 7.1 Three Common Causes of Early Failures
• A communication problem occurs when the customer doesn’t understand what you’re offering or how to maximize its value. Or, the initial offer triggers people to sign up, but attracts people who might not find value in the subscription. You might not be reaching the right people or explaining the value in a clear and compelling way.
• A product problem is when people understand your promise, but don’t feel you’ve delivered on it. For example, you promise a monthly box of great outfits, but most of the outfits don’t fit or look good on the subscribers. Or your app is supposed to let marketers manage their own website, but it requires coding—a skill most marketers lack. Or perhaps you made a pricing error that needs to be adjusted. Fixing product is more expensive than improving communication.
• Execution problems result when the product experience is clunky and has feature gaps. I worked with a streaming content company with unreliable streaming technology. People were losing connection in the middle of the soccer championship. Great promise (live streaming of games). Great product (an app on your phone that streams content). But the thing just didn’t work! This happens more often than you might think. In an era of ready-fire-aim and MVP, sometimes the offer is less than what is minimally necessary. In these cases, the organization might fix the execution problem and then retest. That’s why it’s important to bound the size of any trial so you can test in additional markets if the first one fails.
Incubate and Learn
This incubation and test period differs for every business. Some organizations use this early phase to define their ideal forever promise, develop best practices for operations, and optimize pricing. Other companies take different approaches. Some launch in a single market. Some start with pricing much higher than they expect to use at scale, so they don’t risk cannibalizing an existing business. Some launch with a single feature, planning to add later—like Amazon did with books. Who remembers that Amazon launched its “all products for all people” e-commerce platform with a single product category?
Nike Adventure Club—Nike Experiments with Direct-to-Consumer New Business Models
When I told Dave Cobban, general manager and cofounder of Nike Adventure Club, that I wanted to include their story in this book because I wanted to ensure there were examples of subscription box models, he quickly corrected me. “We’re not a subscription box company—we’re giving parents the ability to access brand-new kids’ shoes in easy and convenient ways. We are more akin to a leasing than transaction model—and we’re recovering those shoes at end of life and either donating to foster kid programs or recycling through Nike Grind.”4
This “circular business” concept was initially tested in 2015 with runners who swap out old shoes for new ones after about 300 miles of use. Runners loved the idea, but parents among them reported: “This would be great for me, but even better for my kids.” Most parents hate the hassle of shopping for kids’ shoes every couple of months.
The kids-focused Nike Adventure Club (initially called EasyKicks) was launched in August 2016 through Nike’s Advanced Innovation Team.
Nike’s Intrapreneurial System Uses a Launch-Leverage-Lead Model
After researching various incubator models, Nike developed a model similar to Launch/Scale/Lead:
Launch. With early “angel” funding, Cobban validated the business problem, solution, and business model.
Scale. Nike got first right of refusal to “buy” the startup, investing financial and other resources as well as contributing brand muscle to scale to profitability.
Lead. Once the business achieved goals established by one of Nike’s business leaders, it was integrated into the organization.
Launch: Proving the Adventure Club Model
The original EasyKicks model was simple. For $20 a month, parents could swap their kids’ sneakers as frequently as desired. Used sneakers were returned for donation or recycling. Nike kids’ shoes average $60, so with a three-month cycle, revenue would remain the same, with a hoped-for deepening relationship and increased share of shoe purchases.
The EasyKicks team soon realized they had two candidates for their “best customer”:
• Convenience moms, who swap shoes every 60 to 90 days
• Proud moms, who swap shoes frequently and act like superusers: tweeting, Instagramming, and making lots of referrals
Convenience moms were the anticipated audience. Proud moms were a surprise, with a totally different forever promise, revolving around fashion.
The startup team made major choices. They changed the pricing structure, which is difficult to do. In 2018, they conducted a multivariate test, experimenting in market with four pricing options at once. (Cobban tells me that Rent the Runway and ClassPass do similar testing.) But they encountered some unexpected challenges from which they were quickly able to learn. The internet is a transparent marketplace; people quickly discover if there are multiple concurrent offers. EasyKicks received many customer support contacts from people saying, “Hold on, you offered my friend a better deal.” But the team learned a huge amount in a short time.
The pricing model that “won” the multivariate test had three tiers, but it still included the unlimited offer:
• Order new shoes every 90 days for $20 a month (base offer)
• Order new shoes every 60 days for $25 a month
• Unlimited swaps for $30 a month
Acquiring customers with this new pricing taught the team two key things:
1. People like choices of plans; it anchors them in the deal they want. Experimentation demonstrated that three choices are optimal.
2. Unlimited plans are cost prohibitive. As MoviePass has learned, it’s impossible to predict how often people will use the service, so costs keep rising over time.
In April 2019, the EasyKicks team pivoted again toward more limited “fixed offer” plans and incorporated the ability to pause and upgrade. This facilitates customer flexibility while ensuring fixed service cost for Nike.
They tested this new formation while still branded EasyKicks to verify there wasn’t false data associated with the inclusion of the better-known, already trusted Nike Swoosh.
Scale: Use the Power of Nike to Reach All Kids
In July 2019, EasyKicks rebranded as Nike Adventure Club. At this point, the team’s scale-up strategy evolved to include a deeper investment in relationships with children as well as moms. The informal motto became “recruit the mum, retain the kid.”
Shoe shipments started including adventure challenges, an adventure journal, and adventure cards with instructions for playing “old school” games like four square or wall ball. Additionally, the company began campaigns promoting the creation of user generated content (UGC), such as images of outdoor games unique to specific locales suitable for posting on the Nike site.
Lead: Continue to Experiment with Membership to Sustain a Forever Transaction
When Adventure Club reaches a predetermined target revenue and profitability, it will integrate into Nike’s North America business with global expansion likely thereafter.
Contemplating how the model might expand across the organization is fascinating. Cobban notes that the membership model still makes great sense for the original intended audience—runners. He believes a significant proportion of Nike’s future business could eventually be driven by the Membership Economy. For Nike to have ongoing connections to its customers without dependence on third-party retailers would be a huge shift in culture and strategy. Cobban thinks membership could unlock that potential.
The Risks of Not Maintaining Momentum
Nike has been disciplined about a phased but aggressive approach to experimentation. Maintaining momentum can be challenging, though. Many organizations lack the focus and energy to quickly build on successes and adjust for early failure. Moving too slowly can result in missing the opportunity entirely. Here are some snapshots of companies that didn’t leap in with both feet:
• The hardware manufacturer that had a small success in services revenue but didn’t apply additional resources. Instead it focused its energy on promotions to encourage existing customers, in a saturated market, to upgrade the hardware, which had a bigger short-term top-line revenue impact. Since hardware upgrades are optional, transactional (one time), and not terribly profitable, the company missed the bigger opportunity to fan the flames of its nascent recurring revenue services business.
• The enterprise software company that started building, in 2008, a “light, SaaS-structured” offering for smaller, nimbler prospects. Its product leverages the cloud and is optimized for subscription, but it was never rebuilt from the ground up. Consequently, it lacks capacity to track behavioral data and allow subscribers to configure their implementation. Meanwhile, the sales team continues to sell the on-premise solution—which 80 percent of them prefer to do because they earn higher commissions on those sales. The organization regressed from ahead of the curve to behind it and now risks missing the wave altogether.
• The newspaper that offers a subscription but is “hedging its bets” through continued dependence on a traditional advertising model. The two models directly conflict; the old ad model requires a large audience that pays only with their eyeballs. The subscribers want relevant, useful content. To maximize views, a paper might use a picture of Kim Kardashian, but content worth paying for is more likely to be a deep analysis of the bond market.
• The consumer packaged goods company offering a replenishment-based subscription model, delivered in ugly, hard-to-open boxes, with complex pricing, no relevant educational materials, lousy support, and terrible returns. This company missed the opportunity to make the experience better than other channels, by personalizing it or creating a deeper connection, or at least making it fun and delightful to open.
Subscription pricing in a vacuum won’t get you to a forever transaction. In Part Two, we’ll talk about building the organizational infrastructure to move fast, accelerating and turbocharging your growth.
Keep the Momentum
As your team analyzes the trial balloons, it should articulate what constitutes success and think beyond first steps. If things look good, what’s next? You may take progressively bigger steps before the entire transformation is complete or you publicly announce the strategic change.
Don’t revel too long in the success of your small experiment before pushing for the next giant step. Know before you launch your test that if it’s successful you will quickly be able to scale. There’s nothing more frustrating than a “successful” test that won’t scale, or one that lacks momentum.
Most organizations assume the rest of the world isn’t going to change as fast as their own organization will. Remember: everyone’s trying to ride that same current. If you’re pushing hard to build this new model without identifying why it’s special for the people you serve, you’re not going to stand out. Nearly every company I’ve spoken to in recent years is experimenting with a forever transaction and a recurring revenue relationship with its best customers. That alone is difficult, but you must also determine how that forever transaction can be expressed broadly through a more fully developed offer, more systemic delivery, or a larger audience.
Before launching your initial experiment, you articulated a compelling vision of where your organization can be “someday.” This forever vision probably includes some combination of community network, subscription pricing, direct-to-consumer connection, and sophisticated content. You’ll probably require a new technology platform to support deeper customer engagement, more sophisticated pricing strategies, and better analytics. All of this happens before you realize the benefits of additional revenue sources.
If you didn’t flesh out this vision earlier, do so now. You need to have a perspective of the future and how to anticipate transformation; you must envision a strategy beyond “keeping up.” How will your forever transaction provide a beachhead for your next phase of organizational growth? Just doing “the basics” takes significant investment.
Leadership and colleagues across the organization may want to slow your progress. They may put roadblocks in your way. You may be the only person in the organization scanning the horizon and envisioning how this transformation can get you there. Clarify the strategic journey and make it compelling for your colleagues. Help them understand the payoff, but also the high cost of moving slowly.
What to Do Next
• Develop a specific set of believable hypotheses that you are trying to prove or disprove before launching broadly, both to feel confident about your plan and to build support from other constituents (partners, colleagues, customers).
• Determine which hypotheses you can answer from the early tests and which you won’t.
• For hypotheses that you cannot answer with your trial, brainstorm ways to research and approximate your answers.
• Make sure you’re testing the most important questions first. Design future tests when you have had a chance to absorb the learning.
• Create a plan for your first few steps so you can retain focus and momentum as you conduct your first tests. You don’t want to lose focus as you wait for the results to come in.