Croll lean analytics workshop (3h) - lean ux nyc april 2014
How to use data to build a better business faster. Based on the book Lean Analytics, this presentation looks at startup metrics and offers a framework for deliberate growth and iterative improvement of a new business. It also includes examples from larger organizations trying to change from within.
Lean Analytics Lean UX NYC April, 2014 @acroll Some housekeeping. Dont sell what you can make. Make what you can sell. Kevin Costner is a lousy entrepreneur. The core of Lean is iteration. Most startups dont know what theyll be when they grow up. Hotmail was a database company Flickr was going to be an MMO Twitter was a podcasting company Autodesk made desktop automation Paypal rst built for Palmpilots Freshbooks was invoicing for a web design rm Wikipedia was to be written by experts only Mitel was a lawnmower company Unfortunately, were all liars. Everyones idea is the best right? People love this part! (but thats not always a good thing) This is where things fall apart. No data, no learning. Analytics can help. Analytics is the measurement of movement towards your business goals. In a startup, the purpose of analytics is to iterate to product/market t before the money runs out. I have two kids. At least one of them is a girl. What are the chances the other is a boy? BB BG GB GG 2 of 3 (66%) are boys. GB GG BG Some fundamentals. A good metric is: Understandable If youre busy explaining the data, you wont be busy acting on it. Comparative Comparison is context. A ratio or rate The only way to measure change and roll up the tension between two metrics (MPH) Behavior changing If youre busy explaining the data, you wont be busy acting on it. The simplest rule bad metric. If a metric wont change how you behave, its a h"p://www.ickr.com/photos/circasassy/7858155676/ Metrics help you know yourself. Acquisition Hybrid Loyalty 70% of retailers 20% of retailers 10% of retailers You are just like Customers that buy >1x in 90d Once 2-2.5 per year >2.5 per year Your customers will buy from you Then you are in this mode 1-15% 15-30% >30% Low acquisition cost, high checkout Increasing return rates, market share Loyalty, selection, inventory size Focus on (Thanks to Kevin Hillstrom for this.) Qualitative Unstructured, anecdotal, revealing, hard to aggregate, often too positive & reassuring. Warm and fuzzy. Quantitative Numbers and stats. Hard facts, less insight, easier to analyze; often sour and disappointing. Cold and hard. Exploratory Speculative. Tries to nd unexpected or interesting insights. Source of unfair advantages. Cool. Reporting Predictable. Keeps you abreast of the normal, day-to-day operations. Can be managed by exception. Necessary. Rumsfeld on Analytics (Or rather, Avinash Kaushik channeling Rumsfeld) Things we know dont know we know Are facts which may be wrong and should be checked against data. we dont know Are questions we can answer by reporting, which we should baseline & automate. we know Are intuition which we should quantify and teach to improve effectiveness, efciency. we dont know Are exploration which is where unfair advantage and interesting epiphanies live. MayAprMarFeb Slicing and dicing data Jan 0 5,000 Activeusers Cohort: Comparison of similar groups along a timeline. (this is the April cohort) A/B test: Changing one thing (i.e. color) and measuring the result (i.e. revenue.) Multivariate analysis Changing several things at once to see which correlates with a result. Segment: Cross-sectional comparison of all people divided by some attribute (age, gender, etc.) Which of these two companies is doing better? January February March April May Rev/customer $5.00 $4.50 $4.33 $4.25 $4.50 Is this company growing or stagnating? Cohort 1 2 3 4 5 January February March April May $5 $3 $2 $1 $0.5 $6 $4 $2 $1 $7 $6 $5 $8 $7 $9 How about this one? Cohort 1 2 3 4 5 January February March April May Averages $5 $3 $2 $1 $0.5 $6 $4 $2 $1 $7 $6 $5 $8 $7 $9 $7 $5 $3 $1 $0.5 Look at the same data in cohorts Lagging Historical. Shows you how youre doing; reports the news. Example: sales. Explaining the past. Leading Forward-looking. Number today that predicts tomorrow; reports the news. Example: pipeline. Predicting the future. A Facebook user reaching 7 friends within 10 days of signing up (Chamath Palihapitiya) If someone comes back to Zynga a day after signing up for a game, theyll probably become an engaged, paying user (Nabeel Hyatt) A Dropbox user who puts at least one le in one folder on one device (ChenLi Wang) Twitter user following a certain number of people, and a certain percentage of those people following the user back (Josh Elman) A LinkedIn user getting to X connections in Y days (Elliot Schmukler) Some examples (From the 2012 Growth Hacking conference. http://growthhackersconference.com/) Which means its time to talk about correlation. 1 10 100 1000 10000 Ice cream consumption Drownings Jan Feb Mar Apr May Jun Jul Aug Sept Oct Nov Dec Correlated Two variables that are related (but may be dependent on something else.) Ice cream & drowning. Causal An independent variable that directly impacts a dependent one. Summertime & drowning. A leading, causal metric is a superpower. h"p://www.ickr.com/photos/bloke_with_camera/401812833/sizes/o/in/photostream/ Growth hacking, demystied. Find correlation Test causality Optimize the causal factor Pick a metric to change Is social action a leading indicator of donation? http://blog.justgiving.com/nine-reasons-why-social-and-mobile-are-the-future-of-fundraising/ Is mobile use? http://blog.justgiving.com/nine-reasons-why-social-and-mobile-are-the-future-of-fundraising/ Why is Nigerian spam so badly written? Aunshul Rege of Rutgers University, USA in 2009 Experienced scammers expect a strike rate of 1 or 2 replies per 1,000 messages emailed; they expect to land 2 or 3 Mugu (fools) each week. One scammer boasted When you get a reply its 70% sure youll get the money By sending an email that repels all but the most gullible, says [Microsoft Researcher Corman] Herley, the scammer gets the most promising marks to self-select, and tilts the true to false positive ratio in his favor. 1000 emails 1-2 responses 1 fool and their money, parted. Bad language (0.1% conversion) Gullible (70% conversion) 1000 emails 100 responses 1 fool and their money, parted. Good language (10% conversion) Not-gullible (.07% conversion) This would be horribly inecient since humans are involved. Turns out the word Nigeria is the best way to identify promising prospects. Nigerian spammers really understand their target market. They see past vanity metrics. The Lean Analytics framework. Erics three engines of growth Virality Make people invite friends. How many they tell, how fast they tell them. Price Spend money to get customers. Customers are worth more than they cost. Stickiness Keep people coming back. Approach Get customers faster than you lose them. Math that matters Daves Pirate Metrics AARRR Acquisition How do your users become aware of you? SEO, SEM, widgets, email, PR, campaigns, blogs ... Activation Do drive-by visitors subscribe, use, etc? Features, design, tone, compensation, afrmation ... Retention Does a one-time user become engaged? Notications, alerts, reminders, emails, updates... Revenue Do you make money from user activity? Transactions, clicks, subscriptions, DLC, analytics... Referral Do users promote your product? Email, widgets, campaigns, likes, RTs, afliates... Stage EMPATHY Ive found a real, poorly-met need that a reachable market faces. STICKINESS Ive gured out how to solve the problem in a way they will keep using and pay for. VIRALITY Ive found ways to get them to tell their friends, either intrinsically or through incentives. REVENUE The users and features fuel growth organically and articially. SCALE Ive found a sustainable, scalable business with the right margins in a healthy ecosystem. Gate Thevestages Empathy stage: Localmind hacks Twitter Needed to nd out if a core assumptionstrangers answering questionswas valid. Ran Twitter experiment instead of writing code Asked senders of geolocated Tweets from Times Square random questions; counted response rate Conclusion: high enough to proceed Stickiness stage: qidiq streamlines invites Survey owner adds recipient to group Survey owner asks question Recipient reads survey question Recipient responds to question Recipient sees survey results (Later, if needed) Recipient visits site; no password! Recipient does password recovery One-time link sent to email Recipient creates password Recipient can edit prole, etc. Survey owner adds recipient to group Survey owner asks question Recipient gets invite Recipient reads survey question Recipient responds to question Recipient installs mobile app Recipient creates account, prole Recipient sees survey results Recipient can edit prole, etc. 10-25%RESPONSERATE 70-90%RESPONSERATE Six business model archetypes (Yours is probably a blend of these.) E-commerce SaaS (freemium?) Mobile app (gaming) Two sided marketplace Media User generated content (Which means eye charts like these.) Customer Acquisition Cost paid direct search wom inherent virality VISITOR Freemium/trial oer Enrollment User Disengaged User Cancel Freemium churn Engaged User Free user disengagement Reactivate Cancel Trial abandonment rate Invite Others Paying Customer Reactivation rate Paid conversion FORMER USERS User Lifetime Value Reactivate FORMER CUSTOMERS Customer Lifetime Value Viral coecient Viral rate Resolution Support data Account Cancelled Billing Info Exp. Paid Churn Rate Tiering Capacity Limit Upselling rate Upselling Disengaged DissatisedTrial Over Model + Stage = One Metric That Matters. One Metric That Matters. The business youre in E-Com SaaS Mobile 2-Sided Media UCG Empathy Stickiness Virality Revenue Scale Thestageyoureat Really? Just one? Yes, one. In a startup, focus is hard to achieve. Having only one metric addresses this problem. www.theeastsiderla.com Moz cuts down on metrics SaaS-based SEO toolkit in the scale stage. Focused on net adds. Was a marketing campaign successful? Were customer complaints lowered? Was a product upgrade valuable? Net adds up: Can we acquire more valuable customers? What product features can increase engagement? Can we improve customer support? Net adds at: Are the new customers not the right segment? Did a marketing campaign fail? Did a product upgrade fail somehow? Is customer support falling apart? Net adds down: Metrics are like squeeze toys. http://www.ickr.com/photos/connortarter/4791605202/ Empathy Stickiness Virality Revenue Scale E- commerce SaaS Media Mobile app User-gen content 2-sided market Interviews; qualitative results; quantitative scoring; surveys Loyalty, conversion CAC, shares, reactivation Transaction, CLV Afliates, white-label Engagement, churn Inherent virality, CAC Upselling, CAC, CLV API, magic #, mktplace Content, spam Invites, sharing Ads, donations Analytics, user data Inventory, listings SEM, sharing Transactions, commission Other verticals (Money from transactions) Downloads, churn, virality WoM, app ratings, CAC CLV, ARPDAU Spinoffs, publishers (Money from active users) Trafc, visits, returns Content virality, SEM CPE, afliate %, eyeballs Syndication, licenses (Money from ad clicks) Better: bit.ly/BigLeanTable What other metrics do you want to know about? Drawing some lines in the sand. A company loses a quarter of its customers every year. Is this good or bad? Not knowing what normal is makes you do stupid things. Baseline: 5-7% growth a week A good growth rate during YC is 5-7% a week, he says. If you can hit 10% a week you're doing exceptionally well. If you can only manage 1%, it's a sign you haven't yet gured out what you're doing. At revenue stage, measure growth in revenue. Before that, measure growth in active users. Paul Graham, Y Combinator Are there enough people who really care enough to sustain a 5% growth rate? Dont strive for a 5% growth at the expense of really understanding your customers and building a meaningful solution Once youre a pre-revenue startup at or near product/market t, you should have 5% growth of active users each week Once youre generating revenues, they should grow at 5% a week Baseline: 10% visitor engagement/day Fred Wilsons social ratios 30% of users/month use web or mobile app 10% of users/day use web or mobile app 1% of users/day use it concurrently Baseline: 2-5% monthly churn The best SaaS get 1.5% - 3% a month. They have multiple Ph.Ds on the job. Get below a 5% monthly churn rate before you know youve got a business thats ready to grow (Mark MacLeod) and around 2% before you really step on the gas (David Skok) Last-ditch appeals and reactivation can have a big impact. Facebooks dont leave reduces attrition by 7%. Who is worth more? Today A Lifetime: $200 Roberto Medri, Etsy B Lifetime: $200 Visits Baseline: Calculating customer lifetime 25% monthly churn 100/25=4 The average customer lasts 4 months 5% monthly churn 100/5=20 The average customer lasts 20 months 2% monthly churn 100/2=50 The average customer lasts 50 months Baseline: CAC under 1/3 of CLV CLV is wrong. CAC Is probably wrong, too. Time kills all plans: Itll take a long time to nd out whether your churn and revenue projections are right Cashow: Youre basically loaning the customer money between acquisition and CLV. It keeps you honest: Limiting yourself to a CAC of only a third of your CLV will forces you to verify costs sooner. Lifetime of 20 mo. $30/mo. per customer $600 CLV $200 CAC Now segment those users! 1/3 spend The Lean Analytics cycle Draw a new line Pivot or give up Try again Success! Did we move the needle? Measure the results Make changes in production Design a test Hypothesis With data: nd a commonality Without data: make a good guess Find a potential improvement Draw a linePick a KPI Do AirBnB hosts get more business if their property is professionally photographed? Gut instinct (hypothesis) Professional photography helps AirBnBs business Candidate solution (MVP) 20 eld photographers posing as employees Measure the results Compare photographed listings to a control group Make a decision Launch photography as a new feature for all hosts 5,000 shoots per month by February 2012 Hang on a second. Gut instinct (hypothesis) Professional photography helps AirBnBs business SRSLY? Draw a new line Pivot or give up Try again Success! Did we move the needle? Measure the results Make changes in production Design a test Hypothesis With data: nd a commonality Without data: make a good guess Find a potential improvement Draw a linePick a KPI Gee, those houses that do well look really nice. Maybe its the camera. Computer: What do all the highly rented houses have in common? Camera model. With data: nd a commonality Without data: make a good guess Landing page design A/B testing Cohort analysis General analytics URL shortening Funnel analytics Inuencer Marketing Publisher analytics SaaS analytics Gaming analytics User interaction Customer satisfaction KPI dashboardsUser segmentation User analytics Spying on users Some non-tech examples. I lied. Everyone is a tech company. http://www.ickr.com/photos/puuikibeach/4789015423 http://www.ickr.com/photos/elcapitanbsc/3936927326 Cost of experiments: down. Cost of attention: way up. Lets pick on restaurants for a while. A line in the sand Labor costs Gross revenue 30% 24% 20% Too costly? Just right Understaffed? = A leading indicator http://www.ickr.com/photos/avlxyz/4889656453http://www.ickr.com/photos/mysticcountry/3567440970 50 reservations at 5PM 250 covers that night (Varies by restaurant. McDonalds Fat Duck.) http://www.ickr.com/photos/southbeachcars/6892880699 Restaurant MVP Is tip amount a leading indicator of long- term revenue? Why does every table get the same menu? Is purple ink better? http://tippingresearch.com/uploads/managing_tips.pdf Growth hacking (is a word you should hate but will hear a lot about.) Growth hacking, demystied. Find correlation Test causality Optimize the causal factor Pick a metric to change Guerrilla marketing Data- driven learning Subversiveness GROWTH HACKING A Facebook user reaching 7 friends within 10 days of signing up (Chamath Palihapitiya) If someone comes back to Zynga a day after signing up for a game, theyll probably become an engaged, paying user (Nabeel Hyatt) A Dropbox user who puts at least one le in one folder on one device (ChenLi Wang) Twitter user following a certain number of people, and a certain percentage of those people following the user back (Josh Elman) A LinkedIn user getting to X connections in Y days (Elliot Schmukler) (from the 2012 Growth Hacking conference) (These are also great segments to analyze.) The leading indicator Growth hacking is simply what marketing should have been doing, but it fell in love with Don Draper and opinions along the way Optimize a factor you think is correlated with growth The growth hack AirBnB and Craigslist What if youre in a big organization? If a startup is an organization designed to search for a sustainable, repeatable business model, then an established company is an organization designed to perpetuate one. Intrapreneurs often have to use proxies Stage Startup metrics Intrapreneur metrics Empathy Customers interviewed (needs & solutions), assumptions quantied, TAM, monetization possibility Non-customers interviewed; assumptions quantied, constraints identied, TAM, disruption potential Stickiness Churn, engagement Support tickets, integration time, call center data, delays Virality Viral coecient, viral cycle time Net Promoter Score, referrals, case study willingness Revenue Attention, engagement Billable activity; signed LOIs; pilot programs; after-development protability Scale Automation Contribution, training costs, licensing The Lean Analytics lifecycle of an Intrapreneur Empathy Find problems; dont test demand. Skip the business case, do analytics Entitled, aggrieved customers Stickiness Know your real minimum based on expectations, regulations Hidden must haves, feature creep Virality Build inherent virality in from the start; attention is the new currency Luddites who dont understand sharing Revenue Consider the ecosystem, channels, and established agreements Channel conict, resistance, contracts Scale Hand the baton to others gracefully Hating what happens to your baby Beforehand Get buy-in Political fallout Some things that work. Frame it like a study Product creation is almost accidental. Unlike a VC or startup, when the initiative fails the organization still learns. http://www.flickr.com/photos/creative_tools/8544475139 Transformative isolation: Skunkworks Use outliers and missed searches to hunt for good ideas & adjacencies (Multi-billion-dollar hygiene product company) 1/8 men have an incontinence issue. 1/3 women do. When search results show a signicant number of men searching, this suggests the adjacent (male) market is underserved. Use data to create a taste for data Sitting on Billions of rows of transactional data David Boyle ran 1M online surveys Once the value was obvious to management, got license to dig. Focus on the desired behavior, not just the information. http://www.psychologytoday.com/blog/yes/ 200808/changing-minds-and-changing-towels 26% increase in towel re-use with an appeal to social norms; 33% increase when tied to the specic room. Energy Conservation Nudges and Environmentalist Ideology: Evidence from a Randomized Residential Electricity Field Experiment - Costa & Kahn 2011 The effectiveness of energy conservation nudges depends on an individuals political ideology ... Conservatives who learn that their consumption is less than their neighbors boomerang whereas liberals reduce their consumption. Understand hidden constraints That pencil story is a myth. Graphite is conductive and explosive. The Minimum Viable Product is Viable for a reason. Know what has to be built in-house SAP integration Employment law Run it as a consulting business rst. (Just dont get addicted to it. Your goal is to learn and overcome integration challenges and nd the 20% of features that 80% of the market will pay for.) When in doubt, collect data From tackling the FTA rate to visualizing the criminal justice supply chain. Everythings an excuse to experiment Find other ways to collect data; everything is an experiment. Dont just collect data, chase it. Some tools and traps Traction graphs Your business model The stage youre at Your one metric ... change often if youre doing it right. So how do you track that over time? Traction graphs Jan Feb Mar Apr May Jun Signups per day Conversion rate Churn rate Viral coefcient This axis changes for each metric Traction graphs Jan Feb Mar Apr May Jun Signups per day Conversion rate Churn rate Viral coefcient 0% Use vanity to get to meaningful metrics Your goal is to produce outcomes If the outcomes require action, and vanity motivates actors, use it But show how the vanity metric is a leading indicator of the real one x Web trac Revenue Activation Cart Size Conversion rate The three threes Three assumptions What big bets are you making? People will answer questions Organizers are frustrated with how to run conferences We'll make money from parents Amazon is reliable enough for our users. Three actions to take What are you doing to make these assumptions happen (or identify theyre wrong and change course?) Product enhancements Marketing strategies Three experiments to run Feature tests Continuous deployment A/B testing Customer survey The three threes Three assumptions Three actions to take Three experiments to run Monthly Weekly Daily Board, investors, founders Executive team Employees Strategy Tactics Execution The three threes Three assumptions Three actions to take Three experiments to run Get more people Increase answer % Test better questions Change the UI Test timings Questions from peers Many people will answer questions The problem-solution canvas CURRENT STATUS List key metrics youre tracking, where theyre at, and compare with last few weeks How are things trending? LAST WEEKS LESSONS LEARNED AND ACCOMPLISHMENTS) What did you learn last week? What was accomplished? On track: YES / NO? The Goal is to Learn The problem-solution canvas HYPOTHESIZED SOLUTIONS List possible solutions that youll start working on next week. Rank them. Why do you believe each solution will help you solve or complete solve the problem? METRICS / PROOF + GOALS Problem #1 (put name here) Metrics youll use to measure whether or not the solutions are doing what you hoped (solving the problem) List proof (qualitative) youll use as well Dene goals for the metric HYPOTHESIZED SOLUTIONS List possible solutions that youll start working on next week. Rank them. METRICS / PROOF + GOALS Metrics youll use to measure whether or not the solutions are doing what you hoped (solving the problem) Problem #2 (put name here) The most important gures that one needs for management are unknown or unknowable, but successful management must nevertheless take account of them. Lloyd S. Nelson Pic by Twodolla on Flickr. http://www.ickr.com/photos/twodolla/3168857844 ARCHIMEDES HAD TAKEN BATHS BEFORE. Once, a leader convinced others in the absence of data. Now, a leader knows what questions to ask. Alistair Croll email@example.com @acroll Ben Yoskovitz firstname.lastname@example.org @byosko