• Lean Analytics Lean Product & Lean UX Silicon Valley - July 2014 Hiten Shah [email protected] Download the slides: h p://kiss.ly/leanprod2014
  • Continuous Learning & Improvement
  • Don’t know where to start?
  • Wondering if you’re working on the right things?
  • Or, you just want to hear me babble!
  • HTTP://KISS.LY/LEANLYTICSBOOK
  • Lean Analytics helps you Optimize for Learning
  • Lean Analytics Cycle AVINASH KAUSHIK HTTP://KISS.LY/LEANAC
  • Metrics  →  Hypothesis  →  Experiment  →  Act
  • HTTP://HELLOBAR.COM
  • What is Hello Bar?
  •  Metrics    Figure  out  what  to  improve    Hypothesis    Make  an  educated  guess        Experiment    Test  your  guess    Act    Decide  what’s  next
  • Our funnel showed a low installation rate Hello Bar Funnel Metrics Steps Visitors Homepage URL Entries Created First Bar Completed Registration Installed Totals 15068 1868 716 688 398 % prev step 12.40% 38.33% 96.09% 57.85% % drop off -87.60% -61.67% -3.91% -42.15% % of visitors 100% 12.40% 4.75% 4.57% 2.64%
  • We  achieved  a  40%  increase  for  our   installa2on  rate  with  just  this  one   first  experiment. The Lean Analytics Cycle At Work
  • Original Hello Bar installation screen
  •  Metrics    Figure  out  what  to  improve    Hypothesis    Make  an  educated  guess        Experiment    Test  your  guess    Act    Decide  what’s  next
  • Find out why people don’t install Hello Bar
  • Didn’t Meet My Expectations
  • I Had Trouble Installing It
  • I Don’t Have A Website
  • Not Ready To Install
  • Other
  • Hello Bar Experiment Hypothesis If  more  installa2on  op2ons  are  given,   then  more  people  will  add  Hello  Bar   to  their  website  due  to  the  need  for  a   WordPress  plugin  and  ability  to  email   the  code  to  someone  else.
  •  Metrics    Figure  out  what  to  improve    Hypothesis    Make  an  educated  guess        Experiment    Test  your  guess    Act    Decide  what’s  next
  • Created a new flow to test
  • I can install code myself.
  • 41.67% responded with “WordPress Plugin”
  • 25% responded with “email my developer / web designer”
  • I need help.
  •  Metrics    Figure  out  what  to  improve    Hypothesis    Make  an  educated  guess        Experiment    Test  your  guess    Act    Decide  what’s  next
  • Full funnel metrics for the experiment Variation Steps Visitors Homepage URL Entries Created First Bar Completed Registration Installed Totals 7456 906 342 329 276 % prev step 12.15% 37.75% 96.20% 83.89% % drop off -87.85% -62.25% -3.80% -16.11% % of visitors 100% 12.15% 4.59% 4.41% 3.70% Control Steps Visitors Homepage URL Entries Created First Bar Completed Registration Installed Totals 7469 925 353 341 197 % prev step 12.38% 38.16% 96.60% 57.77% % drop off -87.62% -61.84% -3.40% -42.23% % of visitors 100% 12.38% 4.73% 4.57% 2.64%
  • Resulted in a 30% installation rate increase HTTP://ISVALID.ORG Experiment resulted in a 40% improvement
  • ACer  a  dozen  more  experiments,  our   installa2on  rate  is  now  hovering   between  5%  and  7%   That’s  an  addi;onal  89%  increase!!! The Lean Analytics Cycle At Work
  • Metrics  →  Hypothesis  →  Experiment  →  Act How You Can Put It Into Practice
  • Metrics   Figure  out  what  to  improve
  • “What gets measured, gets managed.” Peter Drucker
  • Baselines DATA DNA EBOOK HTTP://KISS.LY/OPTDATADNA
  • Funnels KISSMETRICS FUNNEL REPORT HTTP://KISS.LY/KMFUNNEL
  • SaaS Funnel KISSMETRICS FUNNEL REPORT HTTP://KISS.LY/KMFUNNEL
  • eCommerce Funnel KISSMETRICS FUNNEL REPORT HTTP://KISS.LY/KMFUNNEL
  • Revenue BUFFER BAREMETRICS HTTP://KISS.LY/BUFFERBARE
  • Spreadsheets and Calculations HELLOBAR HTTP://HELLOBAR.COM Steps Visitors Homepage URL Entries Created First Bar Completed Registration Installed Created Second Bar Totals 15068 1868 716 688 398 193 % prev step 12.40% 38.33% 96.09% 57.85% 48.49% % drop off -87.60% -61.67% -3.91% -42.15% -51.51% % of visitors 100% 12.40% 4.75% 4.57% 2.64% 1.28%
  • Hypothesis   Make  an  educated  guess
  • “A solid test hypothesis is an informed solution to a real problem – not an arbitrary guess. The more research and data you have to base your hypothesis on, the be er it will be.”
 — Michael Aagaard, h p://kiss.ly/abthypo
  • Research   Tools  &  Tac;cs
  • Customer Development HTTP://WWW.CUSTDEV.COM
  • Jobs To Be Done #JTBD HTTP://JOBSTOBEDONE.ORG
  • Usability Testing HTTP://USERTESTING.COM
  • Live Chat HTTP://OLARK.COM
  • Mouse Tracking HTTP://CRAZYEGG.COM
  • Long Form Surveying HTTP://SURVEYMONKEY.COM
  • On-Page Surveying HTTP://QUALAROO.COM
  • Personalized Messaging HTTP://INTERCOM.IO
  • Email a survey to people HTTP://SURVEYMONKEY.COM
  • Ask people why they canceled HTTP://SURVEYMONKEY.COM
  • Get feedback right when they cancel
  • Exit Surveys HTTP://QUALAROO.COM
  • Discover what’s missing HTTP://QUALAROO.COM
  • Find out what words people use to describe your product HTTP://QUALAROO.COM
  • Learn about people’s expectations HTTP://INTERCOM.IO
  • Learn what frustrates people HTTP://INTERCOM.IO
  • More actionable reading… QUALITATIVE DATA HTTP://KISS.LY/QUALABT
  • Experiment Hypothesis Formula DATA DNA EBOOK HTTP://KISS.LY/OPTDATADNA
  • Experiment   Test  your  guess
  • Only  1  out  of  5  tests  win. 4  out  of  5  tests   cost  you  money. $
  • 1)  Which  top  metric  are  we  trying  to  improve?   2)  What  can  you  anchor  your  metric  against?   3)  What  is  your  hypothesis  about  why  this  metric  is  at  x%?   4)  What  do  we  expect  from  this  experiment?   5)  What  was  the  result  and  what  did  we  learn  from  it? HTTP://KISS.LY/6QSAB Answer key questions for each experiment
  • Prioritize your experiments HTTP://KISS.LY/ABPIE
  • Don’t copy “best practices” blindly
  • Learn what works best for you
  • Act   Decide  what’s  next
  • Explore  the  results  and  make   data  informed  decisions. THE ANALYTICS THAT MATTER TO FACEBOOK HTTP://KISS.LY/FBINFORM
  • KISSmetrics A/B Testing Report KISSMETRICS A/B TESTING REPORT HTTP://KISS.LY/KMABTEST
  • We know that people love Google Analytics
  • KISSmetrics Experiment Hypothesis If  people  authen2cate  with  Google,   then  more  people  will  sign  up  due  to   their  exis2ng  familiarity  with  Google   Analy2cs  and  alignment  with  our   winning  headline.
  • KISSmetrics Homepage Test Control
  • KISSmetrics Homepage Test Variation
  • KISSmetrics Homepage Test Signup Conversion Rate KISSMETRICS A/B TESTING REPORT HTTP://KISS.LY/KMABTEST
  • KISSmetrics Homepage Test Received Data KISSMETRICS A/B TESTING REPORT HTTP://KISS.LY/KMABTEST
  • KISSmetrics Homepage New Control
  • Inspiration
  • KISSmetrics Homepage New Test Variation
  • KISSmetrics Homepage Test Signup Conversion Rate KISSMETRICS A/B TESTING REPORT HTTP://KISS.LY/KMABTEST
  • KISSmetrics Homepage Test Received Data KISSMETRICS A/B TESTING REPORT HTTP://KISS.LY/KMABTEST
  • Growth! HTTP://KISSMETRICS.COM
  •  Metrics    Figure  out  what  to  improve    Hypothesis    Make  an  educated  guess        Experiment    Test  your  guess    Act    Decide  what’s  next
  • Hiten Shah [email protected] @hnshah You will get all you want in life if you help enough other people get what they want. Zig Ziglar Download the slides: h p://kiss.ly/leanprod2014
Please download to view
All materials on our website are shared by users. If you have any questions about copyright issues, please report us to resolve them. We are always happy to assist you.
...

Lean Analytics Cycle

by hiten-shah

on

Report

Download: 0

Comment: 0

33,247

views

Comments

Description

 
Download Lean Analytics Cycle

Transcript

  • Lean Analytics Lean Product & Lean UX Silicon Valley - July 2014 Hiten Shah [email protected] Download the slides: h p://kiss.ly/leanprod2014
  • Continuous Learning & Improvement
  • Don’t know where to start?
  • Wondering if you’re working on the right things?
  • Or, you just want to hear me babble!
  • HTTP://KISS.LY/LEANLYTICSBOOK
  • Lean Analytics helps you Optimize for Learning
  • Lean Analytics Cycle AVINASH KAUSHIK HTTP://KISS.LY/LEANAC
  • Metrics  →  Hypothesis  →  Experiment  →  Act
  • HTTP://HELLOBAR.COM
  • What is Hello Bar?
  •  Metrics    Figure  out  what  to  improve    Hypothesis    Make  an  educated  guess        Experiment    Test  your  guess    Act    Decide  what’s  next
  • Our funnel showed a low installation rate Hello Bar Funnel Metrics Steps Visitors Homepage URL Entries Created First Bar Completed Registration Installed Totals 15068 1868 716 688 398 % prev step 12.40% 38.33% 96.09% 57.85% % drop off -87.60% -61.67% -3.91% -42.15% % of visitors 100% 12.40% 4.75% 4.57% 2.64%
  • We  achieved  a  40%  increase  for  our   installa2on  rate  with  just  this  one   first  experiment. The Lean Analytics Cycle At Work
  • Original Hello Bar installation screen
  •  Metrics    Figure  out  what  to  improve    Hypothesis    Make  an  educated  guess        Experiment    Test  your  guess    Act    Decide  what’s  next
  • Find out why people don’t install Hello Bar
  • Didn’t Meet My Expectations
  • I Had Trouble Installing It
  • I Don’t Have A Website
  • Not Ready To Install
  • Other
  • Hello Bar Experiment Hypothesis If  more  installa2on  op2ons  are  given,   then  more  people  will  add  Hello  Bar   to  their  website  due  to  the  need  for  a   WordPress  plugin  and  ability  to  email   the  code  to  someone  else.
  •  Metrics    Figure  out  what  to  improve    Hypothesis    Make  an  educated  guess        Experiment    Test  your  guess    Act    Decide  what’s  next
  • Created a new flow to test
  • I can install code myself.
  • 41.67% responded with “WordPress Plugin”
  • 25% responded with “email my developer / web designer”
  • I need help.
  •  Metrics    Figure  out  what  to  improve    Hypothesis    Make  an  educated  guess        Experiment    Test  your  guess    Act    Decide  what’s  next
  • Full funnel metrics for the experiment Variation Steps Visitors Homepage URL Entries Created First Bar Completed Registration Installed Totals 7456 906 342 329 276 % prev step 12.15% 37.75% 96.20% 83.89% % drop off -87.85% -62.25% -3.80% -16.11% % of visitors 100% 12.15% 4.59% 4.41% 3.70% Control Steps Visitors Homepage URL Entries Created First Bar Completed Registration Installed Totals 7469 925 353 341 197 % prev step 12.38% 38.16% 96.60% 57.77% % drop off -87.62% -61.84% -3.40% -42.23% % of visitors 100% 12.38% 4.73% 4.57% 2.64%
  • Resulted in a 30% installation rate increase HTTP://ISVALID.ORG Experiment resulted in a 40% improvement
  • ACer  a  dozen  more  experiments,  our   installa2on  rate  is  now  hovering   between  5%  and  7%   That’s  an  addi;onal  89%  increase!!! The Lean Analytics Cycle At Work
  • Metrics  →  Hypothesis  →  Experiment  →  Act How You Can Put It Into Practice
  • Metrics   Figure  out  what  to  improve
  • “What gets measured, gets managed.” Peter Drucker
  • Baselines DATA DNA EBOOK HTTP://KISS.LY/OPTDATADNA
  • Funnels KISSMETRICS FUNNEL REPORT HTTP://KISS.LY/KMFUNNEL
  • SaaS Funnel KISSMETRICS FUNNEL REPORT HTTP://KISS.LY/KMFUNNEL
  • eCommerce Funnel KISSMETRICS FUNNEL REPORT HTTP://KISS.LY/KMFUNNEL
  • Revenue BUFFER BAREMETRICS HTTP://KISS.LY/BUFFERBARE
  • Spreadsheets and Calculations HELLOBAR HTTP://HELLOBAR.COM Steps Visitors Homepage URL Entries Created First Bar Completed Registration Installed Created Second Bar Totals 15068 1868 716 688 398 193 % prev step 12.40% 38.33% 96.09% 57.85% 48.49% % drop off -87.60% -61.67% -3.91% -42.15% -51.51% % of visitors 100% 12.40% 4.75% 4.57% 2.64% 1.28%
  • Hypothesis   Make  an  educated  guess
  • “A solid test hypothesis is an informed solution to a real problem – not an arbitrary guess. The more research and data you have to base your hypothesis on, the be er it will be.”
 — Michael Aagaard, h p://kiss.ly/abthypo
  • Research   Tools  &  Tac;cs
  • Customer Development HTTP://WWW.CUSTDEV.COM
  • Jobs To Be Done #JTBD HTTP://JOBSTOBEDONE.ORG
  • Usability Testing HTTP://USERTESTING.COM
  • Live Chat HTTP://OLARK.COM
  • Mouse Tracking HTTP://CRAZYEGG.COM
  • Long Form Surveying HTTP://SURVEYMONKEY.COM
  • On-Page Surveying HTTP://QUALAROO.COM
  • Personalized Messaging HTTP://INTERCOM.IO
  • Email a survey to people HTTP://SURVEYMONKEY.COM
  • Ask people why they canceled HTTP://SURVEYMONKEY.COM
  • Get feedback right when they cancel
  • Exit Surveys HTTP://QUALAROO.COM
  • Discover what’s missing HTTP://QUALAROO.COM
  • Find out what words people use to describe your product HTTP://QUALAROO.COM
  • Learn about people’s expectations HTTP://INTERCOM.IO
  • Learn what frustrates people HTTP://INTERCOM.IO
  • More actionable reading… QUALITATIVE DATA HTTP://KISS.LY/QUALABT
  • Experiment Hypothesis Formula DATA DNA EBOOK HTTP://KISS.LY/OPTDATADNA
  • Experiment   Test  your  guess
  • Only  1  out  of  5  tests  win. 4  out  of  5  tests   cost  you  money. $
  • 1)  Which  top  metric  are  we  trying  to  improve?   2)  What  can  you  anchor  your  metric  against?   3)  What  is  your  hypothesis  about  why  this  metric  is  at  x%?   4)  What  do  we  expect  from  this  experiment?   5)  What  was  the  result  and  what  did  we  learn  from  it? HTTP://KISS.LY/6QSAB Answer key questions for each experiment
  • Prioritize your experiments HTTP://KISS.LY/ABPIE
  • Don’t copy “best practices” blindly
  • Learn what works best for you
  • Act   Decide  what’s  next
  • Explore  the  results  and  make   data  informed  decisions. THE ANALYTICS THAT MATTER TO FACEBOOK HTTP://KISS.LY/FBINFORM
  • KISSmetrics A/B Testing Report KISSMETRICS A/B TESTING REPORT HTTP://KISS.LY/KMABTEST
  • We know that people love Google Analytics
  • KISSmetrics Experiment Hypothesis If  people  authen2cate  with  Google,   then  more  people  will  sign  up  due  to   their  exis2ng  familiarity  with  Google   Analy2cs  and  alignment  with  our   winning  headline.
  • KISSmetrics Homepage Test Control
  • KISSmetrics Homepage Test Variation
  • KISSmetrics Homepage Test Signup Conversion Rate KISSMETRICS A/B TESTING REPORT HTTP://KISS.LY/KMABTEST
  • KISSmetrics Homepage Test Received Data KISSMETRICS A/B TESTING REPORT HTTP://KISS.LY/KMABTEST
  • KISSmetrics Homepage New Control
  • Inspiration
  • KISSmetrics Homepage New Test Variation
  • KISSmetrics Homepage Test Signup Conversion Rate KISSMETRICS A/B TESTING REPORT HTTP://KISS.LY/KMABTEST
  • KISSmetrics Homepage Test Received Data KISSMETRICS A/B TESTING REPORT HTTP://KISS.LY/KMABTEST
  • Growth! HTTP://KISSMETRICS.COM
  •  Metrics    Figure  out  what  to  improve    Hypothesis    Make  an  educated  guess        Experiment    Test  your  guess    Act    Decide  what’s  next
  • Hiten Shah [email protected] @hnshah You will get all you want in life if you help enough other people get what they want. Zig Ziglar Download the slides: h p://kiss.ly/leanprod2014
Fly UP