Lean Analytics Cycle

Data & Analytics

hiten-shah
of 96
Description
 
Text
  • Lean Analytics Lean Product & Lean UX Silicon Valley - July 2014 Hiten Shah hnshah@gmail.com Download the slides: h p://kiss.ly/leanprod2014
  • Continuous Learning & Improvement
  • Don’t know where to start?
  • Wondering if you’re working on the right things?
  • Or, you just want to hear me babble!
  • HTTP://KISS.LY/LEANLYTICSBOOK
  • Lean Analytics helps you Optimize for Learning
  • Lean Analytics Cycle AVINASH KAUSHIK HTTP://KISS.LY/LEANAC
  • Metrics  →  Hypothesis  →  Experiment  →  Act
  • HTTP://HELLOBAR.COM
  • What is Hello Bar?
  •  Metrics    Figure  out  what  to  improve    Hypothesis    Make  an  educated  guess        Experiment    Test  your  guess    Act    Decide  what’s  next
  • Our funnel showed a low installation rate Hello Bar Funnel Metrics Steps Visitors Homepage URL Entries Created First Bar Completed Registration Installed Totals 15068 1868 716 688 398 % prev step 12.40% 38.33% 96.09% 57.85% % drop off -87.60% -61.67% -3.91% -42.15% % of visitors 100% 12.40% 4.75% 4.57% 2.64%
  • We  achieved  a  40%  increase  for  our   installa2on  rate  with  just  this  one   first  experiment. The Lean Analytics Cycle At Work
  • Original Hello Bar installation screen
  •  Metrics    Figure  out  what  to  improve    Hypothesis    Make  an  educated  guess        Experiment    Test  your  guess    Act    Decide  what’s  next
  • Find out why people don’t install Hello Bar
  • Didn’t Meet My Expectations
  • I Had Trouble Installing It
  • I Don’t Have A Website
  • Not Ready To Install
  • Other
  • Hello Bar Experiment Hypothesis If  more  installa2on  op2ons  are  given,   then  more  people  will  add  Hello  Bar   to  their  website  due  to  the  need  for  a   WordPress  plugin  and  ability  to  email   the  code  to  someone  else.
  •  Metrics    Figure  out  what  to  improve    Hypothesis    Make  an  educated  guess        Experiment    Test  your  guess    Act    Decide  what’s  next
  • Created a new flow to test
  • I can install code myself.
  • 41.67% responded with “WordPress Plugin”
  • 25% responded with “email my developer / web designer”
  • I need help.
  •  Metrics    Figure  out  what  to  improve    Hypothesis    Make  an  educated  guess        Experiment    Test  your  guess    Act    Decide  what’s  next
  • Full funnel metrics for the experiment Variation Steps Visitors Homepage URL Entries Created First Bar Completed Registration Installed Totals 7456 906 342 329 276 % prev step 12.15% 37.75% 96.20% 83.89% % drop off -87.85% -62.25% -3.80% -16.11% % of visitors 100% 12.15% 4.59% 4.41% 3.70% Control Steps Visitors Homepage URL Entries Created First Bar Completed Registration Installed Totals 7469 925 353 341 197 % prev step 12.38% 38.16% 96.60% 57.77% % drop off -87.62% -61.84% -3.40% -42.23% % of visitors 100% 12.38% 4.73% 4.57% 2.64%
  • Resulted in a 30% installation rate increase HTTP://ISVALID.ORG Experiment resulted in a 40% improvement
  • ACer  a  dozen  more  experiments,  our   installa2on  rate  is  now  hovering   between  5%  and  7%   That’s  an  addi;onal  89%  increase!!! The Lean Analytics Cycle At Work
  • Metrics  →  Hypothesis  →  Experiment  →  Act How You Can Put It Into Practice
  • Metrics   Figure  out  what  to  improve
  • “What gets measured, gets managed.” Peter Drucker
  • Baselines DATA DNA EBOOK HTTP://KISS.LY/OPTDATADNA
  • Funnels KISSMETRICS FUNNEL REPORT HTTP://KISS.LY/KMFUNNEL
  • SaaS Funnel KISSMETRICS FUNNEL REPORT HTTP://KISS.LY/KMFUNNEL
  • eCommerce Funnel KISSMETRICS FUNNEL REPORT HTTP://KISS.LY/KMFUNNEL
  • Revenue BUFFER BAREMETRICS HTTP://KISS.LY/BUFFERBARE
  • Spreadsheets and Calculations HELLOBAR HTTP://HELLOBAR.COM Steps Visitors Homepage URL Entries Created First Bar Completed Registration Installed Created Second Bar Totals 15068 1868 716 688 398 193 % prev step 12.40% 38.33% 96.09% 57.85% 48.49% % drop off -87.60% -61.67% -3.91% -42.15% -51.51% % of visitors 100% 12.40% 4.75% 4.57% 2.64% 1.28%
  • Hypothesis   Make  an  educated  guess
  • “A solid test hypothesis is an informed solution to a real problem – not an arbitrary guess. The more research and data you have to base your hypothesis on, the be er it will be.”
 — Michael Aagaard, h p://kiss.ly/abthypo
  • Research   Tools  &  Tac;cs
  • Customer Development HTTP://WWW.CUSTDEV.COM
  • Jobs To Be Done #JTBD HTTP://JOBSTOBEDONE.ORG
  • Usability Testing HTTP://USERTESTING.COM
  • Live Chat HTTP://OLARK.COM
  • Mouse Tracking HTTP://CRAZYEGG.COM
  • Long Form Surveying HTTP://SURVEYMONKEY.COM
  • On-Page Surveying HTTP://QUALAROO.COM
  • Personalized Messaging HTTP://INTERCOM.IO
  • Email a survey to people HTTP://SURVEYMONKEY.COM
  • Ask people why they canceled HTTP://SURVEYMONKEY.COM
  • Get feedback right when they cancel
  • Exit Surveys HTTP://QUALAROO.COM
  • Discover what’s missing HTTP://QUALAROO.COM
  • Find out what words people use to describe your product HTTP://QUALAROO.COM
  • Learn about people’s expectations HTTP://INTERCOM.IO
  • Learn what frustrates people HTTP://INTERCOM.IO
  • More actionable reading… QUALITATIVE DATA HTTP://KISS.LY/QUALABT
  • Experiment Hypothesis Formula DATA DNA EBOOK HTTP://KISS.LY/OPTDATADNA
  • Experiment   Test  your  guess
  • Only  1  out  of  5  tests  win. 4  out  of  5  tests   cost  you  money. $
  • 1)  Which  top  metric  are  we  trying  to  improve?   2)  What  can  you  anchor  your  metric  against?   3)  What  is  your  hypothesis  about  why  this  metric  is  at  x%?   4)  What  do  we  expect  from  this  experiment?   5)  What  was  the  result  and  what  did  we  learn  from  it? HTTP://KISS.LY/6QSAB Answer key questions for each experiment
  • Prioritize your experiments HTTP://KISS.LY/ABPIE
  • Don’t copy “best practices” blindly
  • Learn what works best for you
  • Act   Decide  what’s  next
  • Explore  the  results  and  make   data  informed  decisions. THE ANALYTICS THAT MATTER TO FACEBOOK HTTP://KISS.LY/FBINFORM
  • KISSmetrics A/B Testing Report KISSMETRICS A/B TESTING REPORT HTTP://KISS.LY/KMABTEST
  • We know that people love Google Analytics
  • KISSmetrics Experiment Hypothesis If  people  authen2cate  with  Google,   then  more  people  will  sign  up  due  to   their  exis2ng  familiarity  with  Google   Analy2cs  and  alignment  with  our   winning  headline.
  • KISSmetrics Homepage Test Control
  • KISSmetrics Homepage Test Variation
  • KISSmetrics Homepage Test Signup Conversion Rate KISSMETRICS A/B TESTING REPORT HTTP://KISS.LY/KMABTEST
  • KISSmetrics Homepage Test Received Data KISSMETRICS A/B TESTING REPORT HTTP://KISS.LY/KMABTEST
  • KISSmetrics Homepage New Control
  • Inspiration
  • KISSmetrics Homepage New Test Variation
  • KISSmetrics Homepage Test Signup Conversion Rate KISSMETRICS A/B TESTING REPORT HTTP://KISS.LY/KMABTEST
  • KISSmetrics Homepage Test Received Data KISSMETRICS A/B TESTING REPORT HTTP://KISS.LY/KMABTEST
  • Growth! HTTP://KISSMETRICS.COM
  •  Metrics    Figure  out  what  to  improve    Hypothesis    Make  an  educated  guess        Experiment    Test  your  guess    Act    Decide  what’s  next
  • Hiten Shah hnshah@gmail.com @hnshah You will get all you want in life if you help enough other people get what they want. Zig Ziglar Download the slides: h p://kiss.ly/leanprod2014
Comments
Top