SlideShare a Scribd company logo
Data-Driven
Product Design
J Christopher Garcia
Chris Hyams
Design
J Christopher Garcia
User Experience Lead, Indeed
[@IndeedEng] Data-Driven Product Design
[@IndeedEng] Data-Driven Product Design
[@IndeedEng] Data-Driven Product Design
2008
2013
[@IndeedEng] Data-Driven Product Design
[@IndeedEng] Data-Driven Product Design
[@IndeedEng] Data-Driven Product Design
114 tests
244 variations
84,000 combinations
[@IndeedEng] Data-Driven Product Design
[@IndeedEng] Data-Driven Product Design
[@IndeedEng] Data-Driven Product Design
Direct observation
can be inefficient
Before and after
doesn't work
Traffic doesn't look like this
Hourly traffic looks like this...
...and daily traffic looks like this
...and monthly traffic looks like this
[@IndeedEng] Data-Driven Product Design
[@IndeedEng] Data-Driven Product Design
Local maximum
[@IndeedEng] Data-Driven Product Design
[@IndeedEng] Data-Driven Product Design
[@IndeedEng] Data-Driven Product Design
[@IndeedEng] Data-Driven Product Design
[@IndeedEng] Data-Driven Product Design
[@IndeedEng] Data-Driven Product Design
[@IndeedEng] Data-Driven Product Design
[@IndeedEng] Data-Driven Product Design
What we've learned
UI Principles
Clickable items should
look clickable
[@IndeedEng] Data-Driven Product Design
[@IndeedEng] Data-Driven Product Design
[@IndeedEng] Data-Driven Product Design
[@IndeedEng] Data-Driven Product Design
[@IndeedEng] Data-Driven Product Design
Users read
[@IndeedEng] Data-Driven Product Design
[@IndeedEng] Data-Driven Product Design
[@IndeedEng] Data-Driven Product Design
[@IndeedEng] Data-Driven Product Design
[@IndeedEng] Data-Driven Product Design
[@IndeedEng] Data-Driven Product Design
[@IndeedEng] Data-Driven Product Design
a word is worth a
thousand icons
[@IndeedEng] Data-Driven Product Design
[@IndeedEng] Data-Driven Product Design
[@IndeedEng] Data-Driven Product Design
[@IndeedEng] Data-Driven Product Design
[@IndeedEng] Data-Driven Product Design
[@IndeedEng] Data-Driven Product Design
+14%
more search
results pages
BEWARE MULTIPLE
CHANGES
[@IndeedEng] Data-Driven Product Design
[@IndeedEng] Data-Driven Product Design
[@IndeedEng] Data-Driven Product Design
[@IndeedEng] Data-Driven Product Design
[@IndeedEng] Data-Driven Product Design
Searches: -2.0%
Job Clicks: -1.7%
Saved jobs: -7.6%
[@IndeedEng] Data-Driven Product Design
[@IndeedEng] Data-Driven Product Design
[@IndeedEng] Data-Driven Product Design
[@IndeedEng] Data-Driven Product Design
[@IndeedEng] Data-Driven Product Design
Searches: -1.8%
Job Clicks: -0.8%
Saved jobs: 0%
[@IndeedEng] Data-Driven Product Design
Really?
Really.
[@IndeedEng] Data-Driven Product Design
[@IndeedEng] Data-Driven Product Design
[@IndeedEng] Data-Driven Product Design
[@IndeedEng] Data-Driven Product Design
[@IndeedEng] Data-Driven Product Design
[@IndeedEng] Data-Driven Product Design
[@IndeedEng] Data-Driven Product Design
[@IndeedEng] Data-Driven Product Design
[@IndeedEng] Data-Driven Product Design
[@IndeedEng] Data-Driven Product Design
[@IndeedEng] Data-Driven Product Design
[@IndeedEng] Data-Driven Product Design
[@IndeedEng] Data-Driven Product Design
Product
Chris Hyams
SVP, Product & Engineering
I help
people
get jobs.
What's best for the job
seeker?
How do we know what's
best for job seekers?
Test and measure
EVERYTHING
Measurement
1. Log everything
jobsearch index
abredistime
acmetime
addltime
adsc
adsdelay
adsi
badsc
badsi
boostojc
boostoji
bsjc
bsjcwia
bsji
bsjindapplies
bsjindappviews
bsjrev
bsjwia
ckcnt
cksz
counts
ctkage
ctkagedays
dayofweek
dcpingtime
domTotalTime
ds-mpo
dsmiss
dstime
featemp
fj
freekwac
freekwarev
freesjc
freesjrev
frmtime
galatdelay
iplat
iplong
jslatdelay
jsvdelay
kwac
kwacdelay
kwai
kwarev
kwcnt
lacinsize
lacsgsize
lmstime
mpotime
mprtime
navTotTime
ndxtime
ojc
ojclong
ojcshort
ojcwia
oji
ojindapplies
ojindappviews
ojwia
oocsc
page
prcvdlatency
primfollowcnt
prvwoji
prvwojlat
prvwojopentime
prvwojreq
radsc
radsi
recidlookupbudget
rectime
redirCount
redirTime
relfollowcnt
respTime
returnvisit
rojc
roji
rqcnt
rqlcnt
rqqcnt
rrsjc
rrsji
rrsjrev
rsavail
rsjc
rsji
rsused
rsviable
serpsize
sjc
sjcdelay
sjclong
sjcnt
sjcshort
sjcwia
sji
sjindapplies
sjindappviews
sjrev
sjwia
sllat
sllong
sqc
sqi
sugtime
svj
svjnostar
svjstar
tadsc
tadsi
time
timeofday
totcnt
totfollowcnt
totrev
tottime
tsjc
tsjcwia
tsji
tsjindapplies
tsjindappviews
tsjrev
tsjwia
unqcnt
vp
wacinsize
wacsgsize
acmepage
acmereviewmod
acmeservice
acmesession
adclick
adcrequest
adcrev
adschannel
adsclick
adsenseclick
adve
advt
agghttp
aggjira
aggjob
aggjob_waldorf
aggsherlock
aggsourcehealth
agstiming
api
apijsv
apisearch
archiveindex
archiveindex_shingled_test
bin
carclicks
click
clickanalytics
cobrand
dctmismatch
draw
dupepairs
dupepairs_mini
dupepairs_old
dupepairsall
dupepairsall_mini
ejchecker
emilyops
feedbridge
globalnav
googlebot_organic
homepage
impression
indeedapply
jhst
jobalert
jobalertorganic
jobalertsearch
jobalertsponsored
jobexpiration
jobexpiration2
jobexpiration3
jobprocessed
jobqueueblock
jobsearch
jssquery
keywordAd
locsvc
lucyindexermain
mechanicalturk
mindyops
mobhomepage
mobil
mobile
mobileorganic
mobilesponsored
mobrecjobs
mobsearch
mobviewjob
myindeed
myindfunnel
myindpage
myindrezcreate
myindsession
old
opsesjasx
organic
orgmodel
orgmodelsubset
orgmodelsubset90
passportaccount
passportpage
passportsignin
ramsaccess
recjobs
recommendservice
resumedata
resumesearch
rexcontacts
rexfunnel
reximpression
rexsearch
rezSrchSearch
rezalert
rezalertfunnel
rezfunnel
rezjserr
rezsrchrequest
rezview
searchablejobs
seo
session
sjmodel
sponsored
sysadappinfo
sysadapptiming
testndx
testndx1
testndx2
tmp
usrsvccache
usrsvcrequest
viewjob
webusersignin
16 million jobs
100 million UV
3 billion searches
1TB per day
Everything you ever wanted to know
about logging but were afraid to ask
Logrepo: Enabling
Data-Driven Decisions
2. Index data
3. Build tools to
query indexes
[another tech talk]
Tools answer questions
Mobile searches per hour in
JP vs. UK?
Resume creation by country?
Impressions in job alerts by
email domain?
Percent of app downloads from
iOS, Android, Windows?
How quickly does a datacenter take
on traffic after a failover?
Test EVERYTHING
Text
Visual design
Layout
Control flow
New features
Algorithms [also another tech talk]
Everything
5% 5%
grp:bluebtntst0 grp:bluebtntst1
split:
log:
Measure
statistical relevance
p-value
Big changes require less data
Conversion Improvement p-value Trials
10% +30% 0.038 1,000
source: http://go.indeed.com/jyusg
Small changes require more data
Conversion Improvement p-value Trials
10% +30% 0.038 1,000
10% +1% 0.042 750,000
source: http://go.indeed.com/ommgd
[@IndeedEng] Data-Driven Product Design
Product Principles
There's no such thing as
a no-brainer
[@IndeedEng] Data-Driven Product Design
-30%
Corollary: if a test fails,
it means the test failed
-30%
[@IndeedEng] Data-Driven Product Design
+25%
Little things can make
a big difference
[@IndeedEng] Data-Driven Product Design
organic
sponsored
+34%
Corollary: be skeptical
of big changes
Does color really matter
that much?
a A
a B
a C
+34%
+10%
Are people just bored
of blue?
+29% +22%
Will the effect wear off?
[@IndeedEng] Data-Driven Product Design
Get it right,
then do it right
[@IndeedEng] Data-Driven Product Design
Location backfill: job clicks
+8%
Location backfill: search time
+1,500%
Location backfill: performance fix
+1,500%
Location backfill: performance fix
+1,500% +0%
Location backfill: job clicks
+8%
Location backfill: clicks after fix
+8% +9%
Measure everything
(aka, Beware of unintended consequences)
April Fools' 2010
[@IndeedEng] Data-Driven Product Design
[@IndeedEng] Data-Driven Product Design
[@IndeedEng] Data-Driven Product Design
Organic clicks
+1.8%
Job alert signups
-17%
Revenue
+15.5%
Revenue
+15.5%
[@IndeedEng] Data-Driven Product Design
[@IndeedEng] Data-Driven Product Design
[@IndeedEng] Data-Driven Product Design
[@IndeedEng] Data-Driven Product Design
Job alert signups: top v. bottom
-17% +6%-21%
Corollary: attention is a
zero-sum game
(usually)
Search
Click
Revenue
Email signup
Resume create
...
Relative value of actions
[@IndeedEng] Data-Driven Product Design
Different for everyone
Location auto-complete
Zero-results page
[@IndeedEng] Data-Driven Product Design
[@IndeedEng] Data-Driven Product Design
Zero-results pages
-2.7%
Organic clicks
+0%
Clicks w/ auto-complete
+8%
Keyword ad revenue
+1,410%
Keyword ad revenue
+1,410%
keyword ads
[@IndeedEng] Data-Driven Product Design
[@IndeedEng] Data-Driven Product Design
[@IndeedEng] Data-Driven Product Design
[@IndeedEng] Data-Driven Product Design
Keyword ad revenue after fix
Measure everything,
every time
[@IndeedEng] Data-Driven Product Design
Getting Started
[@IndeedEng] Data-Driven Product Design
[@IndeedEng] Data-Driven Product Design
[@IndeedEng] Data-Driven Product Design
Fewer tests
Focus on big wins
Conversion Improvement p-value Trials
10% +30% 0.038 1,000
10% +1% 0.042 750,000
source: http://go.indeed.com/ommgd
Straw man
Determine # of trials in a week
Determine minimum improvement
One new test each week
If success, roll out; if not, kill & move on
What We Get
Quality
Risk tolerance
Culture of measurement
(Empirically)
better products
fin
[@IndeedEng] Data-Driven Product Design

More Related Content

[@IndeedEng] Data-Driven Product Design