There have been many different approaches to ranking VC Firms over the years I’ve been an entrepreneur and a VC. Each approach I’ve seen has issues. Most are easily gamed or have statistical bias issues.
I got the following note from Roy Bahat at Bloomberg Beta a while ago about a new approach called Founder’s Choice.
We and a few other firms sponsored a “founders choice” version of the Midas List, with a legit (IMHO) rating methodology, built by two Penn students. No vitriol possible (unlike The Funded, etc.). We’ve wanted this to exist for a long time — NPS of us as a firm is too forgiving a metric, everyone scores well.
My first question was:
How are they dealing with sampling bias on this one? For example, we send to all our founders and say “please fill this out and give us high scores.” Mostly just curious on methodology.
Roy had a thoughtful answer that made me a believer after a few more questions.
You are literally the only one (and I’m relieved someone did) to ask on sampling bias. For context, the general way it works is founders auth with LinkedIn and then the product tosses away their identity (or, more accurately, only keeps a hash and disconnects it from their ratings). Then the founders get asked for pairwise comparisons of only the VC firms who have backed them (so this is about who founders like as investors, not who has sour grapes from a pitch). How this addresses, to a degree, sampling bias:
1. Dampens outliers: because it only asks for pairwise comparisons between firms (like an ELO rating in chess, if you’re familiar), one very un/happy respondent can only affect so much, and same for a sample. (As opposed to giving one firm a 10 and everyone else 2’s or something.)
2. At the same time, it forces comparisons. A firm can ask founders to rate them highly, but ultimately founders have to choose who gave them more value. Can’t rate everyone a 10.
3. This is why we’re looking for as broad participation as possible, because the sampling bias will actually probably most show up in which firms even have enough ratings to count. (Like ELO in chess, more ratings doesn’t necessarily help you — you get more “points” if a founder rates you as better than a highly-rated firm. More ratings can just as easily hurt as help.)
If you are a founder, go spend five minutes and anonymously rank your VCs on Founder’s Choice.
Registration for the Venture Deals Fall 2022 course is open.
The course is free and starts on September 20, 2022. This is the second time we are running the new version of the course (v2!) that was co-created with Techstars and Kauffman Fellows.
If interested, sign up now. I hope to see you there in one of the AMAs we will host for anyone who takes the course.