These are great points! Part of why I changed our interview process back then (I’m no longer with Space Monkey) was to actually try and address a lot of these issues.
So, yeah, gotcha trivia questions are terrible! That’s why we stopped whiteboard interviews entirely. They aren’t representative. What we wanted was a way of evaluating a candidate in a less stressful environment (on their own computer, with docs, their IDE, without people staring over their shoulder) doing something more close to day-to-day work.
We picked homework problems as kind of the least terrible way of doing this. The downside of homework problems is the time issue. We replaced whiteboard problems (and time you’d have to spend to come in to the office for them), so this wasn’t in addition to other interviewing. That said, perhaps the homework problem takes longer. Hopefully the homework problem doesn’t take longer than the time it would have taken to come in and do whiteboard problems for a few hours, but still. Because of that, we decided that perhaps it would be better if we paid people to do those homework problems. The paying people is to try and be cognizant of people’s time and make up for it.
I think interviewing is a two-way street so we gave candidates the option of coming to work with us for a day or two on real world tickets. They didn’t have to do this, but if they wanted to we were more than happy to pay them for a short term contract. It wasn’t a requirement, but just an option. I’m all about giving people options. In no case did we ever make someone do a whiteboard problem AND a homework problem AND a short term contract. It was always strictly zero or one of those options. Never more than one.
Last, if a candidate had a body of code samples they wanted to supply us instead that were sufficient, we were more than happy to use those instead of a homework problem. That’s why I said zero or one before.
Incidentally this is still the interview process I use at storj.io. We pay more at Storj for homework problems than I was able to at Space Monkey, which is good, but I’m always trying to improve it to make it better.
Does any of this background make it seem less objectionable? I’m 100% on board with using published papers and open source software and references as the interview if those things are all available, but when that’s not available? By having a homework problem, someone who doesn’t have a catalog of existing work has a real shot at proving themselves to be a very capable candidate in a way that lines up with what work actually is.
This is pretty fun and cool. Thanks for writing this!
[Comment removed by author]
out of curiosity, what do you mean?
[Comment removed by author]
These are great points! Part of why I changed our interview process back then (I’m no longer with Space Monkey) was to actually try and address a lot of these issues.
So, yeah, gotcha trivia questions are terrible! That’s why we stopped whiteboard interviews entirely. They aren’t representative. What we wanted was a way of evaluating a candidate in a less stressful environment (on their own computer, with docs, their IDE, without people staring over their shoulder) doing something more close to day-to-day work.
We picked homework problems as kind of the least terrible way of doing this. The downside of homework problems is the time issue. We replaced whiteboard problems (and time you’d have to spend to come in to the office for them), so this wasn’t in addition to other interviewing. That said, perhaps the homework problem takes longer. Hopefully the homework problem doesn’t take longer than the time it would have taken to come in and do whiteboard problems for a few hours, but still. Because of that, we decided that perhaps it would be better if we paid people to do those homework problems. The paying people is to try and be cognizant of people’s time and make up for it.
I think interviewing is a two-way street so we gave candidates the option of coming to work with us for a day or two on real world tickets. They didn’t have to do this, but if they wanted to we were more than happy to pay them for a short term contract. It wasn’t a requirement, but just an option. I’m all about giving people options. In no case did we ever make someone do a whiteboard problem AND a homework problem AND a short term contract. It was always strictly zero or one of those options. Never more than one.
Last, if a candidate had a body of code samples they wanted to supply us instead that were sufficient, we were more than happy to use those instead of a homework problem. That’s why I said zero or one before.
Incidentally this is still the interview process I use at storj.io. We pay more at Storj for homework problems than I was able to at Space Monkey, which is good, but I’m always trying to improve it to make it better.
Does any of this background make it seem less objectionable? I’m 100% on board with using published papers and open source software and references as the interview if those things are all available, but when that’s not available? By having a homework problem, someone who doesn’t have a catalog of existing work has a real shot at proving themselves to be a very capable candidate in a way that lines up with what work actually is.