I've been thinking about this for a while and have an interesting idea -- feedback welcome!
Imagine a fully open company -- code, tools, library, marketing data, strategy, etc. Literally anything about the company you can think of is accessible on the open and ready to interact with, except for the underlying database data.
Following along? Cool. Now, imagine all of this stuff being on some open repository (not necessarily Git, but for the sake of example let's suppose so). As employees create their backlog of items they add four properties:
- A 1-100 score on how much difficulty they would have completing the item, 100 being extreme, 1 being none
- A 1-100 score on how much difficulty they think an entry employee would have with the item. An entry employee would be defined clearly as the lowest level for their ladder, e.g. Software Developer I.
- A time estimate on how much time they would have completing the item.
- A time estimate they estimate on how long they think an entry level employee would have.
So, knowing on this. Would the perfect hiring process simply not be taking the items, picking things that should take a reasonable amount of time (less than 8 hours) that are in the highest reported entry difficulty possible and then scaling that by the average self difficulty? With this system you could even anonymize the dummy interview pull requests and have employees rate the quality of it. Differences in quality and difficulty could be a positive signal (e.g. high quality for low difficulty item is good, obviously, but high quality for low difficulty item that was estimated to take less time than the average estimate, by employees, is even better).
---
So the purpose of this is less to identify who is objectively great, but rather to figure out how good the candidate is relative to people in your own organization. Personally I think finding an objective measure is an intractable problem.
Domain knowledge is the uncontrollable here. A junior is still probably a junior 3 months in but they are a lot faster for their domain knowledge (codebase, business rules etc). Day 1 (I.e. Your proposed interview) there would be little observable difference between various experience levels/quality of engineer since the overhead of domain knowledge will dwarf anything they already have in their head. That is, unless the problem you pick is so isolated that there is no domain knowledge required. In that case you have expensively reproduced fizzbuzz without the mass memorization of code golf style answers.
Imagine a fully open company -- code, tools, library, marketing data, strategy, etc. Literally anything about the company you can think of is accessible on the open and ready to interact with, except for the underlying database data.
Following along? Cool. Now, imagine all of this stuff being on some open repository (not necessarily Git, but for the sake of example let's suppose so). As employees create their backlog of items they add four properties:
- A 1-100 score on how much difficulty they would have completing the item, 100 being extreme, 1 being none
- A 1-100 score on how much difficulty they think an entry employee would have with the item. An entry employee would be defined clearly as the lowest level for their ladder, e.g. Software Developer I.
- A time estimate on how much time they would have completing the item.
- A time estimate they estimate on how long they think an entry level employee would have.
So, knowing on this. Would the perfect hiring process simply not be taking the items, picking things that should take a reasonable amount of time (less than 8 hours) that are in the highest reported entry difficulty possible and then scaling that by the average self difficulty? With this system you could even anonymize the dummy interview pull requests and have employees rate the quality of it. Differences in quality and difficulty could be a positive signal (e.g. high quality for low difficulty item is good, obviously, but high quality for low difficulty item that was estimated to take less time than the average estimate, by employees, is even better).
---
So the purpose of this is less to identify who is objectively great, but rather to figure out how good the candidate is relative to people in your own organization. Personally I think finding an objective measure is an intractable problem.