System design interviews might get subjective depending on the expertise of both candidate and interviewer. Even if both have worked in backend systems, there’s a big chance that they have worked in quite distant areas. To decrease unconscious bias, it’s worthwhile to use a scorecard and general guidance to stay objective while evaluating a candidate’s skill set.
Score Card
We can score each one of these skill sets from 1-10. We can then use these scores to determine overall performance.
Communication
She listens well, clarifies the problem definition, and takes on board input/output.
Business Acumen
She considers the use cases for the proposed solution and thinks about relevant KPIs to measure success. She thinks of additional use cases where the product/service/system might be fruitful.
Autonomy
She considers different aspects of the problem without prompting. Senior candidates proactively consider all elements of the problem. She evaluates multiple proposed approaches and trade-offs on her own.
System design
She proposes appropriate high-level system design including service decomposition, data stores, and infrastructure. Senior candidates should design for failure.
Efficiency and scalability
She proposes a working solution. She is able to decompose the solution into sub-solutions. She provides horizontally scalable solutions. She finds bottlenecks and single points of failure.
Operational excellence
She comes up with operational metrics, CI/CD, monitoring, alerting, and reliable deployment strategies.
Planning
She delivers a plan to roll out the system iteratively. She estimates team size for each component or overall service/system.
Leveling
We can take multiple angles in leveling. Nevertheless, staying consistent among candidates requires experience. I broke down the overall expectations from each level as follows.
Mid-senior
He comes up with a solution. He has some experience to solve the part of the problem. He explains the rest of the system based on the theory. He focuses on the implementation detail with less focus on the overall system design. He puts less thought into how a problem would be broken down across a team.
Senior
He comes up with a detailed solution. He has experience in building similar systems. He has a good theoretical knowledge of the parts s/he doesn’t have experience building. He has a strong sense of the design of the overall system and how this could be delivered by a team. The interviewer observes evidence of planning, scaling, optimization, and operations.
Staff+
He comes up with a comprehensive solution that considers not only the current business case but also future/other business objectives. He has overseen and has experience in similar solutions. He has excellent practical experience backed by excellent theory. He proposes novel solutions, discusses multiple trade-offs, and anticipates the evolution of the systems. He has an appetite for scaling, reuse, optimization, and operational excellence.
Training
One of the practical ways to get better at interviewing is training. One way to establish expectations is to interview colleagues with the same interview question. We should select colleagues from different levels. We can then define the expectation for each engineering level more clearer. Once we get better clarity on the question, we can begin interviewing. People can shadow the interviewer and then reverse shadow later. Shadowing means observing the interview while someone conducts the interview. Reverse shadowing means interviewing while someone observes. If we can do this for all the system design interview questions, then we can have better success at evaluating candidates objectively.
Evaluation
The scorecard will give us guidance on how the interview has been. It’s important to touch on the scorecard topics. Nevertheless, we shouldn’t be mechanical. I have seen many interviewers who only try to fill in the scorecard. Filling a score card is not a conversation. The interview should be a challenging yet enjoyable conversation for both parties. We should take notes down. We can then leverage our interview notes to mark scores for each skill accurately. Combining leveling guidance with a score card should give us a good sense of how the interview has been.
Final remarks
Interview performance depends on many different factors. One notable mention is the area. If we expect someone to work on databases, we should probably have a system design question that concentrates on databases. I don’t see the point of asking someone a machine learning question if they would work on databases. Thus, each organization should probably have its own set of interview questions.
All in all, I believe both interviewing someone or interviewing for a position is stressful. The interview performance doesn’t necessarily show if someone will be good at the role. I have seen it over and over again. Some people are just good at exams, interviews, and so forth. Some people stress a lot and need time to think. Hence, the interview process doesn’t define anyone’s success. It’s just a tool for hiring people.