Open Source Your Interview Questions Test

Photo by Pawel Czerwinski on Unsplash

Disclaimer: This post covers a controversial topic, and the vast majority of the companies I have worked for in the past would not open source their interview questions. This article represents my personal opinion and does not reflect the perspective of any of my past, present, or future employers.

Most of us have experienced what I call “post interview clarity”. It’s when we walk out of an interview and realize we had a much better answer to a question asked by the interviewer than the one we gave. Perhaps we didn’t really understand the question, or we simply came up with a better example after having some more time to think about it. Maybe the stress of sitting in front of someone who is actively assessing you caused anxiety that made it difficult for us to construct our best responses. Regardless, we really wish we could rewind time and provide our better answer.

Being interviewed is hard. Often, we are being asked complicated questions by strangers who often already have a “right answer” hidden in the unseen recesses of their minds. Contrary to the real world where we would have time to do research and find solutions, interviewing requires us to come up with answers on the spot. However, that means interviewing is often a test to determine if a candidate will be successful in a way that is divorced from how most people find success.

Most companies conduct interviews to select the best people to fill their open roles. Since hiring processes are expensive and can have long cycles, it makes sense to reduce as much friction as possible. I believe that making your interview questions public can add significant effectiveness and efficiency to your hiring processes. More importantly, I think that your hiring process will become more accessible, inclusive and fair.

I’ve worked in the software development industry for my entire career (almost 25 years), and I’m very familiar with the types of interviews in that industry. This includes the initial pre-screen (usually done by HR), a “person” screen, and technical test that’s either done in person or as a take home exercise. Manager roles usually have questions around people management and Product and Design roles usually have a case study presentation. I think all of these types of questions can be made public.

Reasons to Make Your Interview Questions Public

Here are the reasons to open source your interview questions that I find the most compelling.

Better Prepared Candidates

When candidates prepare for interviews without foreknowledge of the questions, they have to guess what questions they’ll be asked. Sure, they can search unofficial online question repositories or ask people who have previously interviewed with the company, but that’s more a test of the candidate’s amount of free time and strength of social connections rather than an assessment of their fitness for the job.

If candidates know the questions in advance, then they can tailor their background, experiences, and skillsets to the actual things the interviewer wants to know about them. Of course, this doesn’t preclude interviewers from taking deep dives into the prepared answers of the candidate or asking follow-up questions.

Level the Playing Field

I already mentioned that candidates can find company interview questions by searching repositories or talking to people who have been through the process. They can also hire interviewing coaches or consultants who can provide company interview questions and how to answer them. However, these methods are often easier to employ by candidates who have a lot of privilege. A single parent who works a job without a lot of flexibly probably won’t have time to do extensive searches through online interview question banks. Also, knowing people who have been through the interview process is usually a function of being in the same demographic group as the people who already work at the company.

Some people don’t like providing off-the-cuff answers so they may struggle when a question is posed to them. They may think “I don’t know the answer” and, therefore, not give one instead of sharing what they know. Women are often socialized to not feel qualified if they don’t know all of the answers, and this results in fewer women making it through interview processes.

If you want to create a hiring process that doesn’t further privilege the privileged, then making your interview questions public can help level the playing field. You provide a crucial asset to people from disadvantaged and lower socioeconomic backgrounds and remove an unfair advantage that has nothing to do with job performance.

Better Support Neurodiversity

Some candidates my be neurodiverse meaning that they have ways of thinking that are often diagnosed as autism, dyslexia, attention deficit hyperactivity disorder (ADHD), Tourette Syndrome, etc. These candidates can have difficulty focusing or may struggle to articulate complex concepts in new environments. Providing the interview questions in advance can help these candidates demonstrate their alignment with your needs.

Go Beyond “Right” and “Wrong” Answers

Having been on hiring teams for several years across multiple companies, I’ve seen a lot of technical interview questions. They sometimes look like lists of engineering minutiae that no one uses in everyday work. I think that the people who use these questions revel in “trick questions” to see if the candidate has properly crammed for the interview. I don’t see much value in these questions.

I prefer to ask questions that see how the candidate thinks about problems. These are usually behavioral questions that ask candidates to give specific examples of how they’ve used specific tools and techniques from their work history. I find this to be a much better way of determining if someone will be successful. These are also questions that are tough to prepare for by memorizing answers. This makes the interview feel like a conversation instead of a quiz.

Broadcast Your Values

You interview questions are a reflection of your values. If you only ask engineers questions about writing code and not about how they work in teams, then that tells candidates a lot about your priorities. Making your interview questions public is a great way to let the market know what’s really important to your organization.

Get Better Interview Questions

If you make your interview questions public, then you have the opportunity to get feedback on your questions or get proposals for better questions than the ones currently on your list. Interviewing can be a time and resource heavy process, and companies often don’t have the bandwidth to regularly assess the quality, effectiveness, and completeness of their questions. Publicizing your questions can be a great force function for making them better.

How to Make Your Open Source Your Interview Questions

Open sourcing your interview questions is fairly straightforward:

Choose a Location for Your Question Bank

Most companies already have an internal list of questions (often called a “question bank”). You can take this existing list and put it in a publicly accessible location. I have some bias, but GitHub is a great place to put your question bank. You can also add the questions to the pages of your website that discusses careers and hiring.

Share the Question Bank

Once you’ve placed your question bank in a public place, you need to make sure that candidates can find it. A great place to put the link is in the emails that companies send to candidates informing them that they have made it to the interview phase of the hiring process. You can also place the link in the job description for each role or make them part of the application process.

If putting all of your interview questions out at once seems daunting, you can follow a process of “progressive revelation”. By that, I mean you can send the link to the initial screen for people who make it that stage. However, only the candidates that pass the initial screen get the questions to the next stage. You can continue to parcel out the questions as people progress through your process.

Companies That Have Open Sourced Their Interview Questions

Here are a few companies that have, at least partially, open sourced their interview questions:

Arguments Against Making Your Interview Questions Public

I want to end by stating that I understand why companies often refuse to make their interview questions public. Here are the arguments I usually encounter and why I think they aren’t very strong.

Candidates Will Rehearse Their Answers

This is the most common response I get to the idea of open sourcing your interview questions. My first answer is usually, “That’s already happening”. My second answer is usually to point out that you can always ask follow-up questions that go beyond any rehearsed answer.

We’ll Have to Grade Candidates Harder

I find it ironic that “we’ll have to make the process harder” is often the response to efforts to make the process more fair and inclusive. While I do think that candidates who don’t take advantage of publicly accessible interview questions should be judged on their preparation skills, I don’t think that the process should become harder for everyone.

A lot of my thinking on this topic came from this Twitter thread I started on May 21st, 2021.