Require a formal degree in CS? That's gatekeeping.

Need to pass a whiteboard exam? Not representative of the actual work.

Live coding session? Biased against people who don't perform well under pressure.

Take home project? It's too much work to do for free.

Showcase a personal portfolio? Not fair to people with families or other obligations.

Either you enforce a minimal level of competency upfront in the form of an academic degree, industry-standard exams such as a PE Exam, etc. OR you push the entire responsibility of vetting prospective applicants downstream to the employer—which is exactly why interviews are multiple week long gauntlets.

The tech world likes to complain about all of this but other occupations 100% DO have high standards - it's just that it's paid up-front.

Want to become a lawyer? - You've got to pass the LSAT, get into law school, and pass the bar.

Want to become a doctor? - You've got to pass the MCAT, get into medical school, and do residency.

Want to become a pilot? - You've got to get your PPL, pass your check ride, do your IFR, multi-engine, commercial rating, ATP

God there are some days that I ABSOLUTELY HATE THIS INDUSTRY.

Take home projects and personal portfolios are the options that are the best. Those are the ones where candidates are given the time to show their best work and don't require the debt of a college degree in a specific major (whose knowledge is probably largely irrelevant to your day-to-day coding).

If someone uses AI to defeat a CAPTCHA, does that mean the CAPTCHA was poorly designed because "the bot is just as valid a user as any human"? Of course not - the CAPTCHA's entire purpose was to establish that correlation between "can solve this" and "is human," and breaking that correlation through automated tools defeats its purpose entirely.

I don't think this sort of CAPTCHA type step should have a significant place in a good interview process in the first place, if it is important to you to put your applicants through this sort of gauntlet you're not treating them with the dignity they deserve.

If you're just looking to fill cogs in your machine, sure, but in that case you're just as likely to get kids who studied a bunch of interview questions but also don't know shit. I'd just as soon have someone who's good at ChatGPT.

Let me put it a different, perhaps simpler way:

Let's agree on the premise that good human engineers are much better hires than good AI users. If your hiring process fails to distinguish between good human engineers and good AI users that's evidence that your process is not producing good data.

In the exact same way that if your CAPTCHA is getting solved by bots it's a bad CAPTCHA.

It seems there are at least two different perspectives:

A) If a human can use AI to pass a remote interview that they would otherwise fail, then the questions are unsuitable for use in interviews, whether remote or in person.

B) If a human can use AI to pass a remote interview that they would fail if it were face to face, then the questions may be reasonable, but the safeguards for remote interviews are insufficient.

I disagree with A.

If you're allowing your programmers to use AI on the job you're not getting particularly useful information if unaided someone fails a question that they would answer correctly with AI assistance. Sure, being able to answer the question in person may still correlate with being a better applicant, but that doesn't mean it's a good question.

Interviews take time, you should make good use of the time by asking questions that tell you a lot. Questions easily answered by AI (this also goes for questions easily answered by studying interview questions by the way) are not good questions. Measure the things that matter.

   (this also goes for questions easily answered by studying interview questions by the way

Not necessarily... You just get the people that are technically competent enough and are self-corruptable enough to pass the test. There is still a correlation with performance but it is a weaker signal and you are more likely to miss out on people with equal ability but higher morals.

In my experience if a problem simple enough to be completed in an hour, it’s simple enough for AI.

I recently had a question where I was just given a bunch of tuples and told "let me know anything you can glean about this data in two hours". I think you learn much more by seeing how someone fails at a complex nebulous task than whether someone can implement a doubly linked list from memory or whatever.

Real life programming is like 80% failing at complex nebulous tasks, picking yourself up and trying again (by volume). Interviews should simulate that. If AI helps you, so be it.

That said, I also don’t know how to adequately test, other than to carefully watch people’s eyes during questions; even that isn’t foolproof.

You can completely change the complexity of a bog standard leetcode question with the slightest of modifications, and I’m certain no extant LLM will catch such a nuance if your problem is a very common one.

A fun exercise is to ask chatGPT what the river crossing goat puzzle is and then say “do the same puzzle but with two goats and a cabbage please”. It can’t get it right even with multiple corrections. Might stop working at some point now that I posted this though!

Currently the cheating mostly aims for coding tasks because they're pretty simple to do and self-contained. Much less for helping with a good dialog.

Over the years I've interviewed many people for technical positions and there are many ways of getting to the core of a prospective employee's competence.

Of course there are questions about training, past experience, qualifications and those about the work you expect them to perform but other matters also enter the equation and they're almost as important (the person's demeanor, confidence, etc.). By letting the person do most of the talking—explaining things about themselves and their work—you can learn a great deal.

Stating the obvious, interviewers ought to be on top of the job they're hiring for but from this story and from other anecdotal info I know that's not always so. In my case as head of the department I was pretty much up to speed on most aspects of the work, so it was clear to me when an applicant was spinning me BS. If they knew more than I did then they almost certainly got the job (life's easier when you've smart employees, for starters one can ask them questions instead of vice versa).

What's that leading up to? Well, when interviewing it was obvious to me (and to the other interviewers) within just the first few minutes whether the person was worth considering. It's uncanny how good one can become at doing this (and my record speaks for itself, I was pretty pleased with those employees I hired).

Let me illustrate with an example: some decades ago I had an applicant apply for an electronics job. He spoke only Chinese and a few broken words in English and none of us on the interview panel spoke Chinese. BTW, he turned up with a Chinese/English dictionary in hand.

I dispelled with many of the questions I usually asked and showed him a somewhat complex circuit diagram with many subsections that performed very different functions (if I showed this at all then it was usually towards the end of an interview).

Within seconds of looking at the diagram he progressed logically through the processes from beginning to the end (note the processes/stages were not drawn in linear progression and consisted of many branches). He did this mainly by pointing.

It was immediately obvious to both my independent technical assessor and me that this guy knew his subject very well. The only point was a minor objection from the nontechnical employment officer who questioned if we'd manage to communicate adequately with someone who didn't speak English. I didn't see that as a significant issue.

He turned out to be an excellent employee who knew his work very well. And within six months his English was good enough not to be an issue.

In summary, if AI is a problem now then it will be more so in the future and that needs to be tackled. That said, I find it hard to accept that an experienced interviewer can't quickly cut to the core even when an interviewee puts up smoke screens and distracting noise.

Like, Popper's falsifiability it doesn't take much to knock a BS or inconsistent argument off its pedestal when one has enough data, refutability usually works well.

That said, one must always keep in mind that just because an applicant can get some things wrong it doesn't mean that he/she isn't the best person for the job. All that tells me is that I have to probe a little deeper.

Edit: asking awkward or trick questions usually ought to be avoided. An interviewee is usually already under considerable stress and can easily be flummoxed when asked them (even if they know the answers they can bind up and not spell them out).

Usually, it's immediately obvious when an interviewee doesn't know an answer to a question or is struggling with it because of stress—body language alone usually makes that abundantly clear. It's then best to gently segue to another question. If the question is really relevant to the job then one can return to it at the end of the interview when the person is more at ease. By then, it's pretty clear to the panel whether the applicant is in the running or not. If not, then do not return to the question.

Selecting people for a job is literally gatekeeping. The company has the equivalent of a gate (or perhaps an actual one), and most people are kept out of it by the fact that they are not employees or authorized visitors.

No part of would-be employee screening can be faulted for being gatekeeping; i.e. being that which it is.

I think performance under pressure is a virtue. An important job is never going to be entirely free from stress - you want somebody reliable who will take that in their stride, you don't want someone who will buckle when things get difficult.

I'm all for in-person live collaborative problem solving, be it on a whiteboard or just a discussion. You see first-hand exactly what the person has in their memory, their problem solving techniques, how quickly they get things, their communication skills, and yes, their ability to work under pressure. If done correctly and collaboratively, you get a dry-run of a real design meeting and have some feeling of what it would be like to work with the person.

Some people experience extraordinary high levels of stress in an interview setting. From your comment, I suspect that you are not one of them and might have trouble imagining what these levels of stress feel like.

From my experience, there is more overlap with people who generally let stress get to them, than people who are solely stressed out by interviews specifically and nothing else.

I have a relative like this. They struggle in interviews due to stress, they also struggle in exams due to stress. That's two things, I don't want to be surprised by the third at a critical and stressful time in my team.

At what point do you question an approach so easily gamed, or any derived metric?

No, it's biased against people who perform perfectly well (sometimes exceptionally well) under real, actual pressure. But who do not take kindly to the bullshit / theatrical pressure typical of the standard interview context, these days. And to companies that just can't tell the difference (or are under the delusion that performance in the latter category is in some way predictive of performance in the former).

Take home project? It's too much work to do for free.

It's more that the companies simply abuse the process in various callous and careless ways -- either by assigning projects with inadequate specification and/or unknown goalposts (or plainly unrealistic expectations of the actual time required, given these ambiguities); or simply assigning scores of these assignments (often automatically to every candidate who applies) when in fact they have no intention of even looking at the vast majority of the submissions.

Easily ameliorated by simply paying people for their time.

What a bizarre way to begin a working relationship.

Employers demanded applications instead of resumes, so candidates used copy machines. Then they wanted block lettering, so candidates used type writers. ATS systems could search for keywords, so candidates flooded their resumes with them. Then employers allowed online applications, and the floodgates opened proper - applications from all over the world, desperate for good employment. So they made you create an account, and applicants wrote or used bots to automate it - which became site features like “Quick Apply” on LinkedIn and Indeed. Practical interviews used to be a single round for only the most senior roles, but then employers wanted everyone to be as qualified as those roles but not pay them that much, which started the cheating arms race.

Ultimately, the problem remains a glut of qualified talent demanding salaries to pay the bills, and employers who want to pay as little as possible. The two sides are adversaries by default, and few companies do the work of mending the relationship into something more cooperative.

This is the natural state of a competitive environment, and interviews will get “better” for candidates when there are more jobs than good candidates. But be careful what you wish for; if the job market gets less competitive for candidates, salaries and benefits and working conditions go with it.

While there might be some truth it to, framing it as an arms race is a mentality that will not help the average candidate, and might do them harm. Employers always have and always will want to pay as little as possible, obviously, that’s the definition of a company. But they do actually compete with each other for good talent, and they do pay a lot relative to other jobs, and some companies’ definitions of pay as little as possible is still to pay enough that employees are well compensated and don’t leave.

I hope the folks we hired are paid enough to retain because there were was absolutely no glut of talent. It was hard to find the folks we wound up hiring. There was a glut of applications, yes, but the vast majority were totally unqualified.

My perspective on this issue is that the reason there's an abundance of candidates in any given application pool is because of a mismatch of expectations between employees and employers, even today. Employers have cultivated an environment where candidates must job hop frequently to grow their earnings, because internal advancement has dried up and employers haven't respected loyalty in decades. This results in a plurality of the workforce "always looking for work", because they see no potential in their current role for growth or advancement; in fact, the whole reason they applied to your role is because they wanted that growth or advancement, and now they have it, so it's time to apply elsewhere for the next step up.

I get that this environment totally sucks for both parties, but it's important to note that employers are the only ones who can change the system, since they hold the bulk of the power in individual negotiations. Unfortunately, the current attitude seems to be more of the status quo: more technology to "make it easier to find the best candidates" (thus removing humans from even more of the hiring process), forcing existing workers out (thus increasing the applicant pool), and then bemoaning about their problems on Business News ("too many unqualified applicants") or forums.

Applicants didn't create this mess, and we can't fix it.

Applicants don’t have to apply to Google or Facebook for jobs they’re not qualified for, and if they stopped, the mess you’re referring to would disappear. At least at Google and Facebook’s doors, anyway. It might move somewhere else simply because there are a lot of people (literally everyone) looking for jobs that pay well.

It’s funny to tell someone on the other side of the interview table that they’re confused, when they’re giving you information that can help you. Knowing that there is an abundance of low quality applications is useful information. Mismatch of expectations might be an accurate generalization, but arguing the expectations of applicants isn’t part of the problem is not accurate at all. Applicants definitely aim high and spam companies with resumes for jobs they’re not qualified for. I even tell new grads to aim a little higher than the exact job description, but within reason. To be fair, a lot of people don’t know what’s reasonable and don’t know how competitive their cohort is and they have little to lose by taking their chances.

The lore about having to job-hop has always existed, it was true 30 years ago, and it’s still a story people tell today. Sometimes it works, and sometimes it doesn’t. Don’t get confused by hearing stories. Neither the number of stories you hear, nor the story-teller’s confidence in their take, proves anything about how the business world works. You need more first-hand knowledge of how many people don’t have to job hop. Companies do promote people, and some companies do respect loyalty. Instead of insisting it’s a fight and wallowing, go find the good jobs at good companies working for good managers and figure out how to stand out from the applicants who are less qualified than you.

Counter-point: employers don't have to pay garbage wages while demanding FAANG-quality candidates, thus focusing the talent pool on the few companies that do pay wages that might help candidates reach into the bottom of the Middle Class.

> It might move somewhere else simply because there are a lot of people (literally everyone) looking for jobs that pay well.

That is the (unspoken) point I've been trying to make, without being so overt about it. People need more money. Prices keep going up, productivity has skyrocketed, but wages have stubbornly refused to keep historical pace in the long view. The need for ever-growing sums of money just to survive is a key driver in the constant glut of applicants for jobs, one that is exacerbated by the staunch refusal of businesses to pay appropriate wages outside of the tech sector.

> It’s funny to tell someone on the other side of the interview table that they’re confused, when they’re giving you information that can help you.

They're really not, though. Employers whine about a lack of qualified domestic candidates as justification for outsourcing jobs or importing migrants on precarious visas, but then also refuse to engage with local Higher Education institutions to "properly train" the candidates they need or contribute to on-the-job training. As the age-old HR koan goes: The CEO asks HR, "What if we pay for job training and they leave?", while HR replies, "What if we don't pay for training and they stay?" There is no zero-sum game being played, here, and the actions of employers do not align with their verbal complaints about the state of job applications and the labor market. They're complaining the basement is flooded while leaving the firehose turned on.

> The lore about having to job-hop has always existed, it was true 30 years ago, and it’s still a story people tell today.

Yeah, nah. This is a recent trend, and a very recent narrative. In the 2000s and early 2010s, I was still being actively penalized for the appearance of job hopping, in the form of lower salaries and limited upward mobility. That really only began changing in the 2014-2016 window, when suddenly it became more acceptable as most millennials and Xers hadn't exactly had stable careers like our predecessors. In less than a decade, that shift from "job hopping=bad" to "job hopping=acceptable" accelerated to "job hopping=required if you want to survive and thrive". That attitude only came about because the cost of everything kept going up, and up, and up, while our wages and upward mobility stagnated if we stayed put. Loyalty hasn't been rewarded since the 80s, but especially in a post-2008 world, it's been a hindrance to individual success and is solely the fault of employers, not employees.

> You need more first-hand knowledge of how many people don’t have to job hop.

I have plenty of first-hand knowledge from my own career, and have second-hand knowledge from the careers of my parents' circles compared to the careers of my senior mentors (Xers), colleagues, and juniors. Add in the data curated by all the other groups trying to sell their own narrative, data gathered by governments, and just casual observations of both domestic and global employment trends, and this isn't exactly a difficult conclusion to reach by anyone reasonably well-informed and suitably objective about it.

> Instead of insisting it’s a fight and wallowing, go find the good jobs at good companies working for good managers and figure out how to stand out from the applicants who are less qualified than you.

It is a fight, though. A fight for everyone to be able to make rent, pay bills, and have a decent quality of life on a full-time job. That's the fight. Such a lifestyle has increasingly been constrained solely to those in Big Tech, especially in urban areas, and it's why there's such a fierce resistance to arbitrary RTO mandates and a willfulness to cheat on exams. Your retorts to me essentially boil down to the age-old propaganda of, "Workers should know their place and follow the rules for their own good," which only works when the rules ensure their basic needs are met. That is not the case, as can be evidenced from even a casual glance at current global events. Populism doesn't arise from peace, love, and good times for all, it arises from a broken system that doesn't satisfy the needs of the masses. That is why hiring is broken, in a nutshell.

A good leader understands the system they operate within, its flaws as well as its strengths, and what is or isn't within their ability to fix or improve upon. Individual managers and employers simply cannot fix what amounts to a broken system rooted in short-termism. Sure, working with recruiters can ease the burden, but that also comes at a substantial cost. The point is that the usual refrains in self-help books or "visionary thought leaders" don't work when the system itself is at fault, and that's very much the case here. It's not possible to eliminate cheating solely through discipline; rather, the system must be amended such that there's little to no incentive to cheat in the first place. Right now, there's every incentive to cheat the system when it could lift you out of poverty or give you a substantially improved quality of life, and that's why it's on the rise professionally today, just as it was in academia not a decade ago.

Desperation is the root cause of a multitude of negative outcomes. To reduce those outcomes, you must reduce the amount of desperation in a society.

I do agree that for any given job a large chunk of applicants are people who have no business applying (IE barely capable coding themselves out of a paper bag or didn't read the job description), but there's always a long tail of plausible applicants if the job isn't a trash posting.

Except you don't always simply know they weren't qualified. Only that they didn't pass your filter.

I know, I know: In many cases, it's reasonably clear the candidate simply isn't even ballpark qualified. But when we get to "maybe" territory, the recall-precision gap, and hence room for false negatives, is also quite large.

Where my sympathy dissipates, though, is the knowledge that workers are merely responding to the ever-increasing demands of employers. It's ultimately up to the employers to fix this mess, because they're the ones who created it in the first place. It might mean an in-person career event to weed out remote candidates, if you're adamant about a local hire, with humans reviewing resumes in-hand, in real time, absent ATS and AI systems getting in the way. It might mean forgoing practical interviews entirely, in favor of better verbal interviews that assess skills and competencies, with hypothetical questions AI can't quickly or reliably answer.

And it also might mean rebuilding company loyalty and incentivizing people to stay instead of leave. It might mean reducing the amount of jobs you're posting externally by resuming promoting internally again. It might mean dangling carrots that keep workers from jumping ship every two years to grow wages in a meaningful way.

Point is, there's a lot of ways to address this, but the current attitude is "MOAR TECHNOLOGY", which doesn't seem to address either side's core grievances around the hiring and employment processes.

That said, I suppose I should amend my conclusion: no single employer can fix this either, as this problem is the collective result of businesses forcing shorter cycles of tenure in general; short-termism, in other words. Your company can make it easier on yourselves by retaining a great recruiting firm or retaining good hires for longer, but you'll still be inundated with a glut of paperwork until and unless other firms do the same thing in large enough quantities to affect real change.

If everyone is struggling to keep their heads above water, those dwindling few manning the lifeboats are undoubtedly going to get swarmed.

I know the price: Instead of an ad on LinkedIn (100€ per recruit in 2019), we now pay recruiters 20% of the yearly salary.

In France the “salaire brut” is 60k€ → We pay 12k€ before an employee starts working, excluding the time WE spend in interviews. That’s a 12.000% increase over a simple ad.

> no single employer can fix this either

Yes, thank you for diagnosing the problem! For me, it’s due to LinkedIn. They know perfectly well that bulk-apply-in-1-click is hell. They could fix it in one day.

The only way to stop people from blasting resumes is to make it cost them something, for example charge $5 — which goes to charity — to apply for the job.

That's just more short-termism. It ignores the desperation of the masses for the Middle Class lifestyle they've been promised, in favor of yet-another-hurdle to be jumped over for a chance at a job they need to survive. All you'll end up doing is encouraging further fraud with such a proposal.

So what can we try instead? Well for starters, we can focus on reducing the need to seek external applicants in the first place. While labor mobility is critical to a healthy workforce, it's also worth noting that most workers would rather have stability over mobility, all else being equal. People want to work to live, not live to work - and having a reliable job, with a reliable wage that keeps pace with inflation, that enables them to buy a home and pay the bills and maybe take a vacation every year, will keep them from seeking other opportunities elsewhere, thus reducing the overall applicant pool for new positions. Workers are surprisingly fine with their excess value going to Capital, provided they're not having to decide between paying for insulin or rent this month because they haven't had a raise in three years.

In addition to decreasing applicant pools by improving job stability and longevity, we can also stop placing arbitrary requirements irrelevant to the position at hand. This sounds counter-intuitive - if you remove the degree requirement from a role, you'll have more applicants - but the reality is that oftentimes workers will pass up roles not requiring a degree because they'll perceive it as being more "junior". This isn't a great idea for every role or business, obviously, and YMMV, but it can be a different way of thinning the herd.

Then there's the tried, the true, the oft-reliable:

> A lot use recruiters to reach out to talent, that is how I got my current job.

Look, in case my comment history doesn't make this glaringly obvious, I'm not great at the whole social networking thing. Recruiters connect highly-effective introverts like myself, with employers who don't want to rely solely on word-of-mouth referrals (especially in an era where said referrals are openly traded or sold on forums like Blind). All of my big career moves came through external recruiters, of whom I'm profoundly grateful to for opportunities I wouldn't have had otherwise. While I'm generally against outsourcing as a practice, the fact is that a good recruiting firm will save your teams time they can use on meaningful work, and only present candidates to you they feel will meet your needs. Whether or not the employer will trust the recruiter is a topic for another day (I've lost count of the number of times my recruiter, myself, and the customer's PoC all thought I was perfect, but a Director somewhere didn't want to trust outsiders), but it's still arguably one of the better investments a company can make in talent acquisition.

Oh are we talking about minimum wage jobs, labor & the proletariat now? I thought we were talking about software engineering, where the average salary is very comfortably in the 6 figure range in the US. If you’d rather be a plumber or a teacher, go for it, I hear the interviews are shorter. ;)

Why would charging a small fee for applications encourage fraud? I don’t see the logic. The problem you complained about is the low-effort spam applications, like LinkedIn’s 1-click apply-all button. If that was no longer free, it would certainly reduce the spam and make applicants think more strategically about their budgets.

Companies already use recruiters. Recruiters are expensive and many provide low quality leads, partly because of the way recruiting is incentivized, and partly because good software engineers don’t have to move around that much.

How do you plan to defraud the banking system when you need to pay $5 to send in your application?

The issue is that it takes almost zero effort to apply for a job so there needs to be proof of work on the applicants side. The easiest proof of work in through our banking system because there is massive amounts of effort to prevent fraud and ensure interoperability.

Alternatively that proof of work could be that you need to physically drop off your resume like in the pre-internet days.

The same need to go for the company side, they need to have skin in the game too when interviewing a candidate. This is why I am against automated tests which are used at places like Amazon.

And if the employer posts a real job but the candidate doesn’t get a response, or is immediately rejected, will the employer get accused of wire fraud?

Sounds like a nightmare for employers.

As an employer, I’m flummoxed that a recruiter with a human fibre is required. It means instead of paying higher salary, 20% goes to the pocket of the intermediary the first year.

Hiring low-level employees requires less vetting. Upskilling for more senior roles within your own company means you get to skip things like cultural fit (because they presumably already are), work history (it's in the employers own records), and references (that would be the coworkers and managers they already have).

Pay enough to keep the ones worth keeping (and increase it with time, to match inflation at minimum), upskill employees, and promote from within, and this problem should pretty well fix itself

We're called freelance consultants. And we charge a high price for that service.

It was, by far, the most boring interview I've ever had to participate in. The leetcode problem was easy (detect if two strings are anagrams; solved with O(n) solution in < 10 LOC; then aggregate sets of anagrams). System design was easy because it is to the point that it is largely "formulaic". It feels like those rhythm games where you need to know when the trigger is coming and the pattern to hit those triggers (talk about functional requirements -> talk about non-functional requirements -> write some API routes -> draw a load balancer (because of course, there's always a load balancer) -> draw a service -> draw Redis/Memcached -> draw a database). It's comically laughable how it's just like a rhythm game at this point.

What I don't understand is why teams don't just "get to the point". A lot of times, the assessment is looking for specific notes. Why not just ask about those notes directly? A friend recently went through an interview where he was docked for not using C# `yield` when implementing an `IEnumerable`. But had the interviewer asked "What is `yield` and why would you use it?", the interviewer would have gained the same exact insight in 15 seconds with one question.

I think what's also striking is how slow orgs are to adapt their interview process to account for AI. There's a distinct lack of focus on code reviews as part of the assessment. If the future of coding is AIs for productivity, then teams should be adding code reviews into their process (check for correctness, check for defects, check for performance, check for security, etc).

"Writing code is easy, reading code is hard"

An interview based on reading, explaining and critiquing complicated code would give a better window into a candidate's skills than having them solve the kind of coding challenge that can be completed in a short interview.

Also, being able to identify which algorithms would be useful in solving a unique problem and explain what the pros & cons are would go further than implementing an algorithm abstractly.

The guy gave me 30 lines of code, and asked questions about it. Including finding the questionable constructs in it.

Interviewer had a small API. Found missing error handling, missing validations, poor choice of ID column type, missing index for a high volume read query, possible XSS vector.

Loved the format so much I ended up making a small FOSS tool for code review as interview: https://coderev.app

    "Imagine that this code was written by a junior dev; review it and tell me how it can be improved, where the problems are, and other feedback".

    "We just hired a new intern and he pushed this PR.  I need you to help me review this code for issues and make sure it's production ready".

And these are all candidates with 5+ years experience, sometimes 10+. I hate the leetcode gauntlet, but I also am really surprised how much worse candidates are now on average that they used to be (I’ve been around for about 25 years now), and how the minute you try to dig a little deeper to see if they know how things really work, the more it seems that they have no understanding whatsoever of the basics and somehow treat computers as magical black boxes.

I do understand I am privileged by having grown with the technology, starting in the 8-bit era, through an electronic engineering degree, and now in k8s etc. but with so much educational material available for free you’d think that even people in their 20s/30s now should be able to have a better idea of how things work.

if not Counter(str1) - Counter(str2): # success, they are anagrams

  str1 = "a"
  str2 = "ab"

Counter(str1) - Counter(str2) | Counter(str2) - Counter(str1)

That ought to work. I'd hope Python's caching doesn't recompute the Counter for long strings.

But the catch is... the people doing the hiring must themselves ideally be talented, energetic, skillful mentors who envisage building a positive culture.

On the other hand, from my perspective and experience on both sides of the interview table, to me it seems like the perception that this is adversarial is coming primarily from young candidates whinging on the internet in large numbers about interviews being unfair or “broken” or doing it the wrong way. Every time this comes up on HN, there are hordes of comments demanding we stop whiteboarding in interviews, or claiming anything not exactly in the job description is out of bounds, or decrying the abuse that coding live inflicts. There are blog posts every week about how to fix the problem, where the problem is usually the interviewee’s comfort, and not the company’s efficiency or effectiveness.

The truth is that companies need people, and the employees there do want new colleagues. If you truly understand that, and also that the selection process is unavoidable and working as designed, then it will help you navigate the selection process and appeal to the company, it will help you get the job.

I’m well aware of the fears young candidates have, since I was one. Getting a job can be hard, and it might even be getting harder over time. But to me it’s also funny that software engineers looking for 6-figure salary jobs complain about interviews and competition and having to demo their skills for an hour or two. Don’t forget to look around at other industries and ask whether it’s really better over there. People in music and films and advertising and art have to grind for years, suffer low-wage or historically unpaid internships. Doctors have to do residency, grad school, and get licensed. Most people make less money, many people have to do labor. Most jobs have much bigger downsides than what we have to do in our interviews as software engineers.

We have opportunity and privilege and with the right practice and attitude and with reasonable expectations, it’s really not that hard to do well in interviews and land a good job. Many kids are hearing stories of FAANG and million dollar salaries and expecting to somehow land their first job out of school in one of the top 5 most coveted and competitive companies. I know this because I just spent a day at a university career fair a few weeks ago talking to hundreds of graduating seniors, most of whom are all expecting to shoot the moon for the same 3 jobs, and most of whom are not very eager to consider the idea of starting at a smaller company and working their way up for a few years. FAANG feels adversarial only because it’s actually legitimately super competitive, but as a candidate, figuring out how to have a non-adversarial approach is the key to getting the job. Tools like this are the opposite of what they need.

these aren't justifications for the confrontational interview filter, but some upstream factors that are worth picking at to see if something better could become a solution to them.

You say that as if there's The One True Interview Routine™ and it doesn't change from team to team, person to person, month to month. I actually would bet that no single organization interviews with the consistency you're describing, not even FAANG with their holier-than-thou grinders

If it wasn't such a monster drain on human productivity, I think it'd be funny if there was "rolling interviews" where each new hire got to send prior hires though "the process" on the yearly anniversary of those prior hires

Lots of eye scanning while looking above the window they’d been typing into. Pauses. Big pastes. One of my excellent colleagues noticed that the candidate made use of exciting C++ casts before ever defining the variable’s types. Complete inability to explain or debug the code just written.

So. Frustrating in two dimensions. First a waste of time for everyone. Second, occasional signs of real ability make me think the candidate might’ve made it work honestly. The fool.

This, and this alone, makes me pine for in-person interviews. But I suspect those won’t be back for some time. (For good reasons that are out of scope here.)

I believe live coding is a scourge and we should get rid of it altogether.

Get a tiny open ended project, which requires more design/architecture skills than actually finishing the solution. You do it at your own time and shouldn't take more than 1-2h. You submit it and provide your mental process in writing.

We then hop on a call and we can dicuss it further, talking about possible issues and what to do to get it released if a requirement changed last minute.

If you can do that, come across amicable, knowledgeable and experienced enough given the seniority of the role, and feel like you are not reading from an hidden ai window then you pass.

"Exciting casts" are absolutely worthless in real life, everyone has to balance shipping vs code artistry.

I would take an hour of live coding any time over a take-home task that is one hour only on paper and takes you three in reality.

Engineering is collaboration, and it's fair to expect candidates to talk, solve problems, and explain their solutions.

Where we could do better as an industry is the type of problems we give to people. I'm not a fan of LeetCode-style questions, especially when multiple ones are asked. Something closer to Earth would be better. But even if you have to ask something algorithmical, I'd prefer the style of "Advent of Code" exercises where the same problem has variations and multiple levels of complexity.

That's how Amazon does their interviews.

I used to really like the idea of take homes, then I had kids and discovered just how hard it is to find time to do these (particularly for multiple applications while working).

Don't get me wrong, I think that they're generally a better test for the actual role, but they're not perfect, and they're game-able by spending way more time on them.

Interviewing is hard. Like, ideally, we'd just centralise the administration of these tests and you'd do a few every year (so formal CPD rather than an expectation that people do it in their off time) but that is never gonna happen in the tech industry for various good and bad reasons.

Your proposed alternative is absolutely subject to the same problem, AI-written code and LLM-written mental process. Think more carefully about whether it solves anything and exactly what problem it solves, because it does not solve the problem of people trying to game interviews with AI, your suggestion is potentially worse on that axis. The whole point of doing it live is to get to the talking part that you suggested more quickly.

Wouldn’t you rather be discussing your thinking immediately and get a sense for what the real questions are and how you’re doing, and have the opportunity to talk about it, than waste several hours doing it at home trying to write something up without getting clues along the way or being able to ask questions, only to have the next guy get hired because they used AI, or because they’re fine with live coding, or because they’re better at writing than coding, or because they got lucky?

You have to live code on the job. Do take a moment to consider what the hiring manager needs and whether take-home problems are either more effective or more efficient for the company. Managers want to find the best candidates, and best involves more than coding, it involves communication and collaboration skills too. Live coding can give some indicators of those things, and take-home problems cannot.

Another one I like was having to code review a repo which had specific issues in it.

Then add some new features.

Then in the final stage of the interview you go through it.

Nope.

100,000 applications. In under a single day.

With that kind of applicant pool, I’m honestly not sure what the best approach is—even though, in a perfect world, your suggestion would be the more appropriate route. The reality, however, is that these numbers are just absurd.

Start interviewing people, and once you have enough people who pass your bar, you tell the rest tough luck.

Secretary problem.

That said, your comment reminds me of a Monte Carlo algorithm I think I’ve heard about. There is a way to have some statistical confidence in getting the top K out of a sample of N without examining all N samples. I’m blanking on what it’s called, and I think it’s related to Reservoir Sampling. I don’t know if I read this or am making it up, but my instinct is that you can get to high levels of confidence after looking at sqrt(N) samples.

Say you want to take 10% at each round. You have to do this three times to get 100 people, and then another round to get the top 30.

If you tighten the criterion, noise matters more, so you will drop your actual best candidates, since people need a high skill+noise. But if you try to keep the unlucky "good" candidates by having a wider band, you pay more.

Also, if you are successful at finding the best candidates in a batch, noise matters even more the next round.

I've been thinking about how to vizualize this. I have it in my mind and I'll try to describe it.

You plot a bunch of points with S, N for each candidate. They are independent, so a scatter plot looks like a 2D gaussian. Lets say skill is vertical, and noise is horizontal.

You want to find the 30 highest points, but you aren't allowed to just look on the scatter plot and select the top points. You have to draw a line S + N = c, and choose the person with the highest c. Basically sliding a ruler out that crosses the S and N axes as far away from the origin as possible.

Observation: if N is high (wider distribution) compared to S, you just get the luckiest people. If S is high compared to N, you just get the most skillful.

If you want to reduce noise, the way to do it is to have more independent measurements (interviewers), not to stop interviewing early.

The good news is that there’s no such actual thing as “best” in this situation and people have many dimensions, they can’t be ranked perfectly, but they can still be ranked approximately. We also don’t need to get the exact 30 people out of 100k people with infinite precision, we will get an amazing set if we can take a random sample of the top 1000 candidates out of 100k candidates. Having 100k candidates gives us the opportunity to end up with a selection from the top 1%, say, whereas if the number of applications was 40 people for 30 jobs, you might be stuck accepting people who are below average.

Instead of a "noisy" sieve, you're just using the time of application as your sieve. Which, unless your job opening values the skill of submitting lightning fast job applications, is pretty much 100% random noise.

You say a sieve approach drops the best candidates. Your approach also drops the best candidates.

And if the bar is actually so high that only the best candidate(s) qualify for the job, then your approach would imply you interview an expected half of the applicants before you fill the positions.

It's pretty obvious a "noisy" sieve (with some signal) is better than sieving by application time or sorting by arbitrary order and taking the top N applications. You don't have a magical solution. The people handling those 100000 applications are not that stupid.

If you're already reasonably confident from the first batch of people what the potential in the pool is, why would you wait? People have work to do and they want those people in as early as possible.

I'm not saying you will do much better sieving by time, in terms of quality. But you will save all the effort of looking at the people at the back of the queue, for little loss.

Sure, half the best people will be at the back, but what is your loss on taking the second best people earlier? Chances are you won't even be able to tell.

In this particular case, MS didn’t have to wait for people to show up, they got 100k candidates before they could have even made a decision on a small subset. And in general, the problem being discussed in this thread isn’t having to wait, it’s how to deal the flood of applications coming in too fast.

* huge edit to this comment after this rattled around my head a little more: actually, duh, sieve and early stopping are completely orthogonal, this is not an either-or situation, and you are probably assuming a sieve in your proposed approach. You can sieve 1 candidate at a time, since it’s a series of criteria ranked in order of how much time each one takes to evaluate. If someone doesn’t meet minimum requirements, they’re rejected quickly and not invited to interview. If you’re going to stop early but you have a lot of candidates to evaluate, then you still have to sieve. Like, I’m pretty sure you’re not proposing to conduct 37,000 interviews for 100k candidates, right? Even if we stop before looking at all candidates, each candidate will have some early criteria used to cull them, and only the ones that pass all the early criteria are invited for an interview. That’s true whether or not we stop early. The sieve is a given and unavoidable. The only question is whether to stop early, which we would only do if it solves a problem. We can, if we want, interview the same number of candidates either way. The sieve allows you to look at all candidates efficiently, but does not require it. If you do look at all candidates, then the more people in the top of the sieve, the statistically better the final interview pool will be. Stopping early is valuable only if the applications process is slow (which is not the case here) or if the early criteria take significant time to evaluate.

You should see the number of people that apply to college.

Are you expecting people to not reference external material, such as documentation or syntax, and only code from memory?

I also tell them that they should tell me when they're checking external resources (the CPP reference is what I mostly have in mind, but StackOverflow or whatever is fine with me) so that I understand how they go about solving problems. Because that's the actual important bit. All the coding and the questioning is a proxy for that.

an example of this, please?

It barely made sense when these were conducted in person, and it’s completely inane over video meeting.

Instead of grilling people, talk to them. Give them meaningful take-home exercises that you can discuss together during the interview. If they can explain the decisions they made and discuss alternatives and trade-offs, that’s a much better indicator of job performance than pretending to invent a leetcode party trick algorithm in 10 minutes.

Both the candidate and the interviewer need to take time out of their day for the interview. If it's an in-person interview, the candidate also takes travel time etc, which could be more disruptive.

But it takes almost no time to give a take home problem, so there's potential for a significant assymetry in time spent between candidate and employer.

FWIW, I've interviewed almost a hundred candidates in the past couple years and I think the number of candidates that I thought might be cheating are in the 3-5% range. It's fairly low, but if the take-home was the main thing that determined whether the candidate was hired, I wouldn't take that 3% chance.

I let candidates use X during coding interviews.

I've personally seen this evolve over the past 20 years

Whiteboard --> Text editor --> IDE --> ...?

Maybe "AI Coding Assistant" is next.

If a company wants me to do some leetcode crap I decline the job.

How do you find an apartment rental in a large city with thousands of options?

You can’t expect to visit them all. And you certainly can’t come up with increasingly bizarre torture tests for landlords to pass (“I might rent from you if you can answer enough trick questions about tenant law within 15 minutes”).

To find the apartment, you probably sample enough of the options to form a picture of what’s available, and then make a decision knowing that you didn’t have perfect information but you had to move ahead. The same applies to hiring.

But if you can give up on those worthless power games, you can have a hiring process that’s better for both parties.

How about you hire the first 1000 people who seem competent, and then tell everyone else you've filled the roles?

Companies already know about cheat methods like this and candidates still have to demonstrate skills in-person on a whiteboard in a 2nd round of tech interviews.

A high-paying FAANG job isn't going to hire candidates based on just one remote Zoom tech interview.

That still may not deter some folks and they'll try to continue to find some cheating technique to use in front of the whiteboard. In that case, some creativity is going to be required. E.g. : https://www.google.com/search?q=chess+cheating+anal+beads

The issue is that "chess board moves" is very low bandwidth and can be efficiently encoded into pseudo-quasi- Morse Code vibration schemes. However, if you're asked to "invert a binary tree" or any other open-ended random puzzles, it's more difficult to compress a secret 2-way high-bandwidth communication with ChatGPT into a hidden butt plug. (But then again, if a candidate is actually able to fool people with hidden ChatGPT hacks in a face-to-face interview, maybe FAANG should hire them based on that alone.)

Keep on the lookout for a github repo with ChatGPT butt-plug communication.

Source - work at a FANG which engineers exclusively via remote video call.

As a first engineer for a tiny startup, show me some past projects and let's grab a (virtual) coffee talking tech, that's all.

Now it's like McDonald's requesting to perform Michelin level dishes under 45 minutes with Gordon Ramsey in the room while still ending up flipping burgers minimum wage at the end of it all.

If this keeps going and getting worse, I wouldn't mind a butt-plug communication, as long as I can pick my favourite AI (I personally kind of prefer Claude right now over ChatGPT.)

even if its not an app, everything is so cheap and small now, 1mm camera and microphone, and very tiny ear piece (invisible with longer hair), or even earless headphones, realtime whisper turbo and groq llama3.3 can absolutely pass most interviews and answer most questions

even without the tech, i am sure people are now cheating on face to face interviews as well

i worked at fairly big company, and did maybe 500 interviews, i never knew how the candidate would look in advance, so i dont see why cant someone else shows up to the interview, pass it and then the real person shows up for the job

What do you think they do? One round of interviews was enough for my Google recruiter to congratulate me on getting a job there.

Did I get the job? No. But the process didn't include any other interviews. There's only one round.

Your Google interview process might be different because you've previously said you were ex-FAANG from 2000s so your work history and street credibility puts you in the "experienced senior hire" bucket. The interview process may be different for that type of candidate.

My recent college grad friend's process went like this (2022 so post-COVID):

- an offsite remote interview over Google Meet using Google Docs as coding canvas. 45-minutes long

- 5 onsite interviews on same day: 4 technical and 1 behavioral

1. The recruiter sent me email suggesting I reapply to Google. She asked for my resume and put me on track to apply for an entry-level position.

2. I had a phone screen. (1 interview, over the phone, graded to a lower standard.)

3. The recruiter informed me I passed the phone screen and scheduled the round of interviews. (5 interviews, back to back over one day, over some kind of internet platform.)

4. I had the round of interviews.

5. The recruiter informed me that I had passed, excitedly wished me congratulations, told me to expect a job offer within six weeks, and told me that I'd be having some "team fit" interviews over the intervening period.

6. Weeks of radio silence.

7. When I inquired what was going on, she asked me if I could provide an altered resume explaining my employment gaps.

8. More weeks of silence.

9. She called me and informed me that although I had passed the interview process, my interview scores were not high enough to qualify for a job.

As to the question of whether there might be more than one round of interviews, I complained about this treatment to several people who worked at Google, and I've been told by multiple of them that the recruiter was correct about the process and there are no further rounds.

Have FAANGs gone back to real onsites? I've only see virtual onsites since the pandemic.

But I predict this won’t do what the authors claim: it won’t fix anything, and it won’t change technical interviews that much. At most it might increase the number of companies that require remote candidates to fly in and interview in person to do the same practical coding problems they used to do online. Which would be the standard from just a few years ago, anyway.

Practical coding problems (also known as work simulation interviews) are the most effective and least biased tools available for evaluating candidates. Not every hiring manager does a good job using these tools, but if you see misuse, take it as a sign that the manager might not be good to work for.

I have trouble understanding the authors thinking here that anything would be worth the cost of this: it’s a trust-busting tool where people are deceiving their potential employer. Trust is foundational to everything, in work and in life. We need more trust in our society, not less.

Had something in the works when I first got to know about multimodal GPT and electron made it super simple to prototype something. Refined it a bit to even do the invisible window thing after feedback from my peers. Didn’t have the ahem courage to do it.

Also recently noticed there are actual companies around this space who are marketing this off as “interview assistance”. Two of them I’m aware of:

1. https://www.senseicopilot.com/

2. https://www.finalroundai.com/

——

I’m sure the knee-jerk reaction from the tech industry to these tools would be:

1. Return to in-person interviews

2. Force install spyware/vanguard like crap to scan your whole personal computer before doing the interview

If you are a tech leader/senior engineer with influence on these matters, I *HOPE*, that this is not the path you take. It’s high time FAANG and others stop gatekeeping with these useless rounds that are equivalent to forcing people to hand-solve complex math equations instead of using calculators. I also partly blame the YT influencers who keep peddling this BS but they will stop if the companies themselves take a stance.

You can rather use the same 1 hour slots to have a conversation, pair program on a realistic problem required for the role you’re hiring for where you and the interviewee can both benefit from the knowledge transfer. I’ve tried this in many of the interviews I’ve conducted and always went away learning something from an excellent candidate or a candidate who didn’t meet the expectations, going out knowing where they fall short.

People who professionally cheat on online exams just use an HDMI splitter/capture card to inject whatever image they need using a second computer. It’s a lost game.

For offline interviews there are hidden earpieces etc.

On-site interviews should be the standard path to take to detect the frauds and cheaters before they become potential bad hires.

Online technical assessments now have little added value given that AI tools like finalround.ai, etc just make it easy for frauds to cheat the technical interview.

This is actually a strong argument to a return to on-site interviews and should just draw the line on this and detect the frauds altogether.

I'd rather be rejected than got job I'm not qualified to do, that's why I never prepare for interview above studying the actual company- and that is more to give me proper signal if I want to work for them, I want to be motivated to make move on more than just money.

I also have never done leetcode, though I like competitive programming problems, especially bot programming contests on codingame.com

As many people have pointed out, the interview process has diverged from real-world, day-to-day tasks you would be expected to accomplish once you get the job. Not actually being able to do a leetcode test (either due to lacking knowledge or interview stress) might not have any reflection on someones ability to succeed in the role they are interviewing for. How many people are dealing with sorting algorithms or traversing binary trees without access to the internet as part of their job?

You and me both. I just read a bit about the company before applying for the job. In the interview, I ask the interviewer only two questions: what’s the management style? Is it a servant leadership style with autonomous work, or is it a hierarchical task-based one? The second question is, what does an average day-to-day job look like? Sure, there are few stressful days and others are relaxed, but on average you can tell. Is it chaotic, always running against deadlines and having new tasks thrown at you, or is everything sorted out properly? Sometimes, I also ask about the meetings. I personally don’t like useless meetings. A lot of middle management uses meetings as part of the power dynamics and exerting power rather than actually discussing and solving issues.

It's no different than if you had a much more technically competent friend off-screen feeding you the answers.

For larger orgs at scale, often the interviewer needs to stick to a script, ostensibly to avoid personal bias (which probably does happen to a degree), and to make scores between interviewers comparable. In such cases it might be difficult for the interviewer to argue that the candidate has done well despite "not getting the answer correct".

A lot of the grievances people have for the interview process are artifacts of scaling the interview process for the largest tech companies. As for the smaller ones, it's basically good ol' cargo-culting. Small startups that follow FAANG processes probably don't realize they can customize their hiring process to their advantage.

I am sympathetic to the argument that software engineers should be licensed, which would reduce the need for dumb technical interviews. But I imagine HN wouldn't like that very much.

Oh yea, HN hates this idea, but I'm sure it would cut down on all the screening, FizzBuzz, hazing that happens during interviews. If you had a - basic, barebones - certification or exam, that all software engineers must pass, that at least says "This candidate can function at a very minimal level" it would at the very least filter out the 50-75% of candidates who literally cannot code or even speak coherently about the basics of programming.

I think people here are really in love with the romantic ideal that someone with no college, no credentials, no formal-this or certified-that can (in theory) jump right into a senior FAANG job. Yea, that's great I guess, but in practice does it really work? It seems to me that in practice, it just means every company gets thousands of totally unqualified candidates every time they post a job offer, and this helps nobody.

How much would it cost e.g. Google to start an independent certification company, with branches in 20 different cities across USA. The top candidates could be also offered a job at Google, but every candidate would get a certificate confirming their skill level that many other companies would probably be happy to accept.

At first, the testing could be done for a symbolic cost, because Google already spends some money to interview its candidates anyway, and this would also serve the purpose. But later, when many companies start accepting the certificate, you could let the candidates pay the full cost.

The first rounds of testing could be done at a computer (provided by the testing company), you would just make sure the candidates are not cheating. Only those who achieve a sufficiently high score would be later interviewed by humans. This would be way more efficient than interviewing everyone individually.

How useful that stuff is/would be in practise, I don't know.

Doctors go through 5-7 years of residency with long hours, low pay, and a full fledged physician watching them who can fail them. And that's after an undergraduate degree and medical school.

If you want to endure a 5 year miserable grind just to avoid interviews, go for it.

One time I asked my optometrist what he was doing when he left me in the examination room for 15 minutes. He said he was looking up the details of the medicine he was going to prescribe me. I don't consider that incompetence, and it sounds a lot like what I do when I need to know about some technical issue. Everybody in my industry knows that Googling stuff is a huge part of the job, but you're not allowed to acknowledge that in an interview.

I think it is actually an interesting and relevant question: why don't past performance or job references play more of a role in hiring, compared to technical screens many people hate and think are pointless? Checking references and former calling employers—if it's done at all—is often a pro forma final step, rather than the first step, which might plausibly be a more effective way to screen candidates.

Faking it is shirking your duties. Most bad medicine doesn't result in obvious harm/cost to a patient or a board complaint. A poorly managed medical practice that runs off support staff and makes patients wait three hours isn't going to threaten a license, and that's faking practice management. A primary care doctor who punts anything beyond the very, very basics to third party vendors or specialists is faking it.

In my major metropolitan area, general practice veterinarians are often absolute crap at doing their (required) medical notes. If you ask them for copies, they'll send you invoices. If you push for real notes, they'll use their 48 hour response window to make something up. They almost always get away with it because during future visits, the vet (whether it's the same person or not) will ask a bunch of questions to (necessarily) fake knowing what was done previously. It's obviously going to affect patient care and cost, and if you look at board complaints that aren't thrown out, the "punishment" is usually because the board saw that the vet has shit for notes. And don't even get me started on vets using AI, which results in hallucinations in the notes -- which I think are notably worse for patient care than just empty notes.

I think that's a good example of even good doctors faking it.

In my mind, faking it is learning what to say in interviews to get the job(s) without having the skills and ability to do the work.

That said, when I interview, I'm really looking for can they describe the output of a program before the write the program, and then can they write the program that works as they said it would. Also, the program should have a loop inside a loop, because nested loops happen all the time and there are too many candidates that can't deal with them.

Candidates that can't manage a nested loop or can't write a program that does what they said it would immediately prior to starting to write the program but somehow managed to get into my interview are faking it in my book. Some candidates clearly are just having a bad day, and some are probably having a bad day but not as clearly.

References are hard because a) some people fake those b) fear of litigation means many companies prefer to only provide start and end dates and maybe eligibility for rehire. Also, the modern world means nobody answers their phone so this process becomes very asynchronous.

And I'd be kind of surprised if a practice shared billing performance info. That'd be like sharing a tech support person's ticket close rate, but even more sensitive. Maybe at a very high level if HR isn't concerned with defamation claims?

It's all anonymizable.

Firms which associate disruption with profit are altogether different. No such thing as malpractice; and if there was, they’d prefer to hire the “risk taker”.

https://github.com/ibttf/interview-coder/blob/main/electron%...

In my experience, LLMs are pretty bad with low-level languages like Rust/C when you need to deal with pointers and ownership, avoid extra copies.

Another option is to ask for something impossible, as LLMs usually do not admit inability to solve the problem and prefer to hallucinate a fictional solution.

I was doing a 45 minute screener interview with a guy the other day who I could see kept glancing over at another window or screen as he was typing in code. Every couple of characters or so he was pausing looking over, then he’d type a handful more characters, then pause and glance again. He was typing incredibly slowly as well, like a character or two per second, even when he was typing. It was honestly a bit of a weird experience to watch unfolding. His solutions were spot on though.

I didn’t make an issue of it, partly because I already had other concerns as well. I can’t think of a situation where there’s anything to gain by getting into an argument with a candidate: I simply didn’t invite him to the next interview stage.

They'd made it through the HR "sift" by having the right words on their CVs but we found that a huge majority of these candidates didn't know even entry-level basics.

There was initially a "quick fire" round where we did some real basic definitions - Confidentiality, Integrity, Availability, risk, etc, etc.

Even back then candidates would hit us with the:

"Hmm, great question"

/tap tap tap tap

[Wikipedia definition of confidentiality]

Another fine reason no one should be using mechanical keyboards :-D

Technical competence happens to be much easier to test for than most of the other factors, and it can compensate to some degree for e.g. bad interpersonal skills. This is indeed part of why tech is such an autism-friendly field. But I've only seen a few people get let go for being great team players, excellent requirements analysts, etc., but just totally unable to code - for better or worse.

(More common is they get shifted to a role which better aligns with their "true" skillset. Why didn't they get this role in the first place? Again: Hard to test for.)

That's not quite where things are now, but the initial phases are beginning to show.

Someone without the confidence to hire someone without going through a bunch of hoops to justify themselves is even less likely to admit humility and say they were wrong. Which makes places that employ these interview techniques terribly passive aggressive places to be.

A colleague yesterday was telling me about how much he and all his friends lied to get their jobs, and he suggested I do the same, because if I don't, I'm competing against people who do, and said that's unfair (for me).

It wasn't even an office job, but I've seen the same thing there too. (Also at university...)

That said,

- the interviewer likely embellished or lied to get his job. Be careful.

- a lot of this thread is about avoiding a bad hire. In engineering, someone technical who can detect deception is too valuable to put in the interviewer role. Send them to conferences to “poach”.

Just like the soft skills of competing with liars, you can also improve detecting deception.

Those are wildly different skill sets and/or personality traits.

Or do you think that technical interviews hold such trait that is always present, that you can label all of them useless?

If you're planning to have N candidates do this when you only have 1 position, you're disrupting the others' lifes so much. It's like one of those chef competition shows. You better pay these people at least 6 months salary to show up for this.

Plus, do you really have the supervisory system in place to have N new hires and evaluate their impact after 1 month. Do you have the work for them to do? Or are they all going to independently work on the same project?

I ask a useless question in interviews, but asking every candidate the same question lets me calibrate the candidates as well as improve how I present the question so we can get to the problem solving and programming skills I want to see. Some questions are worse than others, and I hope mine is less worse; I've had several people tell me they enjoyed working on my question, although I also had a candidate refuse to work on it and ask for another question (which I didn't have, and I was the last interview of the day, so I returned the balance of his time and walked him out)

What I mean is, if the job your applying for is mostly about working with implementing 3rd party APIs IMO it makes sense that your tech test represents that to some degree.

Can you understand basic documentation.

Can you use data structures.

Can you write tests and understand what actually needs to be tested.

Can you handle exceptions.

Do you try and get the entire resource?

Believe it or not I’ve joined a team in the past who owned a service where no one knew what pagination was.

Bonus points if you demonstrate that you understand what a transaction is.

And I forgot to add, can you design a sensible database schema.

Can a candidate talk about those simple concepts?

Now, would I be able to do a tech test based around domains in machine learning? I would have to do a lot of research first!

"Victims"

This just makes it much worse for the industry that is already weeding out frauds or those that claim / lie about their achievements without backing them up when asked very technical tests.

I would say the online technical interview solutions like leetcode, hackerrank, etc are losing their worth given they can be easily cheated but you are just cheating yourself in the on-site interview.

Honestly, it's hard to blame the company. If the employee is willing to lie and cheat even before they were hired, it's expected that companies will take away their tool to do it: remote calls and WFH. As always, the honest worker gets shafted.

Some people think that trust just means a someone blindly trusting another, but of course that's not how it works. People will be trusted to an extent that they can be trusted, and this goes for company-employee relations too. If people prove they cannot be trusted with remote interviews, companies will take remote interviews away. This was a luxury given to us during the pandemic, and it's now being rolled back because of tools like these.

I remember people here on HN complaining about CEOs mandating RTO without data to support it. Well, this certainly doesn't help.

For some reason (legal?) we stopped and now we are doing interviews that have a way worse correlation.

All these online assessment solutions such as leetcode, hackerrank, etc are essentially close to useless given all these AI based cheaters and LLM products being used by candidates.

On-site interviews it is then, and this time you cannot use those LLMs to save you.

Other than a bit of practice it would be completely undetectable visually. It’s extra nice that we encourage thinking out loud.

We should assume people are doing this. The solution is interviews that don’t require information that can be looked up or in-person interviews or permitting the use of AI.

For interviewers, the solution will be proctored testing centers with metal detectors and all devices required to be in a locker outside the test area. (This is how the FAA conducts its written exams.)

It's also true that pretty much the most powerful AIs in existence are available to anyone willing to fork over twenty bucks a month and you'd be hard pressed to get better algorithmic outputs as a nation state with $100m to blow.

So there's an angle that suggests that this is not primarily a story about wealth inequality but rather savvy in wielding tools.

they've become scared of their own shadows.

cf. https://cft.vanderbilt.edu/guides-sub-pages/blooms-taxonomy/

Typical interview question (lower-level thinking, remember): List the primitives in JavaScript. (remember)

Better interview question (lower-level thinking, apply): When would you use an object instead of an array in JavaScript?

Really good interview question (higher-level thinking, evaluate): Here's some raw data that we'd like to put in a database, what data structures would you recommend and why?

As a teacher, it is my job to make sure my students control that AI instead if being controlled by the AI.

I am ALL for this, for a number of reasons: - will help people 'find the words' - it's pretty easy to spot someone who doesn't actually know what they are talking about, even if what they are saying is correct and valid - actual technical capability is only a small part of the job.

:D so, 'find the words', interviews can be stressful as all hell for both sides, as someone who assumes the power of the interviewer that's a heavy stick to wield over someone who is trying to explain why they very much want to be paid by you. Communication is _hard_, even if you gel with someone, actually explaining something to them in a way that both people completely grok can be really hard. If someone joined and front and centre said 'I'm using AI to help me with my answers because I find interviews hard' I'd sit up straight and be all 'cool! lets do this!'

'Are you sure about that? what did it feel like?', ok so this one is VERY subjective but it's a pretty solid baseline to go on. Interviews should be a verification that they actually are interested in working with you and that you are interested in working with them. So to be able to make that judgement you and they have to get to a point where the interview can be concluded, and whilst there's a fair chunk of other stuff to go through, technical understanding is important. To be clear, lack of it is fine, that's why different levels exist, you want to be a software engineer but you have no experience or skill? Well ok but you will be starting at the bottom. Now to that, there will sometimes be people who want to 'show a higher level of capability than they have current attained' and that's what my job is, to try and see that when it happens. It's pretty easy to do to be fair, humans experience deeply, computers do not. So if I want to talk about Entity Framework Linq helpers then I'll be looking for the spark of excitement when it suddenly pinged to them that this is WAY easier! And pretty! It makes the queries dance! ..and yes, that's my experience, but it doesn't take much to guide people on sharing their excitement in this stuff, and frankly if you get to the end of the interview and _haven't_ found that excitement or deep interest then skill level aside, you've probably already got your answer.

Finally is around how relevant is a specific level of technical capability? As I said, that's what we have graded levels for (junior, senior etc), and besides this whole bag (for me at least) is much more about how you solve problems than what you specifically know. I've seen people wildly switch careers into software and sure, there's a lot of hand-holding and stuff early on cos it's a proper throw into the deep end, but it's the mind, not the current capability.

Just talk to the person, talk about the sorts of problems you expect them to solve. Can they understand the problem? Do they have ideas on how to solve it? Maybe give them a small take home exercise related to the work they'll be doing. Discuss it with them to see if they were the ones that actually wrote it.

Sometimes you don't even need the take home exercise. If they have niche knowledge on the subject you care about they're probably good enough by default (since they are clearly invested and studious in that case).

I only hired a few people in my career but every single one was a huge boon to the team, still around even after I quit as a manager. I didn't even give them a take home exercise. Didn't need to.

Even now with LLMs generating almost real time, there is always a slight delay and we all know that LLMs are far from perfect.

It seems the tech community on one hand complains at how bad LLMs are for doing deep tech while at the same time fears them as they can make any candidate pass interviews. So which one is it?

Also, the simplest trick even old-timers professor already know:

1) Prepare an open ended question about X

2) Ask it to ChatGPT, Claude, etc.

3) Take note of the titles, keywords used, etc.

4) Ask it to the candidate

5) Compare responses

I agree asking difficult leetcode questions is less useful because it depends so much on whether people have practiced (or now whether they cheat with AI). Often these questions are so difficult there's no way you could come up with a solution in an interview situation if you hadn't already seen it.

Getting rid of all coding interviews is stupid though. Other industries would kill to have such an easy to test skill! We shouldn't throw away such a nice thing just because some people take it too far.

It's only when there's niche knowledge involved that I think it isn't necessary, because you won't be able to answer any hard questions properly without having actually worked on code for that niche at one point or another.

You have to watch someone produce a compiling for loop to prove they’ve ever built software before. Fraud is rampant.

On the other hand, I personally cannot code properly while someone is conceptually "breathing down my neck". A for loop sure, but I'd probably fail at writing a hash table or whatever despite nearing 10 years of experience on many different projects, including compiler frontends and a simple garbage collector I wrote for the fun of it.

Not hiring them is better than a bad hire.

If it were me, I would not use Hackerrank, Leetcode, or other online puzzles sites. Instead it would be just a conversation in to just show me your open source contributions in large repositories or you building projects (open / closed) that others are using.

That is it.

For contributions, they must be under public code review or you doing the review and for open source projects they shouldn't be hello world / test projects but useful to others which are used as dependencies in other larger open-source projects.

A basic easy conversation that rids of 95% of cheaters and frauds and those who cannot code.

Also, how do you even know if it's the candidate that has developed the code? You simply don't know if the candidate copy-pasted the code.

Secondly, just because someone cheated at an extremely difficult Leetcode problem doesn't in any way mean that they can't code on the job. Jobs hardly require such problems.

Instead, ask the candidate to explain specific parts of your code that you show them. It can also be cheated, but it has both a screen and a voice input component, so cheating would take more work.

Declare a variable with any number type initialized to the value 0. I’ve seen candidates fail that, spend 30 minutes attempting to declare a variable in the LANGUAGE OF THEIR CHOICE.

In my entire career, I only once had a situation where they gave me a trivial technical problem, I wrote the first line, and they were like "okay, you passed the test, no need to finish this, we just use it to filter out the people who can't even start". And then we talked.

But usually it's either just talking, or some puzzle that requires you to remember some extremely specialized optimization technique which you will never in your life use again if you pass the test.

My team was hiring recently. Guy comes on, says he's never used Rust, but has some thoughts on it. We talked about all the pros and cons of it, what the system requirements would be, and so on. He talks us through everything to do with the domain, just two seniors talking to each other at a high level. Very pleasant.

We hire him, and a month later he's making idiomatic Rust.

No need to put people through a ridiculous gauntlet.

This industry is unfortunately full of very charming, polished, smooth talkers who would ace that interview and then 3 months later it would dawn on you that you made a big mistake. There is this growing breed of "LinkedIn thought leader" types who, during character creation, put all their skill points into Charisma. And they're really good at sounding technical, using the right buzzwords, overwhelming the interviewer with word-salad, subtly steering the conversation towards areas where they have crisp, prepared answers, and so on. They look great, speak eloquently, and have those ivy-leaguer mannerisms that allow them to breeze through these kinds of informal chit chat interviews. Doubly so if that kind of interview is conducted by an exec or someone less-technical. Beware!

It only works if you are technical. Yes, your sales guy will not know a diamond from a fugazi.

But as a senior tech guy, I don't think it's ever gone wrong. I've never hired a guy who I thought could code and it turns out he couldn't.

You have to be willing to have the conversation with only a few scripted questions, just a few essentials. Everything else an open field. You let them sculpt it into a garden or dig their own grave, either way you give them a shovel.

It gets even worse than that, with hiring based on arbitrary values. One of the companies I worked for before had an engineering manager who chose one candidate over another simply because he had a repo with thousands of stars.