Host Alex Alonso engages with Hilke Schellmann—Emmy-winning reporter, NYU professor, and author of “The Algorithm”—in a critical discussion on the ethical implications of integrating AI into HR practices. The conversation underscores the urgent need for transparency, accountability, and regulation in AI-driven recruitment and performance management. Schellmann highlights how AI tools, while promising efficiency and cost savings, can inadvertently perpetuate bias and discrimination if not properly monitored and validated. The dialogue reveals a disconnect between job seekers' awareness of AI's role in recruitment and the employers' understanding of these tools' inner workings. The episode calls for a collective effort to establish ethical standards, emphasizing the importance of explainability and transparency to protect marginalized groups and ensure fair hiring practices.
Host Alex Alonso engages with Hilke Schellmann—Emmy-winning reporter, NYU professor, and author of “The Algorithm”—in a critical discussion on the ethical implications of integrating AI into HR practices. The conversation underscores the urgent need for transparency, accountability, and regulation in AI-driven recruitment and performance management. Schellmann highlights how AI tools, while promising efficiency and cost savings, can inadvertently perpetuate bias and discrimination if not properly monitored and validated. The dialogue reveals a disconnect between job seekers' awareness of AI's role in recruitment and the employers' understanding of these tools' inner workings. The episode calls for a collective effort to establish ethical standards, emphasizing the importance of explainability and transparency to protect marginalized groups and ensure fair hiring practices.
Rate/review AI+HI on Apple Podcasts and Spotify.
Alex Alonso:
Welcome to the AI+HI Project. I'm Alex Alonso, Chief Data & Insights Officer at SHRM. Thanks for joining us today.
Each week, we sit with visionaries from the world of business, tech, and HR to provide you with deep insights, actionable strategies, and practical advice for harnessing the transformative potential of artificial intelligence, or AI as we all know it, combined with human intelligence, and in my case, sometimes human ingenuity as well.
This week, we're exploring the need for ethics, transparency, and accountability when using AI for recruitment as well as for performance management. This is particularly relevant for us as we've seen great investments across the board in a variety of different startups, different technologies focused primarily on these two areas, especially in the space of HR. Our guest this week is Hilke Schellmann, author of the book The Algorithm: How AI Decides Who Gets Hired, Monitored, Promoted, and Fired, and Why We Need to Fight Back Now.
I can't wait to hear what you have to say, Hilke. It's something that is particularly telling for me and something that's really close to my heart as an industrial organizational psychologist, so I'm looking forward to this conversation.
Hilke Schellmann:
Yeah, thank you for having me. I'm excited.
Alex Alonso:
For those of you that don't know, Hilke is also an Assistant Professor of Journalism at NYU, as well as an Emmy Award-winning investigative journalist. But one of the things that stands out is I'd love to hear a little bit more about the focus of your research, the kind of reporting that you do. I mentioned in some of our pre-work that I am a big fan of your work, having watched Outlawed and a variety of other things that you've done. What compelled you to write the book, The Algorithm?
Hilke Schellmann:
I think I asked myself that many times while I was writing it. But I think it started in November 2017, so it's been a while, where I was at a conference in Washington, D.C. that had nothing to do with AI or HR. It was a consumer lawyer conference and I needed a ride back to the train station to take a train to New York, and as one does, I called myself a Lyft, a rideshare, got in the backseat, started chatting with the driver and just asked him, "How was your day?" And he was like, "You know, I had a weird day." And in the history of me taking rideshares, that has never happened. I take them somewhat a lot because I live in New York and I don't have a car, and I was like, "So what happened?"
And he told me that he had a job interview with a robot. He had applied to a baggage-handler position at a local airport, and he got a call from a robot that day on his phone asking him three questions. I had never heard about robots doing job interviews, and I was really interested, and I made a note, and kind of forgot about it until a few months later, my colleague, Meredith Broussard, dragged me to an AI conference. I wasn't covering AI at the time. I really didn't really have much of an interest, but there was someone who had just left the Equal Employment Opportunity Commission giving a talk, and she said she can't sleep at night because companies use a very basic algorithm to go through people's calendars to understand how many hours they're absent. And she was really worried that people who have a disability, mothers, or people, employees with caregiving obligations might get reprimanded or terminated because they have longer absentee times and they are protected under the law.
So we started talking. She told me about some other companies and how AI is moving into hiring. I went to CIOB, I went to HR Tech, and I was just blown away by what I was seeing, and I was like, "Wow, this seems to be really moving into this industry." And I was looking around, I was like, "I haven't really seen a lot of journalism about it. This seems to be a really seismic shift in the industry."
Alex Alonso:
A lot of people think about it in terms of something that's been taking off since 2023 because everybody thinks of... What we think of is really ChatGPT, right?
Hilke Schellmann:
Yeah, totally. Yeah.
Alex Alonso:
But in reality, it's been in our field, and really machine scoring, all the different types of machine learning that we know about, they've been around for almost 20, 30 years in some cases, some even going back to 1952.
I'd love to get your reaction to, you're right, that is something that warranted a book in many ways, and I'd love to get your reaction to this notion of what it is that people don't know. Particular that stands out to me as far as what is it that's happening, what is it that's out there in terms of the space, and not just from the candidate perspective and the applicant perspective, but also even from the employer perspective, to your point.
Hilke Schellmann:
Yeah, yeah. I mean, I think maybe not very surprising to a lot of people, I talked to a lot of job seekers, and they are really unaware how much AI is being used. I mean, some folks do know that AI is being used and now that they have ChatGPT and generative AI, they sort of joke about that they use ChatGPT to help them polish their resume and their cover letter and they use it to train for one-way video interviews. They sort of joke about it, "AI against AI," but most people that I talked to on the job-seeker side had no idea that AI is being used on them. They didn't know that there's resume screeners, or resume parsers, that all the big job platforms use some form AI. When I talked to folks who had applied, they thought a human watches it.
So we see there's a real disconnect. The job seekers have no idea, and I'm sure it's somewhere in the fine print, but job seekers, they just want the job so desperately, they're going to click on it and play the game or do the video interview and they don't know how it's going to be analyzed and scored. They just hope they'll make it into the next round.
But I think also what employers may not be aware of, which I found quite striking, that when we use deep neural networks, which we often use in predictive AI, and it doesn't really matter what that is, but what is helpful to think about is we have the training data and we have the results, but we don't necessarily know exactly what the computer scores are that's happening in the deep neural networks. So I think what was interesting to me is that the vendors often actually don't know exactly what the system score candidates on.
And I think employers, they buy these tools and they hear wonderful marketing messages, that this is going to democratize hiring, and Spy is free, and it's going to find the most qualified candidates, we're going to make hiring more efficient, they're going to save a lot of money. I think it makes it way more efficient, it saves them a lot of money, but I found so many instances of possible discrimination and instances of I don't really have any evidence that the tools pick the most qualified candidates. And in fact, from surveys of C-suite leadership, we know that if a company uses AI, about 90% in a survey of over 2,000 C-suite leadership folks, they said they know that their tools reject qualified candidates.
So I think we kind of know that these AI tools are not working as well as we thought they would be or that was the hope. And I do think that, this is from the research I've done, I think that humans are also very biased. I'm not advocating to go back to the old days of human hiring, but I feel like this is the first generation of AI tools that we have seen and we just seem like we have automated a lot of processes that were not quite working in the old days either. So now we automated them and found it's really not working because the problems of the past, we just replicate it now in a much larger scope, and I think that's where the problem lies.
If there's a problem with that tool, it could discriminate against tens of thousands or hundreds of thousands applicants. And I think we know, most employers are getting inundated by resumes. They get so many applications, they don't know how to do it, so they need these kinds of tools, but I think they need to be really aware and understand what goes inside the tools. And I think it's quite striking that the developers often don't know themselves what goes inside of these tools, which I think is different from what IO psychologists did before, like with the regression analysis, you kind of knew the inputs and you kind of knew what was happening, and that's now a little different
Alex Alonso:
To your point, that's fascinating that you mentioned that because in the past, the algorithm was known and it was to refine it with as few variables as possible so that we got to the point where we understood where the variance or the variability in hiring really existed, right?
I think a little bit about because there's an example that comes to mind from your book, and I'm going to dive into recruiting and hiring and retention and all the various processes, but the example that comes to mind is from chapter four, and I think about Alex, the young credit analyst, who really spoke to one of the things that he was finding was he just wasn't getting job offers. He wasn't even being invited to come back.
One of the things that stands out to me though was he was actually... He classified himself as a people person, which is atypical in many cases if you were to think about the personality traits and/or the various things that go into what is your typical financial analyst or credit analyst. So in many ways, what I found particularly interesting about that example was things that would be differentiators were actually differentiators, but in a way that kept him from being included in searches as opposed to including him in searches.
Hilke Schellmann:
Yeah. I mean, we don't know exactly, right? And I would never say that I know exactly why Alex didn't make it to the next round, right?
Alex Alonso:
Oh, sure.
Hilke Schellmann:
Because I'm not inside the algorithms, but what strikes me as difficult is that a lot of the AI tools are not based on a job analysis first to understand what are the skills, capabilities, and maybe experience that you need to be successful in this job? It takes whatever information we can gather about the people currently in the role. In this case, with Alex, he was doing HireVue video interviews, and at the time, this was in 2018, I was working on a story for the Wall Street Journal, HireVue was still using, in their AI tool, emotion recognition software and intonational voice analysis.
And at first, I was like, "Oh, it seems really interesting. Maybe the machines are better than we are and they found something about facial expressions that we humans had no idea is inferring, is predictive of success in the job." Well, it turns out there's really no science underneath this, right? There is no science with facial expressions I should have in a job interview that will predict if I'm going to be successful. That's just a correlation. That's not causation.
And I think the problem is, as you can imagine, that if you have one of a kind of people in the job, that maybe it will start looking for their facial expressions. So first of all, it's not valid, so it's not find anything that is actually predictive of success, and secondly, you might actually discriminate. So I think that strikes me as problematic.
Is it a good idea to take all kinds of data in, or is it actually... I actually would say that maybe traditional, old-school ways where you actually are in charge of the inputs when you do an algorithmic test on an assessment might be better because you actually know what you're testing for and not just take any data.
We see this in resume screeners too. You take all the data in. The tool does what it does best, it does a statistical analysis, and in some cases, it found out that first names like Thomas were predictive of success. Obviously it has nothing to do with the job. And then it went into places like Syria and Canada were suddenly predictors of success, so that could be discrimination based on national origin. The word African American, those are things that everyone in hiring knows this is like, "You are on the path of discrimination and you definitely don't want to go there," but I think if we don't supervise these systems and monitor them, this can easily creep in.
So I encourage everyone to do pilot studies, to really do their due diligence on these tools, but you also have to monitor them after you start using them because the algorithm changes and the input data changes every time with new job seekers, new people who were successful in the job, you put their resumes in. So you have to constantly monitor the system, and I think that takes a lot of money, it takes a lot of time, and I think a lot of talent acquisition HR departments buy these tools to save money. So it's like they probably don't want to hire now people that now supervise these systems, but I think actually you have to.
Alex Alonso:
You raised an excellent point in that regard.
Pretend that I'm a CHRO or that I'm talking to a bunch of HR professionals. What kind of concrete steps do you think the employers can take? Especially in the space of recruitment and hiring and employee selection, what are the kind of things that you think they might do to tweak their hiring processes to allow for that natural evaluation, that consistent feedback loop, that responsible use, right?
Hilke Schellmann:
Everyone needs to put their critical thinking hat on, first of all, and really understand what's my problem that I want to solve and what could be a solution? And then really critically assess these tools and really think through like, "Well, is there science to actually back this up, this idea that this company is putting forward?" And then do a pilot study, and then just continuously... There is a way. If you use deep neural networks, we can go back to the computer and the software and ask the software, "What did you actually predict on?" So you can get these lists. I mean, obviously thousands of keywords or something like that in case of a resume parser. It's not interesting to look at these lists, but it has to be done to just make sure it works.
Also, if you buy a tool from a vendor, I think if that company doesn't have a technical report, it's a first red flag, and then really scrutinize these technical reports. I've only seen a few, and that does bug me a little bit because I think that should be actually public. If you built this tool and you validated it and you made sure that it passes the four-fifths rule and it passed people at somewhat equal rates, why wouldn't you make that public? I'm not understanding why that couldn't be public.
And so a couple times that I was able to see it, I was like, "Wait a second. Some of these tools are validated on college students in their 20s. What happened to all the other people? How on Earth can this be generalizable?" Obviously, the vendor doesn't want it to be known if there's a problem, but I also feel like employers are shy who use the tool. And I've talked to so many now who've used some of these tools for years and then they're like, "Oh, yeah, we have the same concerns that you outlined in the book and we had the same problem. So we quietly abandoned it," and that's good for that company, but the whole community is none the wiser, and the next company, the next HR department will then buy the software again and again.
I do think that AI is a great promise. I have built AI tools for journalists, I use AI all the time, but I really think about the use case and I really think through, "Is this an ethical use? Does this actually work? Does it discriminate?"
Alex Alonso:
We've done such a good job of educating the average consumer on the importance of reliability and validity, but we haven't gone that extra step. And then this is just my one man's opinion, but we haven't gone that extra step to say, "Valid and reliable for what?"
Hilke Schellmann:
Yeah, and you don't want it just to work for a few, and I really do worry about people with disabilities. In one of the games that I play to find my personality or my capabilities, I had to hit the space bar as fast as possible to build up a bar. While you're doing that, I was wondering first of all, about face validity. I was like, "What does it have to do with any kind of job? Why am I hitting the space bar as fast as possible on my keyboard?" But then when I played it with somebody who had a motor disability, he was really concerned, like, "What is with people who can't use their hands as fast as others? Will they be rejected?"
That's obviously a very basic example, but it's also a question about these tools are quantitative tools that look for the statistical medium, like most people together, what the middle has in common that is statistically relevant. So the problem is people with disabilities are already underrepresented in the workforce, so they're probably not part of the training data or are underrepresented in the training data that we use to train these rules.
And then on top of that, their disabilities are presenting themselves so individually. So maybe you have somebody who's autistic who plays the game and hopefully, somehow their data will be somehow statistically relevant, so that's already a high bar, but then the next person who is autistic, their disability can present itself so differently. So the statistical analysis doesn't catch people on the margins, and especially people who've already been marginalized because they never make it into the training data.
Then what's also troubling is yes, people can ask for an accommodation, but obviously people with disabilities don't necessarily want that because they're afraid of getting on this imagined pile B that no one will be looking at. The people with disabilities is on this pile B. But I think also the problem is with some of these tools, the job seekers don't even know that they're being assessed by AI. They play a game, they have no idea what comes in the game. They don't even know if they're going to need an accommodation. So I think there's a lot of questions here. How can this work for people who are in marginalized groups?
And I was really sad and disappointed when I was working on an audit and an auditor checked one of the video games companies for the four-fifths rule and found that in the intersection, when they check for African American women, they found that they didn't pass at similar rates. They were below their four-fifths rule, and that didn't make it into the final audit because the company paid for the audit.
So I would be very careful if I was an employer using a tool that has been audited by a third party to really understand was there a conflict of interest?
Alex Alonso:
You know, one of the things that strikes me is even in your book, you talked a little bit about things outside the realm of recruitment, selection, and so on, and you talked a little bit about performance monitoring and evaluation, some of the stock, core bread and butter of what makes good talent optimization real, and that's naturally part of the world of human resources, as you might imagine.
I'd love to hear some examples of the discrimination you found and also what it is that you see employers doing or could be doing in some way.
Hilke Schellmann:
You know, this one woman described to me how she set a timer to go to the bathroom because she only had 15 minutes, and if she needed longer in the bathroom, she also wanted to eat a quick snack, but she needed to be back because as soon as she was over her 15-minute limit, her boss would call her, email her, and be like, "Where are you? You need to be working." And she described it as so stressful and it really took... Her mental health took a nosedive.
We also know this from science. The surveillance of workers is actually not leading to them being more productive. What it leads to is productivity theater that you try to always have the green bar on, the active status, that you check it early on Slack, and we like, "Ready to work" at 7:30, and then you take your dog for a walk, or you go to meetings that you don't even want to or need to go to, but you just want to show your face, and that cannot be helpful in the workplace.
So I think what is a much, much better measurement, probably much harder... So we can technically check if you are in front of your computer, but are you actually performing the work and are successful at doing the work? We don't actually know that. In fact, a lot of people just sit at the computer and do other things or just think around. And honestly, I tested that myself and surveilled myself and monitored myself with a couple of these companies, and a lot of things that I do is like phone calls and other things that were zero productive under their measurement standards, which I do think is essential to my job, but obviously that can be tweaked, but I think it has a really human toll.
And I think there is sort of a weird discrepancy here that we see a lot of CEOs saying, like, "In the age of AI, we need creative humans, critical thinking, find new ways to think through problems," and it's like, but at the same time, you're going to monitor every keystroke and take screenshots of my face that I'm really at my computer? That doesn't feel like you want me to be creative or be a critical thinker and a team worker. And we really need to think through how can we measure success?
And I think we've seen that again and again, like a company that I write about that did key-swipe data to understand who they should promote, so who was the longest in the office should be promoted. It's already pretty flawed because you can be in the office and not be productive, but when then it came time to layoffs during the pandemic, they also wanted to use this data and layoff the people that have the least hours in the office. Well, here you are with the problem again that it could be people with disabilities, caregivers have longer absentee times and they're protected classes.
Alex Alonso:
I have an example for you that I think is arguably one of the weirdest ones I've ever come across in just studying this area a little bit, especially as far as it relates to performance monitoring.
I worked with a company in particular that was trying to understand how ingenuity worked its way into their workforce, and more importantly, how the culture fostered it. And one of the things that they asked people were, "Have you ever purchased a mouse jiggler?" Asking, "Knowing there is this keystroke and this mouse-click technology out there and that we're employing it, have you actually used it?" And people who had and done so were actually not scored negatively, but actually scored positively.
But again, how do you know what the algorithm is? What is the right way to respond to that?
Hilke Schellmann:
Yeah, and I think because sometimes I do wonder, we want the easy ways, right? So we have understood, I think at one point, there was this idea that if you use the particular browser, you are more intelligent than the general population. Well, that's not true for everyone. And also, why didn't you just check for intelligence versus a browser? That seems to be a very bad proxy.
And I mean, I talked to the former head of talent acquisition at Walmart, and I'm always baffled where problems can come in from what different sides. They did a survey of people and they found out... Retention is a big deal, right? The people who stay longer at the stores, the associates, they often do so because they know somebody at the store. It seems totally facially neutral, but it turns out when they looked at the demographics, people who are Asian American have more acquaintances at the store and stay longer. And if they would have used this indicator, they would've discriminated essentially against African Americans who had less connections to people in the store.
Alex Alonso:
Yeah, it's funny, you really are hitting upon the theme of the show, which is it's AI+HI, right?
Hilke Schellmann:
Yes.
Alex Alonso:
It takes both things to really make it effective.
I'd love to hear your thoughts around what are you seeing out there that really are the risks that might be there, the possible gains that might be there as well around employee culture, and employee morale, well-being, satisfaction, those kinds of things?
Hilke Schellmann:
Yeah, yeah. I mean, I think most employees are not against AI. They want to use AI as well, but they want to know how it's used to supervise them in a way or monitoring them. And I think if it's about safety, if you were monitoring in a nuclear power plant, if you are monitoring my exposure with AI to nuclear materials, everyone is going to say yes to that because they want to be kept safe, so I think those are pretty easy things. But I think even employees are... They understand this is a business necessity. I think they're willing to go with it unless it's all-out surveillance, I think no one likes it.
I think if you bring employees together and they have a say in this, I think they can be great co-governors in using these tools and really understanding how can we use them? How did the company assess it? You can actually build trust and transparency in your organization, which I think will greatly benefit the culture versus a top-down. And I always wonder, what happens when employees find out that you're doing this? Are they going to be happy or not? If they're not going to be happy, you might lose them. They might go somewhere else. And I don't know if you want that because hiring seems to be very expensive. It's really hard to find the right people, so if you have good people in place, I would try to keep them.
Alex Alonso:
Yeah. You know, it's fascinating because that's actually what happens in many cases is I think many organizations are evaluating their talent processes on efficiency metrics as opposed to quality metrics in some cases, and so you start to see some of those things. I find that the most refined organizations in the world are actually doing both. They're capturing both. So they're trying to figure out what throughput is in terms of their talent, but also trying to figure out how they optimize that talent.
Hilke Schellmann:
So sometimes I wonder about some of these success metrics in hiring. I speak to recruiters and they get paid how quickly they hire people. I mean, I get you want to speed that up, but I do feel like, wait, wouldn't you want to pay them if it's the right person? Did the person stay for a year and actually get a good performance review or something? Can you pay a recruiter two-thirds of their pay at the beginning and then a third after a year if that person actually turned out to be a good performer? Can we change the incentives here?
I know it'd be harder, but I feel like is that a good incentive for recruiters to just hire, how fast they hire people? That cannot be. That just gets you the people who are most likely to be hired, I will find those and work with them, but am I really thinking through who would be the best qualified person for the job? I don't know. So I think there can be really things that can be changed.
Alex Alonso:
One of the things that stands out is what you've talked about is actually not an anti-AI perspective, but rather an AI perspective that is both monitored and used responsibly with the involvement of human intelligence and human evaluation, right? All of that rests on a foundation of transparency, accountability, and even regulation to some degree.
Hilke Schellmann:
Yeah.
Alex Alonso:
What kind of practices can employers do both to put in place more internal accountability, but then at the same time, also foster greater transparency with their workforces?
Hilke Schellmann:
So I do think we need more regulation, but I do think regulation is tough because we've seen, for example, New York City Law 144, and I think that everyone agrees, from employers to vendors to experts and advocates, that it's a disappointment that it really didn't work. And I think that's not good. We can't have regulation that doesn't work, but regulation is really tough because there's so many use cases, so many different ways.
So I think transparency is better, explainability is better. If we can have employers give that to candidates, that would also really help me as journalists and others to piece together how do these algorithms work? And I'm always down for finding good use cases and finding ways that this works.
And so I hope that the industry can maybe regulate itself a little better and have higher standards, maybe also add to the four-fifths rule and actually look at intersectionality and recommend that. Yeah. And sometimes I hope maybe unions will start putting some of these AI tools into their bargaining contracts so that employers need to be more transparent and there is accountability built in.
Alex Alonso:
So if there was one word that you could use to offer a piece of guidance and advice to your CEO, your CTO, and your CHRO in terms of really integrating AI into talent, what is it that you would offer?
Hilke Schellmann:
I would think about transparency and explainability. And I do this a lot when I talk to vendors. I always ask, "If you were in front of a judge," and I usually take all these tests, or I've done so many AI video interviews and games and all of this, and I was like, "can you tell me how exactly I was scored? How did the 89% come together?" If you can't explain that to me, that's a problem.
Alex Alonso:
You know, it's fascinating. You sort of hit upon something that stands out even in terms of explainability and taking all those games, right? So really with the aim of explainability, but more importantly, explainability and then repeatability, right? I'd love-
Hilke Schellmann:
Oh, God, yeah. Oh, we saw so many problems for some of these tools. They couldn't repeat the same thing and it's like there's so brittle. I mean, that is so flawed that they shouldn't be even on the market. And I don't know if they can be out-regulated, but I think we just assume, we all always assume that, "Oh, it's on the market, so somebody must have tested it." But no, that's actually not the case. Nobody has to test these tools. They can just exist.
Alex Alonso:
I need to thank you. I have to be honest, this has been such a joy, Hilke. And honestly, I want to thank you for sharing your experiences, your deep insights with us. This has been so great, and I'm truly grateful.
For our listeners, if you enjoyed today's episode, please take a moment to comment, leave a review, et cetera, whatever you think is critical. And then finally, if you can find all our episodes on our website, or want to find them, please go to SHRM.org/AIHI.
Thanks for joining the conversation, Hilke. We've really appreciated your time here today, and thank you all for joining our conversation here today, and we'll catch you next time on the AI+HI Project.