Over the past year or two, companies have started using so-called artificial intelligence agents as bona fide “employees,” even including them in their organizational charts.
Emma Wiles, a Boston University professor who studies how A.I. affects workers, stumbled onto this phenomenon in October, at a conference where two human resources executives said that treating A.I. agents like real employees was a way to increase productivity and to put their companies on the cutting edge.
But when Dr. Wiles and three collaborators from Boston Consulting Group investigated further, they discovered a pitfall. In an experiment involving dozens of companies with A.I. employees, the researchers found that managers tended to vet documents less carefully when told an A.I. employee had produced them. The managers missed errors that other managers caught when told they were vetting the work of a human.
Dr. Wiles speculated that managers didn’t think sussing out mistakes made by A.I. employees was their responsibility. If something went wrong, they could dismiss it as the fault of the tech team, or of the executives who wanted A.I. employees in the first place. “But it’s not your problem,” she said, channeling the managers’ mind-set about their own roles.
In the years since A.I. burst onto the scene, many companies have become aware of flaws produced by the technology and, at times, taken steps to offset them. They know that A.I. models can be biased against certain groups of people, like nonwhites. They know that chatbots can provide confident but incorrect answers to queries. They know that the bots sometimes spill the beans on information that should remain private.
But as companies race to bring A.I. into their day-to-day operations, researchers are discovering more subtle defects. In principle, these flaws could be corrected, too. For example, companies could hold managers directly responsible for the mistakes of A.I. subordinates.
But in practice, most corporate users appear to be blissfully unaware of these issues, raising the possibility that A.I.’s promise of increased productivity and vast cost savings could be undermined.
Even researchers who study A.I. may be aware of only a fraction of the problems that the technology introduces. “There are a whole host of unknown unknowns,” Dr. Wiles said.
One well-documented but underappreciated flaw of artificial intelligence models is that they tend to favor work produced by artificial intelligence. A 2025 paper in The Proceedings of the National Academy of Sciences found that several large language models had a low opinion of text written by humans, creating a “potentially consequential form of implicit ‘anti-human’ bias.”
But many companies seemed unaware of this problem, or at least unable to imagine how it might wreak havoc on their operations. When a team of scholars spelled it out in a subsequent paper, finding that the A.I. models that companies use to evaluate résumés tend to favor those written with the help of A.I. over those written entirely by humans, it caught the attention of some corporate recruiters.
Jane Yi Jiang, an operations professor at Ohio State University who is an author of that subsequent paper, said that she and her co-authors were happy to help when recruiting firms inquired about “how to improve their processes.”
But they noted that this was almost certainly not the only problem companies were inadvertently introducing in their rush to adopt A.I. “People are moving so fast to use L.L.M.s without thinking too much about the implications, biases,” she said.
For example, some companies now use A.I. to help answer questions like how much to charge for a product, or where to open a new location. Relying on the technology for such purposes, however, can quickly go off the rails.
When left to their own devices, humans often cooperate and seek win-win outcomes. But when A.I. models assess a situation, they tend to adopt the more coldly calculating, “rational” mind-set that arises from basic game theory. They might, say, lead a company to aggressively undercut a competitor, even though it risks a damaging price war.
“Most of the L.L.M.s we test think that human beings are more rational than they actually are,” said Jiannan Xu, a Ph.D. candidate at the University of Maryland and collaborator of Dr. Jiang’s. “But the most rational response leads to a bad situation for all” in many cases.
In principle, developers and users of A.I. can correct for these biases. Dr. Jiang and Mr. Xu, for instance, found they could reduce anti-human bias by simply instructing models to focus on the quality of the written material they evaluate, and to avoid considering the author.
But A.I. researchers can’t correct for biases they aren’t aware of, and several scholars said the impact of these undetected biases could grow. One way is if future models are trained on data produced by today’s models without sufficient care, creating a kind of self-reinforcing loop.
In that case, “the tendency to consolidate on existing perspectives and behaviors seems likely,” said Shayne Longpre, an A.I. researcher and founder of the Data Provenance Initiative, a group that monitors A.I. infrastructure.
And then there are the blind spots that arise not so much from A.I. itself, but from the way humans use it.
Scholars who turn to A.I. at every stage of the research process — asking A.I. what questions are worth studying; seeking its advice on how to answer these questions; enlisting it to analyze data; relying on it to help write up findings — could inadvertently narrow the scope of their work.
“We don’t necessarily notice it at the individual level,” said Cecilie Steenbuch Traberg, a psychologist at the Copenhagen Business School and an author of a recent paper on the topic. “You’re sparring with a chatbot, it’s helping me come up with ideas, you might think it sounds great. But at the collective level, it looks pretty similar. Everyone is sounding alike.”
Dr. Wiles, the Boston University professor who examined the way humans manage A.I. employees, said the shortcomings weren’t necessarily intrinsic to the technology, but arose when humans adopted it with little attention to what could go wrong.
She and her colleagues surveyed more than 1,000 corporate managers, and found that about one-third said their organizations referred to A.I. as a “teammate or employee,” and that nearly one-quarter said their employer included A.I. agents on its organizational charts. “We call it Scout,” one manager told the researchers in an interview, referring to an A.I. agent. “It’s technically an equivalent peer on your team.”
Dr. Wiles and her colleagues gave all the managers they surveyed a set of five documents that contained errors, and gave them 20 minutes to review as many as possible. In some cases they told the managers that an A.I. employee had done the work; in some cases they said that an A.I. tool had done the work; and in some cases they said that a human had done the work.
In general, the stated source of the documents didn’t make much of a difference in how closely managers vetted them.
But managers at companies that included A.I. agents on their organizational charts caught substantially fewer mistakes when told they were reviewing the work of an A.I. employee.
People who manage humans tend to assume that “if someone on my team makes a mistake, that’s on me,” Dr. Wiles explained, which is why they closely check the work of these subordinates. Managers also seem to assume that they’re on the hook for work produced by an inanimate A.I. tool. But managers at companies with A.I. employees don’t seem to feel the same responsibility for the work of those employees.
Her takeaway: Over the past few centuries, scholars and business leaders have developed a reliable set of practices for managing humans. But the psychology of managing anthropomorphized A.I. is vastly different, and “we’re going out there blind.”
She worries that the problem is about to get worse. At the same conference where she first heard H.R. officials talk up the virtues of their A.I. employees, one went even further, saying her company would soon have A.I. employees managing humans. “A hush went over the room,” Dr. Wiles recalled.
“We’ll need someone to study that, too,” she added.
