
As artificial intelligence tools like ChatGPT become an increasingly popular avenue for people seeking personal therapy and emotional support, the dangers that this can present – especially for young people – have made plenty of headlines. What hasn’t received as much attention is employers using generative AI to assess workers’ psychological well-being and provide emotional support in the workplace.
Since the pandemic-induced global shift to remote work, industries ranging from health care to human resources and customer service have seen a spike in employers using AI-powered systems designed to analyze the emotional state of employees, identify emotionally distressed individuals, and provide them with emotional support.
This new frontier is a large step beyond using general chat tools or individual therapy apps for psychological support. As researchers studying how AI affects emotions and relationships in the workplace, we are concerned with critical questions that this shift raises: What happens when your employer has access to your emotional data? Can AI really provide the kind of emotional support workers need? What happens if the AI malfunctions? And if something goes wrong, who’s responsible?
The workplace difference
Many companies have started by offering automated counseling programs that have many parallels with personal therapy apps, a practice that has shown some benefits. In preliminary studies, researchers found that in a doctor-patient-style virtual conversation setting, AI-generated responses actually make people feel more heard than human ones. A study comparing AI chatbots with human psychotherapists found the bots were “at least as empathic as therapist responses, and sometimes more so.”
This might seem surprising at first glance, but AI offers unwavering attention and consistently supportive responses. It doesn’t interrupt, doesn’t judge and doesn’t get frustrated when you repeat the same concerns. For some employees, especially those dealing with stigmatized issues like mental health or workplace conflicts, this consistency feels safer than human interaction.
But for others, it raises new concerns. A 2023 study found that workers were reluctant to participate in company-initiated mental health programs due to worries about confidentiality and stigma. Many feared that their disclosures could negatively affect their careers.
Other workplace AI systems go much deeper, analyzing employee communication as it happens – think emails, Slack conversations and Zoom calls. This analysis creates detailed records of employee emotional states, stress patterns and psychological vulnerabilities. All this data resides within corporate systems where privacy protections are typically unclear and often favor the interests of the employer.
Workplace Options, a global employee assistance provider, has partnered with Wellbeing.ai to deploy a platform that uses facial analytics to track emotional states across 62 emotion categories. It generates well-being scores that organizations can use to detect stress or morale issues. This approach effectively embeds AI into emotionally sensitive aspects of work, leaving an uncomfortably thin boundary between support and surveillance.
In this scenario, the same AI that helps employees feel heard and supported also generates unprecedented insight into workforce emotional dynamics. Organizations can now track which departments show signs of burnout, identify employees at risk of quitting and monitor emotional responses to organizational changes.
But this type of tool also transforms emotional data into management intelligence, presenting many companies with a genuine dilemma. While progressive organizations are establishing strict data governance – limiting access to anonymized patterns rather than individual conversations – others struggle with the temptation to use emotional insights for performance evaluation and personnel decisions.
Continuous surveillance carried out by some of these systems may help ensure that companies do not neglect a group or individual in distress, but it can also lead people to monitor their own actions to avoid calling attention to themselves. Research on workplace AI monitoring has shown how employees experience increased stress and modify their behavior when they know that management can review their interactions. The monitoring undermines the feeling of safety necessary for people to comfortably seek help. Another study found that these systems increased distress for employees due to the loss of privacy and concerns that consequences would arise if the system identified them as being stressed or burned out.
When artificial empathy meets real consequences
These findings are important because the stakes are arguably even higher in workplace settings than personal ones. AI systems lack the nuanced judgment necessary to distinguish between accepting someone as a person versus endorsing harmful behaviors. In organizational contexts, this means an AI might inadvertently validate unethical workplace practices or fail to recognize when human intervention is critical.
And that’s not the only way AI systems can get things wrong. A study found that emotion-tracking AI tools had a disproportionate impact on employees of color, trans and gender nonbinary people, and people living with mental illness. Interviewees expressed deep concern about how these tools might misread an employee’s mood, tone or verbal queues due to ethnic, gender and other kinds of bias that AI systems carry.
There’s also an authenticity problem. Research shows that when people know they’re talking to an AI system, they rate identical empathetic responses as less authentic than when they attribute them to humans. Yet some employees prefer AI precisely because they know it’s not human. The feeling that these tools protect your anonymity and freedom from social consequences is appealing for some – even if it may only be a feeling.
The technology also raises questions about what happens to human managers. If employees consistently prefer AI for emotional support, what does that reveal about organizational leadership? Some companies are using AI insights to train managers in emotional intelligence, turning the technology into a mirror that reflects where human skills fall short.
The path forward
The conversation about workplace AI emotional support isn’t just about technology – it’s about what kinds of companies people want to work for. As these systems become more prevalent, we believe it’s important to grapple with fundamental questions: Should employers prioritize authentic human connection over consistent availability? How can individual privacy be balanced with organizational insights? Can organizations harness AI’s empathetic capabilities while preserving the trust necessary for meaningful workplace relationships?
The most thoughtful implementations recognize that AI shouldn’t replace human empathy, but rather create conditions where it can flourish. When AI handles routine emotional labor – the 3 a.m. anxiety attacks, pre-meeting stress checks, processing difficult feedback – managers gain bandwidth for deeper, more authentic connections with their teams.
But this requires careful implementation. Companies that establish clear ethical boundaries, strong privacy protections and explicit policies about how emotional data gets used are more likely to avoid the pitfalls of these systems – as will those that recognize when human judgment and authentic presence remain irreplaceable.
This article is republished from The Conversation, a nonprofit, independent news organization bringing you facts and trustworthy analysis to help you make sense of our complex world. It was written by: Nelson Phillips, University of California, Santa Barbara and Fares Ahmad, University of California, Santa Barbara
Read more:
The authors do not work for, consult, own shares in or receive funding from any company or organization that would benefit from this article, and have disclosed no relevant affiliations beyond their academic appointment.
LATEST POSTS
- 1
全国的な「クマ禍」の影響はスポーツ界にも。サッカー、駅伝、野球、ゴルフ、様々な競技にその余波が #エキスパートトピ(下薗昌記) - エキスパート - Yahoo!ニュース - 2
タワマンなしでは成り立たない再開発…「下町」にタワマンが急増する意外な理由(ビジネス+IT) - 3
韓国野球が日本に追いつく日は本当に来るのか?ドジャースのキム・ヘソンが語った“日韓格差”のリアル #エキスパートトピ(金明昱) - エキスパート - Yahoo!ニュース - 4
Figure out How to Recognize Early Indications of Depressions - 5
鷹・浜口遥大が現役引退を表明「やりきった」 DeNAから加入も1年で戦力外「区切りとして決断」(Full-Count)
「きのこブーム」到来か!?“顔サイズ”の巨大ステーキや“きのこジェラート”も人気【THE TIME,】 (TBS NEWS DIG Powered by JNN)
Instructions to Floss Appropriately and Forestall Gum Sickness
『ポケパーク カントー』来年2月5日開業で注意事項 ポケモン初の屋外常設施設で110段の階段を上れない人など入場できず(オリコン)
Change Your Home into an Exercise center with These Famous Wellness Gadgets
Number_i、新曲の “差別表現” が物議…「ポカホンタス」安易な使用に専門家は「注意が必要」と指摘、TOBEの回答は?(SmartFLASH)
6 Top Computer game Control center
5 Great Youngster Care Administrations To Watch in 2024
日本テレビ・菅谷大介アナが死去、亡くなる前日まで仕事「同じ経験する人の参考に…」語っていた意欲(スポーツ報知)
Well known Travel Booking Locales: What's Your Pick?













