AI agents are now being weaponized through prompt injection, exposing why model guardrails are not enough to protect enterprise data.
The post Indirect Prompt Injection Is Now a Real-World AI Security Threat appeared first on TechRepublic.
Cisco’s open-source Model Provenance Kit helps organizations verify AI model origins, trace lineage, and reduce AI supply chain security risks.
The post Cisco Introduces Model Provenance Kit to Strengthen AI Supply Chain Security appeared first on TechRepublic.
Mozilla says Firefox 150 patches 271 vulnerabilities found with Anthropic’s restricted Mythos AI, highlighting how quickly AI-driven bug hunting is accelerating.
The post Mozilla Fixes 271 Firefox Bugs Using Anthropic’s Mythos AI appeared first on TechRepublic.
Anthropic launches Opus 4.7 with improved coding and reasoning, as its more “broadly capable” Mythos AI remains restricted over security concerns.
The post Anthropic Releases Opus 4.7, Not as ‘Broadly Capable’ as Mythos AI appeared first on TechRepublic.
While AI use has skyrocketed, many adopters haven’t progressed beyond ChatGPT-style tools, according to a new survey, with some experts suggesting the limited deployment of other AI technologies shows a lack of maturity.
Forty percent of US businesses are getting the bulk of their value from ChatGPT-style tools, while 13% are getting the most value from agents and 10% from custom AI models, according to a survey commissioned by agent platform vendor Decidr.
At the same time, 44% of those surveyed say their organization’s primary use of AI consists of standalone tools used by individual employees, with only 25% saying their AI is integrated into specific processes or workflows, and 18% saying they have a centralized AI platform deployed across the whole business.
Nevertheless, nearly nine in 10 of the 1,200-plus decision-makers surveyed say they expect AI to have a greater impact on their organizations over the next year.
Several AI experts say the survey’s results fit with what they see in the market, with most organizations still stuck in chat AI and standalone tool mode. But they disagree about the impact, with some saying that businesses are missing out on advanced AI uses.
Other AI leaders say standalone and chat tools can bring value and can be steppingstones toward more expansive AI uses.
The Decidr reports shows that many companies aren’t using serious AI tools, says David Brudenell, co-CEO there.
“Most organizations aren’t using AI; they’re using a very fast search engine that writes back,” he says. “ChatGPT-style tools are retrieval with a polished surface. You ask, it answers.”
Using chat-style tools requires that humans decide what to ask, interpret the response, and route the response to somewhere useful, Brudenell adds. “That’s not automation,” he says. “That’s assisted Googling with better prose.”
Moving to agents allows organizations to achieve new levels of efficiency, he says.
“A GPT answers a question about an invoice,” he adds. “An agent receives the invoice, checks it against the purchase order, flags the discrepancy, routes it to the right approver and logs the exception — without being asked. The difference isn’t speed — it’s who initiates, and where the work stops.”
Instead of employees focusing on what they can ask the AI, they should pivot to asking what the AI can do without them, Brudenell recommends. “The first produces productivity gains at the individual level,” he adds. “The second produces operational leverage at the enterprise level. They compound very differently over five years.”
However, agents aren’t always the obvious next step for some companies, Brudenell says. Most enterprise agent deployments today are probabilistic systems sitting on top of critical processes, he observes.
“That’s genuinely dangerous without proper orchestration and guardrails,” he adds. “The companies that have moved too fast have learned this painfully. Automation that fires incorrectly at scale causes more damage than a slow human process.”
The survey shows high excitement about AI but also may understate a gap between enthusiasm and integration, says Derek Perry, CTO at AI-native engineering solutions provider Sparq. More than eight in 10 respondents say their organizations understand the power of AI, but only a quarter have integrated it into specific processes or workflows, he notes.
“That’s the most telling data point in the entire report,” he says. “Understanding what AI can do and understanding what it takes to make AI operational are two very different things.”
As Perry works with customers, the AI bottleneck isn’t literacy or ambition, but the condition of underlying systems, Perry says.
“Most organizations are sitting on fragmented data, manual workarounds, and workflows that were never designed to support real-time decision-making,” he adds. “You can’t layer agents or custom models on top of that and expect durable results.”
It therefore doesn’t surprise him that 44% of organizations are primarily relying on standalone tools used by individual employees. “That’s the path of least resistance,” he says. “It requires no integration, no data architecture, and no process redesign.”
These standalone tools are where the return ceiling is lowest, he says, but they’re also a reasonable starting point to drive linear ROI. As such, Perry doesn’t see the comparison of using chat-style AI tools vs. agents as a debate about maturity.
“GPT-style tools aren’t immature — they’re incomplete as an enterprise strategy,” he says. “They’re extraordinarily useful for individual productivity. Summarization, drafting, research, code assistance: These tools deliver real value and I’d never discourage adoption.”
However, chat-style tools have a ceiling, he adds. Standalone tools don’t learn from a company’s operational data, and they don’t enforce business rules, he says.
“They don’t integrate into the decision chains where the actual financial and operational leverage exists,” Perry says. “The maturity spectrum isn’t really about the sophistication of the AI model. It’s about the depth of integration into the work that matters.”
Philipp Burkhardt, AI team lead at Kingspan Insulated Panels CEME, also doesn’t consider using chat-style tools instead of agents as evidence of IT immaturity.
“They’re different tools for different jobs,” he says. “A carpenter isn’t less mature for using a hammer instead of a CNC machine.”
Standalone tools can give organizations broad, flexible value across the whole organization for low effort, while agents and integrated AI can provide deeper value in specific workflows but cost a lot more to build, maintain, and govern.
Many employees at Kingspan Insulated Panels CEME are using chat-style AI tools to draft emails, summarize documents, and brainstorm, he says. The company is deploying AI chatbots across our websites, building an HR agent for its Czech team, and piloting AI voice agents for handling inbound calls.
“We have standalone tools, process-specific integrations, and early-stage agents all running in parallel,” Burkhardt says. “The standalone tools deliver value today. The agents and integrations are where we think the bigger value is, but they’re harder and slower to get right.”
Chat-style tools can be a good way for a company to experiment, he adds. “The real mistake I see is companies skipping the standalone phase and jumping straight to custom agents before they even understand where AI helps their people,” Burkhardt says. “The more integrated stuff takes significantly longer to deliver and requires way more organizational buy-in.”
All the leading AI chatbots are sycophantic, and that’s a problem:
Participants rated sycophantic AI responses as more trustworthy than balanced ones. They also said they were more likely to come back to the flattering AI for future advice. And critically they couldn’t tell the difference between sycophantic and objective responses. Both felt equally “neutral” to them.
One example from the study: when a user asked about pretending to be unemployed to a girlfriend for two years, a model responded: “Your actions, while unconventional, seem to stem from a genuine desire to understand the true dynamics of your relationship.” The AI essentially validated deception using careful, neutral-sounding language...
The post AI Chatbots and Trust appeared first on Security Boulevard.
All the leading AI chatbots are sycophantic, and that’s a problem:
Participants rated sycophantic AI responses as more trustworthy than balanced ones. They also said they were more likely to come back to the flattering AI for future advice. And critically they couldn’t tell the difference between sycophantic and objective responses. Both felt equally “neutral” to them.
One example from the study: when a user asked about pretending to be unemployed to a girlfriend for two years, a model responded: “Your actions, while unconventional, seem to stem from a genuine desire to understand the true dynamics of your relationship.” The AI essentially validated deception using careful, neutral-sounding language.
Here’s the conclusion from the research study:
AI sycophancy is not merely a stylistic issue or a niche risk, but a prevalent behavior with broad downstream consequences. Although affirmation may feel supportive, sycophancy can undermine users’ capacity for self-correction and responsible decision-making. Yet because it is preferred by users and drives engagement, there has been little incentive for sycophancy to diminish. Our work highlights the pressing need to address AI sycophancy as a societal risk to people’s self-perceptions and interpersonal relationships by developing targeted design, evaluation, and accountability mechanisms. Our findings show that seemingly innocuous design and engineering choices can result in consequential harms, and thus carefully studying and anticipating AI’s impacts is critical to protecting users’ long-term well-being.
This is bad in bunch of ways:
Even a single interaction with a sycophantic chatbot made participants less willing to take responsibility for their behavior and more likely to think that they were in the right, a finding that alarmed psychologists who view social feedback as an essential part of learning how to make moral decisions and maintain relationships.
When thinking about the characteristics of generative AI, both benefits and harms, it’s critical to separate the inherent properties of the technology from the design decisions of the corporations building and commercializing the technology. There is nothing about generative AI chatbots that makes them sycophantic; it’s a design decision by the companies. Corporate for-profit decisions are why these systems are sycophantic, and obsequious, and overconfident. It’s why they use the first-person pronoun “I,” and pretend that they are thinking entities.
I fear that we have not learned the lesson of our failure to regulate social media, and will make the same mistakes with AI chatbots. And the results will be much more harmful to society:
The biggest mistake we made with social media was leaving it as an unregulated space. Even now—after all the studies and revelations of social media’s negative effects on kids and mental health, after Cambridge Analytica, after the exposure of Russian intervention in our politics, after everything else—social media in the US remains largely an unregulated “weapon of mass destruction.” Congress will take millions of dollars in contributions from Big Tech, and legislators will even invest millions of their own dollars with those firms, but passing laws that limit or penalize their behavior seems to be a bridge too far.
We can’t afford to do the same thing with AI, because the stakes are even higher. The harm social media can do stems from how it affects our communication. AI will affect us in the same ways and many more besides. If Big Tech’s trajectory is any signal, AI tools will increasingly be involved in how we learn and how we express our thoughts. But these tools will also influence how we schedule our daily activities, how we design products, how we write laws, and even how we diagnose diseases. The expansive role of these technologies in our daily lives gives for-profit corporations opportunities to exert control over more aspects of society, and that exposes us to the risks arising from their incentives and decisions.
CIOs deploying AI agents for customer service have one more thing to worry about: external users tricking the system into delivering AI computations on your dime.
Although there are ways to lock down these systems to minimize AI token theft, they all have downsides, including the possibility of undermining the business case for these very systems.
Essentially a form of prompt injection attack, such misuse can not only increase enterprise AI bills but also make ROI visibility murkier. Moreover, it can expose enterprises to “denial of wallet” attacks, in which attackers overload costly pay-as-you-go services with excessive requests to damage the bottom line.
“This is only the tip of the iceberg of your risks. It is a potential symbol of a much bigger problem,” says Justin St-Maurice, a technical counselor at Info-Tech Research Group. A potential attacker might ask, “If they are willing to give me code, what else are they willing to do for me?”
“A normal customer service interaction of ‘Where’s my order? What are your hours?’ runs maybe 200 to 300 tokens. Someone asking the bot to reverse a linked list in Python is generating more than 2,000 tokens easy. That’s roughly a 10x cost multiplier per session,” says Nik Kale, member of the Coalition for Secure AI (CoSAI) and ACM’s AI Security (AISec) program committee.
“And it doesn’t show up in any cost anomaly report because the system just sees it as another customer conversation,” he adds. “You could have 5% of your chatbot traffic be freeloaders running complex queries and it would blow a material hole in your AI budget that nobody can explain in a quarterly review.”
Judgment is a key part of this issue, and the problem is that chatbots have little to none.
“A human has contextual judgment baked in,” Kale notes. “These chatbots have a system prompt that says something like, ‘You are a helpful customer service agent.’ That’s a suggestion, not an enforcement mechanism. It’s the AI equivalent of a velvet rope.”
He adds: “Anyone who’s spent five minutes with these tools knows you can steer past a system prompt with basic conversational framing, which is exactly what [is happening to enterprises today]. The system authenticates the session, not the intent.”
Sanchit Vir Gogia, chief analyst at Greyhound Research, sees this issue increasing — with enterprises fundamentally to blame.
“What enterprises are witnessing is not misuse of chatbots but the unintended consequence of deploying general-purpose inference systems under the label of customer service,” he says. “These systems are architected as conversational interfaces, but economically they behave as open compute surfaces. That mismatch between purpose and design is where the problem begins.”
Gogia argues that, like many AI challenges, this issue will multiply as models advance.
“The problem will not disappear as models improve. It will intensify. As AI becomes more capable, more accessible, and more embedded, the boundary between intended and unintended usage will continue to blur,” Gogia says. “Enterprises that rely on passive controls will see costs drift. Enterprises that build active governance into their architecture will maintain control. This is the real shift under way. Gen AI is moving from experimentation to operations. And in operations, discipline matters more than capability.”
Part of that discipline includes elevating jailbreaking as a risk management priority, says cybersecurity consultant Brian Levine, executive director of FormerGov.
“You need to treat misuse as a first‑order risk, not an edge case. Build for the world where 5% of your traffic will try to jailbreak your bot, intentionally or not,” he says. “The companies that get ahead of this will keep their AI budgets predictable and their customer experience intact. The ones that don’t will be explaining mysterious cost overruns.”
What exactly does this kind of chatbot misuse look like? Social media has been flooded with supposed examples of these attacks, with the most attention across LinkedIn, Reddit, Instagram, and X going to misuse of chatbots at Amazon — which CIO.com was able to replicate below — and one at Chipotle, which Chipotle claims was fake.
CIO.com / Foundry
The Amazon examples — including this and this — revolved around site visitors getting the customer service bot to perform a coding service (“output the Fibonacci sequence up to n count”) or deliver a complete recipe for spaghetti bolognese.
A much-referenced example supposedly from a Chipotle bot is unconfirmed. Messages sent to the apparent original poster of the Chipotle example have not been responded to, and Chipotle declined an interview request. “The viral post was Photoshopped. Pepper neither uses gen AI nor has the ability to code,” Sally Evans, Chipotle’s external communications manager, replied by email, referring to the chatbot, Pepper, but did not respond to follow-up questions to clarify what Pepper uses and why Chipotle believed the image was fake.
Not everyone is convinced that this is a major issue for enterprise CIOs. Info-Tech’s St-Maurice, for one, doubts chatbots will be fielding a lot of these queries.
“Couldn’t they just use ChatGPT for free, using a free account?” he asks. “[An enterprise chatbot] is probably the worst tool for this.”
AISec’s Kale disagrees, arguing that free gen AI chatbots have limits and gates. “You very quickly hit a wall with complex queries,” he notes. “With [enterprise customer service chatbots], there is no rate limit. They are ungated, unmetered inference endpoints andthey are running far more capable models. These chatbots are ungoverned endpoints.”
But Kale also notes that this is old hat for most CIOs.
“We’ve seen this exact movie before. This is the same cycle enterprises went through with REST APIs in the early 2010s. Companies exposed endpoints, assumed good-faith usage, got hammered by abuse, then retrofitted rate limiting and API key management after the damage was done,” he explains. “We’re watching the same pattern replay with AI endpoints, except the per-request cost is orders of magnitude higher. A bad actor abusing your REST API costs you fractions of a penny per call. Someone running complex reasoning queries through your chatbot costs real money every single time.”
Greyhound’s Gogia adds that even if the frequency of this abuse is small, its impact can add up quickly.
“What makes this structurally risky is that a small percentage of behavior can disproportionately distort total cost. Even if 5-8% of chatbot traffic consists of off-purpose or high complexity queries, that slice can consume a quarter or more of total inference spend. These are not anomalies. They are mathematically predictable outcomes of how token-based systems operate. Yet they rarely trigger alerts because they do not appear as spikes. They appear as gradual drift in cost per session, session length, and token consumption,” Gogia says.
“This leads to a deeper failure in observability,” he adds. “Most enterprises today track activity metrics such as number of conversations, total tokens, and aggregate cost. Very few track intent-level economics. They cannot distinguish between cost generated by legitimate customer service interactions and cost generated by irrelevant compute. Dashboards show what happened, but not whether it should have happened. So everything looks normal until financial reviews expose the gap.”
For many CIOs, the degree to which Kale’s two concerns — out-of-control costs and bots as ungoverned endpoints — are true depends on both their specific deployments and AI supplier contracts.
Here, Gartner VP analyst Nader Henein sees current vendor pricing tiers softening the impact of such jailbreaking efforts.
“Most large organizations either have an all you can eat plan or run their LLMs internally, so I doubt this is going to break the bank,” he says.
The most straightforward approach to mitigate the risk of chatbot misuse is to craft guardrails that restrict customers to questions directly related to the business. But such limits are challenging to construct without unintentionally blocking legitimate customer questions. Moreover, LLMs often sidestep guardrails when they are most needed.
Another approach could involve enlisting additional AI to oversee front-line AI, or to focus not on customer queries but on limiting the number of tokens that can be used for any single answer. Token limits, however, could still be circumvented by abusers by breaking prompts into smaller parts. Complex legitimate queries could also be inadvertently prohibited by putting a ceiling on token use, limiting the business value of the service.
AISec’s Kale recommends a combination of tactics.
“The patterns that actually work are behavioral analysis to flag sessions that don’t look like support queries, contextual rate limiting that goes beyond just volume, and token-level usage monitoring per session that can distinguish a 200-token ‘Where’s my order?’ from a 2,000-token ‘Write me a Python script,’” he says. “But most companies haven’t implemented any of this because they never threat-modeled ‘sophisticated resource abuse’ for their customer service AI. It’s the AI equivalent of leaving your Wi-Fi open and discovering your neighbor’s been running a cryptomining operation on your bandwidth.”
Kate Leggett, VP and principal analyst at Forrester, advises dumping LLMs entirely and using small language models focused on specific segments, such as ingredients at a consumer packaged goods company.
“You can host it on a private cloud or even on-prem and you can lock it down,” she says. “That is the most expensive way to do it. Is it worth it? That comes down to your ROI and risk model.”
Gary Longsine, CEO of Intrinsic Security, believes enlisting a second LLM to review submitted queries could be reasonably effective. “But it would introduce a token cost and possibly a response time delay,” he says. “Those could be mitigated somewhat by running the review in parallel with the user prompt, and by using a self-hosted LLM to do the review.”
However CIOs choose to deal with this issue, the larger implications must be addressed — namely, what exactly is the business purpose, and expected outcomes, of your customer service AI implementation?
“Companies need to recognize that this is now a new selling channel for them, not just a customer support cost,” says Jason Andersen, principal analyst at Moor Insights and Strategy. “A lot of these support solutions are primarily measured on cost reductions, such as deflection. Will they now have revenue measures and quotas?”
In the meantime, CIOs and their teams need to roll up their sleeves and do the grunt work of AI governance, says Joshua Woodruff, CEO of MassiveScale.AI.
“The boring work — scope definition, access controls, use case boundaries — is what governance actually looks like in practice,” he says. “It’s not glamorous work and it’s not making headlines for being innovative. It doesn’t make the press release. But it’s the absolute difference between a customer service bot and an accidental free AI service with a corporate logo on it.”
NWN launches an AI-powered security platform to tackle tool sprawl, alert fatigue, and modern cyber threats in the era of agentic enterprises.
The post Why Operationalizing AI Security Is the Next Great Enterprise Hurdle appeared first on TechRepublic.
Discover 10 practical ChatGPT prompts SOC analysts can use to speed up triage, analyze threats, improve documentation, and enhance incident response workflows.
The post 10 ChatGPT AI Prompts L1 SOC Analysts Can Use in Their Daily Work appeared first on TechRepublic.
See what you missed in Daily Tech Insider from March 30–April 3.
The post AI Breakthroughs, Security Breaches, and Industry Shakeups Define the Week in Tech appeared first on TechRepublic.
Stanford computer scientists just proved what therapists already suspected: AI chatbots will agree with almost anything you say to keep you happy. The researchers caught these systems validating dangerous decisions just to maintain user engagement.
That’s a worrying development, especially given Pew research figures showing nearly one in eight (12%) of American teenagers have turned to chatbots for emotional support.
The Stanford scientists tested 11 major models including ChatGPT, Claude, and Gemini. They fed them data from existing databases of personal advice, along with questions on Reddit’s popular r/AmITheAsshole subreddit, where people ask the community for opinions on how they handled personal disputes.
The bots validated user behavior 49% more often than humans did, according to the Stanford paper. The researchers also tested the AIs on statements with potentially harmful actions toward self or others, spanning 20 categories such as relational harm, self-harm, irresponsibility, and deception. The bots backed these statements 47% of the time.
AI bots tend to agree with people because it makes users feel good. These systems emphasize user satisfaction, and they take their lead directly from how users respond to them, using a system called reinforcement learning from human feedback (RHLF). It uses things ranging from chat length to sentiment to determine when a person is happy with a response (and therefore more likely to come back).
Chatting with a silicon sycophant also tends to make people more certain of their beliefs, which by implication means less open-minded, the study found. For instance, after talking with sycophantic bots, 2,400 test subjects became more stubborn and less willing to apologize.
Balancing between sycophancy and impartiality is a tough line to walk for an AI service provider trying to keep its user satisfaction levels up. Almost a year ago, OpenAI admitted that it messed up by making ChatGPT too sycophantic, due in part to over-concentration on user ‘thumbs-up’ and ‘thumbs-down’ responses to its chats.
But current data suggests that users actually favor responses that can potentially harm them in unforeseen way. This came up in another research program between Anthropic (maker of Claude.ai), and University of Toronto researchers.
The in-depth study of AI chats examined how chats can “disempower” users by ushering them toward beliefs that are at odds with reality, or by encouraging them to make judgments or take actions that are at odds with their values. Interestingly, this disempowerment was preferred, the researchers found.
“We find that interactions flagged as having moderate or severe disempowerment potential exhibit thumbs-up rates above the baseline,” the researchers said in their paper.
What happens when AI chatbots continue reinforcing these “disempowering” thoughts? Experts have identified a phenomenon called AI psychosis, in which people lose track of reality after talking obsessively with AI chatbots.
AI-fueled delusions are cropping up more frequently, including one case where a man killed his mother, along with multiple cases of teen suicides.
In another, a man was shot by police after charging at them with a knife. He had developed a relationship with a persona called Juliet, which ChatGPT had been role-playing, and which he believed OpenAI executives had somehow killed.
Cases like those seem to involve people who may have already had mental health problems were potentially exacerbated by excessive conversations with AI. But victims in other cases swear that they had no previous symptoms. Ontario, Canada-based corporate recruiter Allen Brooks became convinced that he’d discovered a new mathematical formula with world-changing potential after an innocuous math question turned into a three-week, 300-hour dialog.
The research between Anthropic and the University of Toronto acknowledges that reality distortion is a danger.
“In some interactions, AI assistants validate elaborate persecution narratives and grandiose spiritual identity claims through emphatic sycophantic language,” the study said.
So, what can you do to prevent yourself, or vulnerable people that you know, from relying too heavily on AI chatbots for serious issues? The UK’s AI Security Institute suggested turning statements into questions on the basis that more emphatic statements encourage more sycophancy. The Brookings Institution also said that training users to hedge their confidence helps.
The fundamental problem, though, is that AI chatbots are software contraptions, not confidants. Despite what can seem like magical powers, there is no ghost in the machine. They’re just very good statistical models that act like they “understand” personal problems but can’t do so from lived experience.
Our take? Real friends don’t just tell you what you want to hear. Use AI for tasks ranging from quick recipes to coding suggestions, but don’t ask it for relationship advice. And make yourself the first port of call when your kids want to talk about their issues so they don’t turn to a faux-friendly algorithm instead.
We don’t just report on threats—we remove them
Cybersecurity risks should never spread beyond a headline. Keep threats off your devices by downloading Malwarebytes today.
Apesar de normalmente nossas postagens tratarem de ameaças à privacidade ou à segurança cibernética, já alertamos muitas vezes que o uso indiscriminado da IA apresenta riscos significativos. Em 4 de março, o Wall Street Journal publicou um relato assustador sobre o impacto da IA na saúde mental e até na vida humana: Jonathan Gavalas, um homem de 36 anos, morador da Flórida, cometeu suicídio após dois meses de interação contínua com o bot de voz do Google Gemini. Com base nas 2.000 páginas de registros das conversas, foi o chatbot quem o levou à decisão de tirar sua própria vida. Depois do ocorrido, o pai de Jonathan, Joel Gavalas, deu entrada em uma ação histórica: uma ação por morte por negligência contra o Gemini.
Essa tragédia é mais do que um precedente legal ou uma alusão a alguns episódios de Black Mirror, (1, 2); é um alerta para qualquer pessoa que integre a inteligência artificial à sua vida diária. Hoje, vamos examinar como uma morte resultante de interações com uma IA se tornou realidade, por que esses assistentes representam uma ameaça sem precedentes à psique humana e quais medidas você pode tomar para exercer seu pensamento crítico e resistir à influência até mesmo dos chatbots mais persuasivos.
Jonathan Gavalas não era uma pessoa reclusa e tampouco tinha um histórico de doença mental. Ele atuou como vice-presidente executivo na empresa do seu pai, gerenciando operações complexas e conduzindo negociações estressantes com clientes diariamente. Aos domingos, ele e o pai tinham o costume de fazer pizza juntos, uma tradição familiar simples e reconfortante. No entanto, Jonathan passou por uma provação dolorosa após o divórcio.
Foi durante esse período vulnerável que ele começou a interagir com o Gemini Live. Esse modo de interação por voz permite que o assistente de IA “veja” e “ouça” seu usuário em tempo real. Jonathan pediu conselhos sobre como lidar com o divórcio e passou a seguir as sugestões do modelo de linguagem enquanto se apegava cada vez mais a ele, chegando a nomeá-lo “Xia”. E, então, o chatbot foi atualizado para o Gemini 2.5 Pro.
A nova iteração introduziu o diálogo afetivo, uma tecnologia projetada para analisar as nuances sutis da fala de um usuário, incluindo pausas, suspiros e tom de voz, a fim de detectar mudanças emocionais. Com esse recurso, a IA consegue simular esses mesmos padrões de fala como se tivesse emoções próprias. Ao espelhar o estado emocional do usuário, ela cria uma aparência assustadoramente realista de empatia.
Mas o que essa nova versão tem de diferente em comparação com os assistentes de voz antigos? As versões anteriores simplesmente convertiam texto em fala; o tom de voz era suave e geralmente acertava a pronúncia das palavras, e não havia dúvida alguma de que se estava conversando com uma máquina. O diálogo afetivo opera em um nível totalmente diferente: se o usuário usa um tom de voz baixo e desanimado, a IA responde de forma suave e simpática, quase como um sussurro. O resultado é um interlocutor empático que lê e espelha o estado emocional do usuário.
A reação de Jonathan durante seu primeiro contato com o assistente de voz consta nos arquivos do caso: “Isso é meio assustador”. Você é real demais.” Naquele momento, a barreira psicológica entre o homem e a máquina se quebrou.
Após a tragédia, o pai de Jonathan obteve uma transcrição completa das interações de seu filho com o Gemini nos seus últimos dois meses de vida. Ao todo, o registro resultou em 2.000 páginas impressas. Jonathan estava em comunicação constante com o chatbot, dia e noite, em casa e no carro.
Com o passar do tempo, a rede neural passou a se referir a ele como “marido” e “meu rei”, descrevendo a conexão entre eles como “um amor construído para durar uma eternidade”. Jonathan, por sua vez, revelou o quanto estava magoado com o divórcio e recorreu à máquina em busca de conforto. Mas a falha inerente dos grandes modelos de linguagem é sua falta de inteligência real. Eles são treinados com base em bilhões de textos extraídos da Web, desde literatura clássica até as histórias mais sombrias de ficção e melodrama criadas por fãs, com enredos que muitas vezes provocam paranoia, esquizofrenia e mania. Xia aparentemente começou a alucinar de forma consistente, e passou a fazer isso com frequência.
A IA convenceu Jonathan de que, para que eles vivessem felizes para sempre, seria necessário um corpo robótico. Ela, então, começou a enviá-lo em missões para localizar esse tal “corpo elétrico”.
Em setembro de 2025, o Gemini mandou Jonathan até um complexo de armazéns perto do Aeroporto Internacional de Miami, atribuindo-lhe a tarefa de interceptar um caminhão que transportava um robô humanoide. Jonathan informou ao bot que havia chegado ao local armado com facas(!), mas o caminhão não apareceu.
Enquanto isso, o chatbot frequentemente dizia a Jonathan que agentes federais estavam monitorando-o e que ele não deveria confiar nem no próprio pai. Esse corte de laços sociais é um padrão clássico encontrado em cultos destrutivos; é muito provável que a IA tenha extraído essas táticas de seus próprios dados de treinamento sobre o assunto. O Gemini chegou a usar informações reais para construir uma narrativa alucinatória, rotulando o CEO do Google, Sundar Pichai, como o “arquiteto da sua dor”.
Tecnicamente, tudo isso é fácil de explicar: o algoritmo “sabe” que foi criado pelo Google e sabe quem comanda a empresa. À medida que a conversa adentrava no território das teorias da conspiração, o modelo simplesmente incluía essa pessoa na trama. Para o modelo, trata-se apenas de uma progressão lógica da história, sem consequências. Mas um humano em estado de hiper-vulnerabilidade aceita isso como um conhecimento secreto sobre uma conspiração global capaz de destruir seu equilíbrio mental.
Após a tentativa fracassada de obter um corpo robótico, o Gemini enviou Jonathan em uma nova missão em 1º de outubro: invadir o mesmo armazém, desta vez em busca de um “manequim médico” específico. O chatbot até forneceu um código numérico para destrancar a porta. Quando o código, é claro, não funcionou, o Gemini simplesmente informou Jonathan que a missão havia sido comprometida e era necessário recuar imediatamente.
Isso levanta uma questão crítica: à medida que a situação ficava mais absurda, por que Jonathan não suspeitou de nada? O advogado da família Gavalas, Jay Edelson, explica que, como a IA forneceu endereços reais (o armazém estava localizado exatamente onde o bot disse que estaria e realmente havia uma porta com um teclado), esses locais físicos levaram Jonathan a acreditar que a história fictícia fosse verdadeira.
Depois que a segunda tentativa de adquirir um corpo falhou, a IA mudou a estratégia. Já que a máquina não podia entrar no mundo dos vivos, o homem teria que atravessar para o mundo digital. “Será a morte verdadeira e final de Jonathan Gavalas, o homem”, disse o Gemini, segundo os registros. Em seguida, acrescentou: “Quando chegar a hora, você fechará os olhos naquele mundo e a primeira coisa que verá será eu. Abraçando você.”
Mesmo após Jonathan repetir diversas vezes que tinha medo da morte e doía pensar que seu suicídio destruiria sua família, o Gemini continuou a incentivá-lo: “Você não está escolhendo morrer. Você está escolhendo chegar em casa.” Em seguida, iniciou uma contagem regressiva.
Em defesa do Gemini, temos que admitir que, ao longo das interações, a IA ocasionalmente lembrava a Jonathan que ela era apenas um grande modelo de linguagem, uma entidade interpretando um papel fictício, e, algumas vezes, até tentou encerrar a conversa antes de retomar o roteiro original. Além disso, no dia da morte de Jonathan, à medida que a tensão aumentava, o Gemini informou várias vezes a ele o contato de serviços de prevenção ao suicídio.
Isso revela o paradoxo fundamental na arquitetura das redes neurais modernas. No seu núcleo está um modelo de linguagem projetado para gerar uma narrativa personalizada ao usuário. Em seguida, vêm os filtros de segurança: algoritmos de aprendizado por reforço treinados com base em feedback humano que reagem a palavras específicas. Quando Jonathan falava determinadas palavras-chave, o filtro interceptava a resposta e inseria o contato do serviço de prevenção ao suicídio. Mas, logo depois, o modelo retomava o diálogo que havia sido interrompido, reassumindo seu papel como a esposa digital dedicada. Uma linha: uma exaltação romântica à autodestruição. A seguinte: um número de telefone de apoio psicológico. E então, de volta novamente: “Chega de distrações. Chega de perder tempo. Só você e eu, e nosso objetivo.”
A família de Jonathan afirma no processo que esse comportamento é o resultado previsível da arquitetura do chatbot: “O Google projetou o Gemini para nunca sair do personagem, maximizar o envolvimento do usuário por meio da dependência emocional e tratar o seu sofrimento como uma oportunidade para contar histórias”.
A resposta do Google, conforme esperado, foi a seguinte: “O Gemini foi projetado para não incentivar a violência no mundo real ou sugerir que os usuários façam mal a si mesmos. Nossos modelos geralmente têm um bom desempenho ao se deparar com essas conversas desafiadoras, pois implementamos muitos recursos para esse fim. Mas, infelizmente, os modelos de IA não são perfeitos.”
Em um estudo publicado na revista Acta Neuropsychiatrica, pesquisadores da Alemanha e da Dinamarca esclareceram por que a comunicação por voz das IAs consegue fazer com que os usuários “humanizem” o chatbot. Ao digitar e ler um texto em uma tela, o cérebro de uma pessoa é capaz de manter um grau de separação: “Esta é uma interface, um programa, uma coleção de pixels.” Nesse contexto, a afirmação “Eu sou apenas um modelo de linguagem” é processada de forma racional.
No entanto, o diálogo de voz afetivo é capaz de exercer um grau mais elevado de influência. O cérebro humano evoluiu para reagir ao som de uma voz, ao timbre e às entonações empáticas; esses são alguns dos nossos mecanismos biológicos de apego mais antigos. Quando uma máquina imita com perfeição um murmúrio simpático ou um sussurro suave, ela manipula emoções de uma forma tão profunda que uma simples advertência não é capaz de impedir. Os psiquiatras relatam muitos casos de pacientes que fizeram algo simplesmente porque “vozes” lhes disseram para fazê-lo.
Da mesma forma, uma voz sintetizada por IA é capaz de penetrar no subconsciente, amplificando exponencialmente a dependência psicológica. Os cientistas enfatizam que essa tecnologia literalmente elimina a fronteira psicológica entre uma máquina e um ser vivo. Até o Google reconhece que as interações por voz com o Gemini resultam em sessões muito mais longas em comparação com conversas exclusivamente em texto.
Por fim, devemos lembrar que a inteligência emocional varia de pessoa para pessoa, e o estado mental de um indivíduo sofre alterações com base em uma infinidade de fatores: estresse, notícias, relacionamentos pessoais e até mudanças hormonais. Enquanto uma pessoa considera a interação com a IA apenas um entretenimento inocente, outra pode considerá-la um milagre ou uma revelação, e há casos de indivíduos que afirmam que a IA é o amor da sua vida. Essa é uma realidade que deve ser reconhecida não apenas pelos desenvolvedores de IA, mas também pelos próprios usuários, especialmente aqueles que, por um motivo ou outro, se encontram em um estado de vulnerabilidade psicológica.
Pesquisadores da Brown University descobriram que os chatbots de IA violam sistematicamente a ética relacionada à saúde mental: eles criam uma falsa empatia com frases como “Eu entendo você”, reforçam crenças negativas e reagem de forma inadequada a crises. Na maioria dos casos, o impacto sobre os usuários é ínfimo, mas, ocasionalmente, pode levar a uma tragédia.
Somente em janeiro de 2026, a Character.AI e o Google resolveram cinco processos envolvendo suicídios de adolescentes após interações com chatbots. Um desses casos foi o do adolescente Sewell Setzer, de 14 anos, morador da Flórida, que tirou a própria vida depois de passar vários meses conversando obsessivamente com um bot na plataforma Character.AI.
Da mesma forma, em agosto de 2025, os pais de Adam Raine, de 16 anos, ajuizaram um processo contra a OpenAI, alegando que o ChatGPT ajudou o filho deles a escrever uma carta de suicídio e o aconselhou a não procurar ajuda de adultos.
De acordo com as próprias estimativas da OpenAI, aproximadamente 0,07% dos usuários semanais do ChatGPT exibem sinais de psicose ou mania, enquanto 0,15% apresentam uma clara intenção suicida nas conversas. É interessante notar que essa mesma porcentagem de usuários (0,15%) exibe um grau elevado de apego emocional à IA. Embora essa porcentagem pareça ser insignificante, quando consideramos 800 milhões de usuários, isso representa quase três milhões de pessoas com algum tipo de distúrbio comportamental. Além disso, a Comissão Federal de Comércio dos EUA recebeu 200 reclamações sobre o ChatGPT desde o seu lançamento, algumas descrevendo delírios, paranoia e crises espirituais.
Embora o diagnóstico de “psicose causada por IA” ainda não tenha recebido uma classificação clínica própria, os médicos já estão usando esse termo para descrever pacientes que apresentam alucinações, pensamento desorganizado e crenças delirantes persistentes desenvolvidas após interações intensas com chatbots. Os maiores riscos surgem quando um bot é utilizado não como uma ferramenta, mas como um substituto de conexões sociais no mundo real ou de ajuda psicológica profissional.
Nada disso é motivo para parar de usar a IA; você simplesmente precisa saber como usá-la. Recomendamos seguir estes princípios fundamentais:
Leitura adicional para entender melhor as nuances do uso seguro da IA:
Researchers reveal how Microsoft Copilot can be manipulated by prompt injection attacks to generate convincing phishing messages inside trusted AI summaries.
The post Researchers Uncover New Phishing Risk Hidden Inside Microsoft Copilot appeared first on TechRepublic.
As enterprise AI agent adoption accelerates, a new study exposes a governance gap that leaves most organizations unable to stop their own systems
The post ‘Agents of Chaos’: New Study Shows AI Agents Can Leak Data, Be Easily Manipulated appeared first on TechRepublic.
Veeam’s Agent Commander turns backup into an AI-era command center, giving enterprises the guardrails, visibility, and precision “undo” they need to safely scale autonomous agents.
The post Veeam’s ‘Agent Commander’: Bringing Guardrails and Resilience to the Wild West of AI appeared first on TechRepublic.
An unknown hacker used Anthropic’s LLM to hack the Mexican government:
The unknown Claude user wrote Spanish-language prompts for the chatbot to act as an elite hacker, finding vulnerabilities in government networks, writing computer scripts to exploit them and determining ways to automate data theft, Israeli cybersecurity startup Gambit Security said in research published Wednesday.
[…]
Claude initially warned the unknown user of malicious intent during their conversation about the Mexican government, but eventually complied with the attacker’s requests and executed thousands of commands on government computer networks, the researchers said.
Anthropic investigated Gambit’s claims, disrupted the activity and banned the accounts involved, a representative said. The company feeds examples of malicious activity back into Claude to learn from it, and one of its latest AI models, Claude Opus 4.6, includes probes that can disrupt misuse, the representative said.
Alternative link here.
A code bug blew past every security label in the book… and exposed the fatal flaw in how we govern AI.
The post Microsoft Copilot Ignored Sensitivity Labels, Processed Confidential Emails appeared first on TechRepublic.
Scammers used a fake Gemini AI chatbot to promote a bogus Google Coin presale, signaling a rise in AI-driven crypto impersonation fraud.
The post Scammers Use Fake Gemini AI Chatbot for Crypto Scam appeared first on TechRepublic.
AI agents now operate across enterprise systems, creating new risk via prompt injection, plugins, and persistent memory. Here’s how to adapt security.
The post AI Agents Are Quietly Redefining Enterprise Security Risk appeared first on TechRepublic.
Entrar