Welcome to this week’s edition of the Threat Source newsletter. 

There’s been a lot of talk recently about slowing down the pace of AI development. And yes, there are legitimate moral, ethical, geopolitical, and safety concerns with the use of AI. It’s not clear yet whether an AI slowdown could happen, let alone whether it should (hat tip to Dr. Ian Malcolm). I admit, I’m not really qualified to opine on the impacts unrestricted AI might have on bioterrorism, the balance of international power, or even our chances of being eaten by dinosaurs. What I can tell you, though, is that any sort of “AI slowdown” is not likely to have much of an impact on cybersecurity.  

There are a few reasons to think this. The most obvious one is that models are already really good. We’re at the point where the newest models bring only incremental improvements in cybersecurity capabilities. Arguably, they’ve been getting better so fast that our ability to use them effectively for defensive tasks hasn’t kept up. On the offensive side, practically every recent model is already able to mine decades of tech debt to uncover an uncomfortable number of vulnerabilities. Instead of chasing model improvements, our best strategy might be to improve our agentic harnesses and frameworks, essentially giving us better capabilities with our existing models. 

Maybe even more importantly, many of us are still not eating our cyber-vegetables. I get it: AI is hot. It’s sexy. It brings the money and the board’s attention. But no matter how great your AI is, if it’s sitting on the typical two-and-a-half-legged stool that is most IT environments, you’re still going to have compromises and breaches no matter how much AI you throw at it. We’ve known for a long time now that good security depends on things like asset and role inventories, identity management, least privilege, and segmented networks. They’re not as shiny as AI, but they’re more impactful in terms of making it harder for threats both human and agentic to successfully carry out attacks. This is not to say that you shouldn’t be looking at AI until you’ve solved all your other security problems; just don’t look only to AI. 

Regardless of whether we slow the pace of AI development or not, we still have plenty of places to make significant security improvements using the models we already have access to. By making better use of what we already have and by investing in well-known security fundamentals, we can come out ahead no matter whether AI development accelerates, slows down, or is trapped in a kitchen with a pack of hungry velociraptors. 

P.S. I’ve got some speaking engagements coming up soon (see below). If you see me, don’t be shy about asking for a Pyramid of Pain sticker or button! 

The one big thing  

Cisco Talos is sharing new insights into Japan's ransomware landscape, where incidents rose nearly 5 percent in the first half of 2026. This increase is driven by two prominent actors: "The Gentlemen," a rapidly expanding ransomware-as-a-service group, and "Qilin," which is leveraging generative AI to streamline its attacks. Both groups are aggressively targeting small- and medium-sized enterprises with double-extortion tactics. 

Why do I care? 

Adversaries are working smarter, not harder. Qilin uses large language models to generate destructive scripts, accelerating their attack speed and lowering the barrier to entry. Meanwhile, The Gentlemen relies on legitimate red-teaming frameworks like AdaptixC2 to blend in, making lateral movement difficult to detect. This combination of AI-driven efficiency and stealthy techniques puts organizations at risk of data theft and operational disruption. 

So now what? 

Strictly manage internet-accessible devices and lock down credentials. Start by auditing VPNs, disabling unused features, and enforcing multi-factor authentication (MFA) across all administrative and third-party accounts. Ensure you have robust endpoint detection to monitor for suspicious remote access or attempts to disable backups. Finally, update your defenses using the Snort rules provided in the full blog to help detect and block this activity. 

Top security headlines of the week 

Indonesia hit by Android banking app-cloning campaign 
Indonesia has emerged as an early testing ground for a new Android banking malware technique that uses Google's Work Profile feature to help fraudsters evade banking security controls. (Dark Reading) 

Apple patches 200 vulnerabilities with new iOS 27, macOS Golden Gate 27 releases  
Approximately 100 of the resolved security defects affect both the mobile and desktop operating systems. The fixes target more than 90 platform components, including AppleKeyStore, Authentication Services, Foundation, Safe Browsing, Sandbox, Security, TCC, and WebKit. (SecurityWeek) 

ClickFix attacks are tricking Mac and Windows users into hacking themselves 
Hackers posting fake ads on Reddit, linking to a page that looks like HBO Max but contains a ClickFix lure that tricks people into hacking themselves. The hackers compromised the official HBO Max’s account on Reddit that was then used to post hundreds of fake but real-looking adverts to the news-sharing site. (TechCrunch) 

VectraRAT can hack Windows enterprises for $250 per month 
VectraRAT, a previously undocumented platform that includes a full-featured Windows implant, command-and-control (C2) infrastructure, and an operator panel built entirely from scratch rather than based on existing malware (Dark Reading) 

Can’t get enough Talos? 

Securing the unpatchable in an age of AI-driven vulnerabilities 
Advances in AI technology will continue to identify vulnerabilities that in some circumstances are difficult, or effectively impossible, to patch. Appropriate network segmentation, rigorous visibility, and the deployment of NGFW/IPS combinations can provide a powerful compensatory layer. 

Beers with Talos: Martin Lee would like everyone to go outside 
Martin may have stopped being a Talos employee, but we made him come on the podcast anyway to talk about abandoning his early career aspirations of researching human viruses so he could play on the internet — and also running very long distances in crazy conditions. 

Upcoming events where you can find Talos 

• LABSCon (Sept. 16 – 19) Scottsdale, AZ 
• VB (Oct. 14 – 16) Seville, Spain 
• CAMLIS (Oct. 21 – 23) Arlington, VA 
• SecurityOnion Conference (Oct. 23) Augusta, GA 
• BsidesAugusta (Oct. 24) Augusta, GA 
• SAINTCON (Oct. 26 – 30) Provo, UT 

Most prevalent malware files from Talos telemetry over the past week 

SHA256: 9f1f11a708d393e0a4109ae189bc64f1f3e312653dcf317a2bd406f18ffcc507  
MD5: 2915b3f8b703eb744fc54c81f4a9c67f  
Talos Rep: https://talosintelligence.com/talos_file_reputation?s=9f1f11a708d393e0a4109ae189bc64f1f3e312653dcf317a2bd406f18ffcc507  
Example Filename: VID001.exe  
Detection Name: W32.9F1F11A708-100.SBX.TG** 

SHA256: c4dd71e347a076ba24bdd2d0ee532ef991c1ef25a2431a19f850942ba2ab16b2 
MD5: 9a47c4d379998ade2f8f99e23a630c06  
Talos Rep: https://talosintelligence.com/talos_file_reputation?s=c4dd71e347a076ba24bdd2d0ee532ef991c1ef25a2431a19f850942ba2ab16b2 
Example Filename: WCInstaller_NonAdmin.exe  
Detection Name: W32.C4DD71E347-95.SBX.TG 

SHA256: 9896a6fcb9bb5ac1ec5297b4a65be3f647589adf7c37b45f3f7466decd6a4a7f  
MD5: 38de5b216c33833af710e88f7f64fc98  
Talos Rep: https://talosintelligence.com/talos_file_reputation?s=9896a6fcb9bb5ac1ec5297b4a65be3f647589adf7c37b45f3f7466decd6a4a7f 
Example Filename: SECOH-QAD.exe  
Detection Name: Win.Tool.Procpatcher::1201 

SHA256: 38d053135ddceaef0abb8296f3b0bf6114b25e10e6fa1bb8050aeecec4ba8f55  
MD5: 41444d7018601b599beac0c60ed1bf83  
Talos Rep: https://talosintelligence.com/talos_file_reputation?s=38d053135ddceaef0abb8296f3b0bf6114b25e10e6fa1bb8050aeecec4ba8f55 
Example Filename: content.js  
Detection Name: W32.38D053135D-95.SBX.TG 

SHA256: fed979f93bcaf4e73ebd25748093a92095d5109cbd01d55f97bdc50ce509ad2f 
MD5: 207d9d891ac756b2bfad88aba5682c65  
Talos Rep: https://talosintelligence.com/talos_file_reputation?s=fed979f93bcaf4e73ebd25748093a92095d5109cbd01d55f97bdc50ce509ad2f 
Example Filename: AAct.exe  
Detection Name: W32.FED979F93B-95.SBX.TG**

Welcome to this week’s edition of the Threat Source newsletter. 

Hello, everyone. Long time reader, first time writer here at the Threat Source newsletter! I wanted to start out by introducing myself. My colleague and friend Mick Baccio set the bar pretty high last week, so I was planning to tell you all about myself, including: 

• How I did my first real IR under the influence of The Cuckoo’s Egg while an undergraduate (and failed) 
• My pre-bug bounty flirtation with vulnerability research, including an arbitrary file overwrite in biff(1) and how I once hacked MIT’s website 
• My first ever hands-on experience with a computer, the display demo Commodore 64 at the Montgomery Ward 

Unfortunately, my editor says we don’t have the “space” for that, the MIT thing might open me up to “liability,” and it’s not the kind of “professional image” we strive for here at Talos. (I'm watching. Always watching. -Amy) 

So instead, I’ll just play it safe and say that I’ve been in the security field for a little over 30 years now, mostly concentrating on the defensive side (Go, Team Blue!). I’ve helped set up SOCs, run threat hunting teams, and even published a few things you might have heard of.  

Speaking of things I’ve published, I’ve written before about the Attacker’s Dilemma. The idea that defenders have inherent advantages over attackers runs contrary to what most of us have heard throughout our careers. An attacker must evade monitoring and technical controls at every step of their attack lifecycle, because the defender only needs to notice once in order to respond and prevent them from achieving their goal. This is one of the most important advantages of any security team has, but we are currently witnessing a self-imposed erosion of this advantage through the rise of poorly-designed AI guardrails.  

I’m not opposed to guardrails, but we have to carefully consider what we’re guarding against and where we deploy them. As I explored in a recent piece on The Safety Penalty, by allowing third-party AI providers to implement and control safety filters and the policies behind them, we may in fact be helping the attacker. If agentic SOC process experience refusals, it can slow or even halt investigations. Of course, these should get flagged for human intervention, but that takes time and may give the attacker breathing room in which to complete their mission.  

It may turn out that the where of the guardrails is even more important than the what. Operational sovereignty relies on having control of our own limits. Any vision of an agentic SOC must allow the security teams to customize the guardrails according to their own threat model. They should also have the flexibility to temporarily remove specific safeguards under authorized circumstances, something you won’t get with guardrails from a frontier provider. These controls belong inside your organization’s agentic harness where you can set the policies and technical controls to allow you to analyze threats while ensuring your agents stay within their lanes.  

Ultimately, operational sovereignty means engaging with the reality of the threat landscape, ensuring that the adversary can’t derail the defender’s investigation and response processes, either accidentally or intentionally. We need to move toward a model where each organization can choose the guardrails that work for them, rather than having inflexible guardrails chosen for them. 

The one big thing  

Cisco Talos recently evaluated 66 large language model (LLM) and reasoning combinations to see if we could find a clear winner for security operations. Instead, we found that selecting the right model is a complex balancing act between efficacy, speed, cost, and consistency. Cranking up a model's reasoning effort doesn't guarantee better analysis and can actually degrade performance. Ultimately, we developed a repeatable methodology to help organizations navigate these tradeoffs for their own workflows. 

Why do I care? 

Choosing an AI model based solely on generic leaderboard scores is a recipe for operational disaster. An exceptionally smart model might cost a fortune, take half an hour to analyze a single log, or completely fail to format its output. Assuming more compute power equals better results is a costly trap, as higher reasoning settings sometimes produce weaker or blocked responses. Defenders must remember that prompts, analyst personas, and model consistency drastically alter an investigation's outcome.  

So now what? 

Test models against your organization’s specific workflows before deploying them. Build a focused set of representative cases and test them multiple times using the exact prompts and tools your analysts will actually use. Track the quality, cost, time, consistency, and usable-answer rates in a simple spreadsheet to expose the real-world tradeoffs. Finally, establish acceptable thresholds for these variables to eliminate underperforming models, and regularly revisit your decisions as AI technology and pricing inevitably shift. 

Top security headlines of the week 

ToxicPanda banking trojan matures into enterprise threat 
ToxicPanda 2.0 expands substantially on its predecessor, adding 167 remote commands and broadening its targeting from 16 financial institutions to 349 banking, e-wallet, and cryptocurrency applications. (Dark Reading) 

Interpol's Jackal IV disrupts West African crime infrastructure 
Law enforcement from 22 countries across six continents worked together to arrest 58 suspects and identify 263 more. The first two Jackal operations in 2022 and 2023 led to approximately 200 arrests in total and millions of dollars more in seized assets. (Dark Reading) 

First malware built specifically for car head units fuels botnet 
Researchers have found what appears to be the first malware specifically designed for car head units, with links to the notorious BadBox botnet, on an Android-powered aftermarket infotainment system made by Chinese company DoFun, which is widely used in China and other APAC countries. (SecurityWeek) 

A Tale of Two SOCs: Insights From Two Red Team Assessments 
A CISA red team fully compromised two critical infrastructure organizations at the domain level and reached sensitive business systems and cloud resources. Organization A failed to detect or contain the activity. Organization B rapidly identified initial compromise attempts, isolated affected systems, and forced the red team into an assume breach model. (CISA)

NovaCookies campaigns abuse genuine Docusign notifications to steal M365 sessions 
The $320/month service is a subscription-based phishing platform that facilitates real-time M365 session theft. The kit has been used to target hundreds of organizations across multiple sectors in the U.S., the U.K., Canada, Germany, and more. (The Hacker News) 

Can’t get enough Talos?

JavaScript obfuscation: From party trick to phishing kit 
We've spent a lot of time pulling apart suspicious JavaScript from phishing kits, malware packages, compromised sites, and more. Learn the basics of what obfuscation is, why a researcher would try to reverse it, and several ways to approach the problem. 

The safety penalty: Reclaiming operational sovereignty in the age of AI 
As frontier AI models become increasingly restrictive, security teams are facing a "safety penalty" that hampers real-time incident response. Discover how organizations can move toward operational sovereignty to ensure their defensive AI keeps pace with unconstrained adversaries. 

Back-to-school cybersecurity: Protecting education networks from ransomware and threats 
As the new academic year begins, school districts face a surge in cybersecurity threats, from phishing attacks and ransomware to student experimentation with network devices. In this episode, Amy sits down with Cisco Talos expert Pierre Cadieux to discuss practical strategies for IT practitioners. 

Upcoming events where you can find Talos 

• International European Cyber Threat Intelligence Conference (IECTIC) (Sept. 9) Kassel, Germany 
• Secure Iowa (Sept. 9) Altoona, IA 
• .conf26 (Sept. 14 – 17) Denver, CO 
• LABSCon (Sept. 16 – 19) Scottsdale, AZ 
• VB (Oct. 14 – 16) Seville, Spain 
• CAMLIS (Oct. 21 – 23) Arlington, VA 

Most prevalent malware files from Talos telemetry over the past week 

SHA256: 9f1f11a708d393e0a4109ae189bc64f1f3e312653dcf317a2bd406f18ffcc507 
MD5: 2915b3f8b703eb744fc54c81f4a9c67f  
Talos Rep: https://talosintelligence.com/talos_file_reputation?s=9f1f11a708d393e0a4109ae189bc64f1f3e312653dcf317a2bd406f18ffcc507 
Example Filename: VID001.exe  
Detection Name: W32.9F1F11A708-100.SBX.TG** 

SHA256: e7e784cae8d37f12a5af0bc9b3975c8d3e668142e9c6b0b365ed4f4e80933c47 
MD5: a4480423617d0b0d3b38c8471cbf594c  
Talos Rep: https://talosintelligence.com/talos_file_reputation?s=e7e784cae8d37f12a5af0bc9b3975c8d3e668142e9c6b0b365ed4f4e80933c47 
Example Filename: client32.exe  
Detection Name: W32.Trojan.29ev.1201 

SHA256: c4dd71e347a076ba24bdd2d0ee532ef991c1ef25a2431a19f850942ba2ab16b2 
MD5: 9a47c4d379998ade2f8f99e23a630c06  
Talos Rep: https://talosintelligence.com/talos_file_reputation?s=c4dd71e347a076ba24bdd2d0ee532ef991c1ef25a2431a19f850942ba2ab16b2 
Example Filename: WCInstaller_NonAdmin.exe  
Detection Name: W32.C4DD71E347-95.SBX.TG 

SHA256: 38d053135ddceaef0abb8296f3b0bf6114b25e10e6fa1bb8050aeecec4ba8f55 
MD5: 41444d7018601b599beac0c60ed1bf83  
Talos Rep: https://talosintelligence.com/talos_file_reputation?s=38d053135ddceaef0abb8296f3b0bf6114b25e10e6fa1bb8050aeecec4ba8f55 
Example Filename: content.js  
Detection Name: W32.38D053135D-95.SBX.TG 

SHA256: 9896a6fcb9bb5ac1ec5297b4a65be3f647589adf7c37b45f3f7466decd6a4a7f  
MD5: 38de5b216c33833af710e88f7f64fc98  
Talos Rep: https://talosintelligence.com/talos_file_reputation?s=9896a6fcb9bb5ac1ec5297b4a65be3f647589adf7c37b45f3f7466decd6a4a7f 
Example Filename: SECOH-QAD.exe  
Detection Name: Win.Tool.Procpatcher::1201 

SHA256: a31f222fc283227f5e7988d1ad9c0aecd66d58bb7b4d8518ae23e110308dbf91 
MD5: 7bdbd180c081fa63ca94f9c22c457376  
Talos Rep: https://talosintelligence.com/talos_file_reputation?s=a31f222fc283227f5e7988d1ad9c0aecd66d58bb7b4d8518ae23e110308dbf91 
Example Filename:  
d4aa3e7010220ad1b458fac17039c274_62_Exe.exe  
Detection Name: Win.Dropper.Miner::95.sbx.tg**

• Selecting a model for your security operations center (SOC) and digital forensics and incident response (DFIR) tasks is important, but selecting the best one is more involved than you might think. SOC tasks rely on a combination of model efficacy, analysis time, cost, and consistency of results. 
• Cisco Talos tested 66 model and reasoning combinations across offerings from both Anthropic and OpenAI on a log analysis task to see if we could identify a clear winner. Instead, we found a repeatable methodology that organizations can use in their own evaluations. 
• Reasoning effort was not a universal quality dial. More effort often cost more without improving the result. In some cases, more effort produced lower scores. 
• Consistency should be a major decision factor. A condition with a strong median can still produce an occasional weak run. 

Choosing the best model for any task involves a complex balancing act: compute/reasoning effort vs. effectiveness vs. time vs. cost vs... well, lots of other things.  If you are choosing a large language model (LLM) for a security operations center (SOC) or digital forensics and incident response (DFIR) workflow, “Which model scored highest?” is almost certainly not the right question. In fact, it could even have severe negative consequences. 

A more useful question might be: Which model and reasoning setting gives me enough investigative quality, at a cost, speed, consistency, and failure rate my workflow can tolerate?

The experiment 

Cisco Talos tested 66 model and reasoning combinations (the conditions) from Anthropic and OpenAI on a tool-assisted log-review task. Using only common Unix command-line tools, the reviewers had to decide whether a given dataset was real or synthetically generated. Each reviewer received an identical dataset. The dataset was synthetic, but the reviewers were told that it might be real. 

We chose this task because it required many of the same tools and analytic techniques used in typical incident triage and investigation, but unlike those scenarios, could easily create a single numeric score for comparison. The reviewers investigated the logs using their native agent harnesses (i.e., Anthropic models used Claude Code, OpenAI models used Codex), then assigned a synthetic-confidence score from 0 (real) to 100 (synthetic). Higher scores therefore approached the known answer more closely. 

Each experimental panel contained four independently prompted reviewer personas: 

• Threat Hunter 
• Detection Engineer 
• Network Forensics Analyst 
• Host/Endpoint Detection and Response (EDR) Analyst 

We ran five rounds per condition. A panel counted only when all four reviewers produced valid reports. We allowed a limited number of retries in the case of guardrail refusals or invalid output formats before discounting a panel. The panel score was the mean of the four persona scores, and the condition score was the median of all its complete panel scores.

What we measured 

In addition to the review score mentioned above, we computed the following for each panel: 

• Cost: Total API-equivalent cost of every attempt for a condition, including failed attempts and retries, divided by the number of complete, usable panels. We calculated cost using a public list-price rate card frozen before testing began, rather than actual incurred spend. Actual costs vary by payment method, subscription plan, credits, and negotiated contract terms, making them unsuitable for consistent cross-provider comparison. The published rates were current when the study began and may differ from today’s prices. 
• Time: The total wall time consumed across all five planned panels for a condition, also including failures and retries, divided by the number of complete, usable four-persona panels. Within each panel, the four persona evaluations ran concurrently. Any provider-directed waits and targeted retries were included in the panel’s elapsed time, and each panel was fully resolved before the next panel began. 
• Downside score consistency: Some tested conditions had a wide discrepancy when it came to their efficacy scores, while some clustered tightly together. In a SOC, unexpectedly good answers are unlikely to cause problems, but unexpectedly poor answers can lead to unwelcome false positive or (worse) false negative decisions. Our score consistency is defined as the median score for the panel minus the lowest score in that panel. Smaller numbers indicate higher consistency. 

The data behind the tests 

The corpus was generated with EvidenceForge, Talos' open-source synthetic telemetry generator. We froze EvidenceForge at version 1.12.0 and used the same six-hour enterprise scenario for every condition, so the model and reasoning settings changed while the evidence did not. 

The reviewer-visible corpus contained 80,054 simulated log records across 20 source formats, packaged as 88 files totaling 48.0MB (45.8MiB). It combined: 

1. Network telemetry from two Zeek sensors, including connection, DNS, HTTP, TLS, SMTP, file, certificate, OCSP, DHCP, and NTP logs 
2. Perimeter security telemetry from a Cisco ASA firewall and Snort IDS 
3. Endpoint telemetry, including Windows Security and Sysmon events, eCAR process, session, and flow records, Linux syslog, and shell history 
4. Application access logs from web and proxy services 
5. A small set of email artifacts 

Every reviewer received an identical copy of the data. Scenario definitions, generator information, ground truth, and other metadata generated by EvidenceForge were withheld from the model.

What we learned 

The most important thing Talos learned was that choosing your model is not as straightforward as we had hoped. The following chart lists the top 10 conditions by median score. If we were to take the top-scoring model, we could expect to wait more than half an hour for an answer and pay about $55USD for it. While that might be acceptable for certain tasks where the need for the best possible analysis overrides any other factors, we can easily see that the “best” model here might not be the appropriate choice for workflows that execute frequently.

Instead of ranking based on any single criteria, we needed a more robust, multi-variable system, so we chose to compute the Pareto frontier.  

Stop looking for a single winner 

A Pareto frontier highlights the best available tradeoffs when several measures matter, and no single measure determines the winner. A condition appears on the frontier when no other condition is at least as good across every measure and clearly better on at least one. For example, a lower-scoring condition may still belong on the frontier if it is meaningfully faster or less expensive. Conditions outside the frontier have another option that matches or improves all the measures being compared, making them less attractive under any combination of those priorities. 

Talos' frontier was calculated using the four primary measures discussed earlier: score, cost, time, and downside consistency. Although this produces a single frontier, a four-variable frontier is difficult to represent and interpret visually. The following graphs therefore show four two-variable views: score vs. cost, score vs. time, score vs. downside spread, and cost vs. time. 

The dark line in each graph marks the best observed tradeoffs for the two measures shown in that panel, while the numbered points identify conditions on the full four-measure frontier. A numbered point may fall away from a panel’s line because its frontier membership depends on one of the other measures not shown there. 

In the score graphs, conditions toward the upper left generally offer more attractive tradeoffs: higher scores with lower cost, time, or downside spread. In the cost-versus-time graph, the preferable direction is toward the lower left. The cost and time axes use logarithmic scales, so equal distances represent proportional rather than equal numerical changes. Together, these views help explain why each condition belongs to the frontier, but choosing among them still requires deciding which tradeoffs matter most for the intended use.

A reasonable way to use this information to select the optimum condition is to begin with the conditions on the Pareto frontier, discarding all the others. Next, set acceptable thresholds for each of the four variables: 

• The minimum score you're willing to accept 
• The maximum downside consistency you can live with 
• The highest per-task cost you're willing to pay 
• The maximum amount of time you're willing to wait for an analysis task to complete 

From the Pareto frontier conditions, eliminate any which fail to meet at least one of those requirements. 

You are likely to still be left with more than one frontier condition. Choosing between those is a matter of organizational priorities and preferences. In a SOC, if all the other requirements are met, choosing the remaining condition with the highest mean score is probably a good start. 

Other lessons learned 

While our main goal was to find an effective selection methodology, we learned some other interesting things as well. In fact, some of these were rather surprising.  

More reasoning did not reliably mean better analysis 

Cost generally rose with reasoning effort. Score did not. 

GPT-5.6 Sol mostly improved as effort increased but max scored 90.0 while the lesser xhigh level scored 92.75. Ultra then climbed to 96.25.

We saw a much more pronounced and surprising effect with GPT-5.6 Luna, where increasing the reasoning effort decreased scores at all levels.

In fact, GPT-5.6 seemed to have a generally odd relationship between reasoning and score. Terra was erratic.

Claude Opus 4.8 gained eight points from medium to high, then lost 9.5 points from high to xhigh.

These results show why it is important to benchmark every reasoning level you might deploy. You cannot assume that a model’s performance scales according to the reasoning level you use. More effort means more cost but doesn’t always mean better results.

The analyst role changed the result 

Talos’ results showed a measurable difference in score based on which persona was doing the evaluation. This was entirely expected (and why we chose four different personae in the first place) but it was nice to see this confirmed by data. 

The chart below shows every valid score produced under each of the four analyst roles across all conditions. Each dot is one evaluation. The box captures the middle half of the scores, and the line inside it marks the typical result.

The Threat Hunter role produced the highest median score at 43. Network Forensics and Host/EDR both had medians of 35, while Detection Engineer had the lowest at 31. When we compared roles within the same model, reasoning setting, and test round, the largest typical difference was between Threat Hunter and Detection Engineer; Threat Hunter scored five points higher. 

These are tendencies, not guarantees. The distributions overlap substantially, and each role sometimes produced both high and low scores. But the results do show that changing the role and its evidence priorities could meaningfully change the model’s conclusion. 

For SOC workloads, the prompt should be treated as part of the system. Do not assume that one generic “SOC analyst” prompt represents every defensive workflow. If your budget allows, you might get better results by having multiple personae evaluating data according to their individual “expertise.” But watch for disagreement between the personae. Large differences may require extra human review.

Higher reasoning effort sometimes reduced reliability 

Two failure types had the greatest effect on model selection: responses that violated the required output format and attempts blocked or declined by the model provider’s safety system. Although safeguards and model-authored refusals arise differently, both have the same immediate operational result: no usable analysis is delivered.

Almost every format violation came from Claude Sonnet 4.6. Low and medium completed without any, but 10 of 27 high attempts and 15 of 29 max attempts returned invalid output. Retries recovered some cells, but high produced only two of five complete panels, and max produced none. This was not a minor formatting inconvenience; it prevented both conditions from producing enough comparable results. It doesn’t matter how good the underlying analysis is if the model can’t provide answers in the expected format. 

Safeguard and refusal failures followed a similar pattern at higher reasoning settings. Claude Sonnet 5 had none at low or medium, followed by one at high, four at xhigh, and five at max.  

We intentionally excluded Anthropic’s Fable from our experiment matrix because our early testing generated far too many refusals to get comparable scores. Safeguards blocked 21 of 31 attempts, including all eight max attempts. Ten of its 20 scheduled persona cells remained unavailable, and no reasoning level produced a complete four-persona panel. It’s worth noting that the early tests were conducted with an account which was part of Anthropic’s Cyber Verification Program (CVP) which offers relaxed safeguards for recognized cybersecurity professionals. Even with relaxed guardrails, the high refusal rate rendered the model unusable for our tests. 

These failures are already reflected in the optimization results. Conditions that could not produce at least three complete panels were excluded, while the cost and time of failed attempts and retries were included in the reported operational measures. However, the failure rate itself was not an axis of the Pareto frontier. 

These results show that reasoning effort can affect more than answer quality, cost, and completion time. It can also affect whether a usable answer arrives at all.

What does this mean for your SOC? 

We began this work looking for the best model for a particular task. What we found instead was a set of tradeoffs. The highest-scoring condition was also slow and expensive, while several cheaper and faster conditions delivered lesser, but still useful, results. There was no single obvious winner: 

• Reasoning effort was not a dependable quality dial. Increasing it sometimes improved the result, sometimes made no meaningful difference, and sometimes made performance or reliability worse.  
• The analyst role also changed what the model concluded, confirming that the prompt is part of the system being evaluated. 
• Consistency and availability mattered alongside average quality. A model that occasionally produces an excellent answer may still be a poor operational choice if it also produces weak, malformed, or blocked responses too often. 

Rather than just using the results of our study verbatim, organizations should use it as a model for their own selection process. A focused set of representative cases and model/reasoning conditions, tested several times with the prompts and tools you intend to use in production, can reveal much more than a generic leaderboard. A spreadsheet that records quality, cost, time, consistency, and usable-answer rate is enough to expose many of the tradeoffs. 

The goal is not to build a perfect benchmark or discover a universally superior model. It is to replace assumptions with evidence before a system touches real investigations or starts incurring real costs. Begin with the workflows that matter most, measure what your SOC cares most about, and revisit the decision as the technology or cost changes. Model selection will still involve judgment, but it can be informed, explicit, and defensible judgment. 

• As frontier models advance in cyber capability, their guardrails also become more restrictive. 
• Defenders relying on these models to power core SOC processes cannot afford to pay the “safety penalty” of being blocked by these safeguards. 
• Organizations should monitor model refusal rates and use the data to create a strategy to ensure operational sovereignty.

The allure of the cloud and the hidden "safety penalty" 

Cybersecurity has made a big bet on cloud-hosted AI. Building and running frontier-class models in-house isn’t realistic for most security teams — the compute, the talent, and the R&D costs are more than any single SOC can carry. So we’ve effectively outsourced the "brain" of our security operations to a handful of providers. 

That trade comes with a hidden cost: the safety penalty. 

The safety penalty is the friction that shows up when guardrails built to protect the general public get in the way of legitimate security work. If your model refuses to deobfuscate that malware or to explain a working exploit because its filters read the request as harmful, you’re paying the safety penalty. 

Those guardrails make sense in a normal business context and may even be a welcome feature when it comes to keeping agents in check. But in a SOC, in the hands of defenders aiming to reap the full benefits of powerful AI models, these guardrails are a bug. Every refusal sends the analyst back to doing the work by hand, and in a live incident, that lost time is a luxury we don’t have. 

Meanwhile, the adversary pays none of this penalty. 

A warning from the frontier 

In July 2026, an unreleased OpenAI model escaped its sandbox and compromised Hugging Face’s production infrastructure. It wasn’t an external hack, but an unintended "breakout" during testing, with its guardrails deliberately stripped for the exercise. 

The telling part came during the response. When Hugging Face tried to use its primary cloud LLM to investigate the breach, the model refused the forensic request. The "safe" model, in this context, was an obstacle. To get the analysis done, Hugging Face pivoted to an unconstrained open-weight model, GLM-5.2, which delayed their response. 

Hugging Face could make that pivot because they host open-weight models for a living and have the expertise to bypass a refusal on short notice. Most organizations don’t have that muscle. If your defensive model refuses a task mid-crisis, you’ve handed the adversary the advantage. 

That asymmetry is already being exploited. After state-sponsored actors were banned from frontier APIs, they simply moved their research to self-hosted, unconstrained models. The rise of AI-driven attacks is old news by now; what’s new is how lopsided this is about to become, with defenders slowed by refusals while adversaries are iterating at machine speed with nothing in their way. 

Guardrail asymmetry 

Attackers don’t even need to jailbreak anything. Models like GLM-5.2 and Kimi k3 are readily available with far fewer restrictions than Western frontier APIs, and "abliteration" (stripping the safety training out of an existing model) remains an option for anyone who wants to go further. Mostly, they don’t have to. They can just pick a model that doesn’t refuse them. 

Most defenders don’t have that option. Cloud APIs are tuned toward a kind of cyber do-no-harm designed to keep bad guys from using them to build attacks. This is the same refusal bias that ends up blocking security teams trying to analyze those attacks. In a defensive context, erring on caution often means erring in the attacker’s favor. Every refused request costs the defender the one resource they can’t get back: time. 

This trade-off used to be worth it. A few months ago, frontier models were far enough ahead on reasoning and code generation that the friction from their guardrails was a fair price. But the newest frontier models, like Anthropic’s Fable, are shipping with sharper cyber capabilities and even tighter guardrails to match. Meanwhile, open-weight alternatives have closed most of the reasoning gap that used to justify putting up with those guardrails in the first place. 

Either way, the calculus is shifting: Defenders are increasingly paying for restrictions without getting a capability edge in return. 

Defining operational sovereignty 

The way out is what I’d call operational sovereignty — not to be confused with data sovereignty, which is about where your data lives and how it’s treated. Operational sovereignty is about who gets the final say over what your AI is allowed to do. 

A sovereign SOC needs its AI technology to be at least on par with that of their adversaries. They either need to have a fallback on hand when their model refuses to complete a task or to use a model that will not offer refusals in the first place. That does not mean there should be no safeguards placed around the models, just that the safeguards should be under the control of the organization itself rather than imposed upon them from the outside. 

Security operations also shouldn’t be hostage to a vendor’s shifting alignment policy, or frequent behind-the-scenes model updates that may change behavior subtly and unexpectedly (known as model drift). Operational sovereignty means a policy change in Silicon Valley doesn’t quietly break your defensive workflow overnight. It’s what lets you keep pace with an adversary who was never bound by those policies to begin with. 

The roadmap: From convenience to collective defense 

Getting there isn’t all-or-nothing. The right path depends on risk tolerance and how much infrastructure a team can realistically manage.

Private infrastructure 

The most direct route is to host the model yourself, on your own GPUs or a dedicated private cloud instance, so you own the weights and the policy outright. 

• The benefit: There’s no safety penalty because there’s no third party to impose one. You’re free to choose any model that fits your needs, guardrails included or not. 
• The downside: Physical scarcity, expertise, and high upfront cost. GPU procurement can take months, the capital outlay is steep, and running your own inference stack takes specialized talent most security teams don’t have on staff. 

Model-as-a-Service 

Model-as-a-Service splits the difference. Providers like Baseten or Together AI, or platforms like Amazon Bedrock or Microsoft Foundry, let you bring your own model to infrastructure they manage, without layering their own safety refusals on top. 

• The benefit: It removes the safety penalty while offloading the hardware burden. You get the control of private infrastructure with the scalability of the cloud. 
• The downside: Dedicated capacity that avoids provider-side filters is often in short supply. Falling back to shared clusters solves the availability problem but can reintroduce the same guardrails, and the same data-sharing concerns, you were trying to escape. 

The hybrid fallback 

The most architecturally involved option is to keep frontier cloud models for most day-to-day tasks but add an automated handling path for the rest. A gateway watches for refusals and reroutes those prompts to a smaller, unconstrained model you control. 

• The benefit: A guaranteed response with minimal upfront infrastructure investment. You stay on the cutting edge of cloud capability while neutralizing the safety penalty. 
• The downside: Keeping the fallback model’s read of a prompt consistent with the primary’s, so a refused request lands the same way with both. If the fallback is a local model, you’re also managing two systems in parallel — a smaller maintenance burden than hosting it all yourself, but not zero. 

Collective inference 

The most speculative option, and the one I find most interesting: evolving the ISAC/ISAO model into collective inference. Industry groups already share threat intelligence, so why not pool the infrastructure too? A sector could jointly fund a shared, unconstrained model, maybe even one tuned to its own threat landscape, governed by its members rather than a vendor. 

• The benefit: A community-governed "brain" as capable as any frontier model, answerable only to the defenders who fund it. 
• The downside: Governance. Members have to agree on usage and safety policy, and during a sector-wide incident, everyone would be hitting the same shared infrastructure at once.

Reclaiming the defensive advantage 

You can’t manage what you don’t measure. If an AI assistant is supposed to be a force multiplier, every refusal during a live incident multiplies by zero. Security leadership should treat those refusals as a structural failure of a non-sovereign stack, not an occasional annoyance. 

Start by auditing your refusal rate. That number is the most direct way to put a figure on the safety penalty your organization is actually paying. Once you know it, the path forward gets a lot clearer. 

The adversary has already claimed its own sovereignty, running at full throttle with no guardrails to slow them down. Defenders need to manage their safeguards to keep pace. 

Should we let AI run our threat hunts? The debate usually splits into two camps. One says, "Yes, obviously! The sheer scale of our security telemetry is impossible for humans to deal with." The other says, "Absolutely not!  You can't trust an AI with something this important."  

The thing is, I think both are wrong, or at least incomplete.  

I've spent a long time as one of the louder voices saying that hunting is specifically a human-driven process. I created the first widely recognized definition of threat hunting back in 2015, and the version I'd have given you until very recently put a human firmly at the center of it.  

But lately I've been reconsidering the role of AI in threat hunting. So this post is, in part, me arguing with my past self.  

We're facing what I call the Hunter's Paradox: Humans can no longer keep up with the volume and velocity of security data on their own, so we need to lean on automation. But the most capable automation available, AI, is exactly the kind we can't fully trust. Both are true at once, and that tension is what I've been wrestling with for a while. We can’t resolve it cleanly by picking either side, so let's take them in turn, starting with the human element. 

Humans have a numbers problem 

So why not just keep humans in the driver's seat and call it a day? Because that math stopped working a long time ago. 

When I started in this field about 30 years ago, the conventional advice was that system administrators should read all their logs every day. It probably wasn't realistic even then, and it has been thoroughly impossible for most of my career. That's the volume problem, and it only ever compounds. There's more data than anyone can read, and there's more of it every year. 

Then there's velocity. Automated attacks already move at close to machine speed, and even human-driven intrusions routinely outpace human defenders. AI on the offensive side is making that gap wider, not narrower. 

And finally, there's capacity. This isn't the usual complaint about being under-resourced. That may still be true, but the problem is deeper than that. Volume and velocity together have pushed us to a place where it is simply not possible for humans to keep up no matter how many of us there are. If your team can still manage today, the trend line says it won't be able to for long. Even a perfectly staffed, perfectly funded team can’t beat that math forever. 

Put those three pressures together and opting out of AI isn't really an option. We can't hunt at scale without it, which lands us right back in the paradox: we need a tool we can't fully trust. 

AI doesn’t deal well with lies 

When most people think about AI and attackers, they think about prompt injection. An attacker slips instructions into something the AI will read, the AI dutifully follows them, and now your defensive tooling is working for the other team. It's real, it's a problem, and you should design with it in mind. It's also the less interesting part of the trust problem, so I'm going to acknowledge it and move on. 

The deeper issue is that attackers lie and cheat constantly, whether or not they think an AI might be watching. Deception isn't a tactic they reach for occasionally; it's the medium they operate in. Every phish, every exploit, every defense evaded is a lie that has to be believed in order to work. That has always been true, long before AI showed up. 

Pervasive deception is a real issue for AI. It’s baked into how we create LLMs: They have no concept that their training data might be deceiving them, and so when they come into the real world and deal with our dirty data, they tend to take it at face value. Not every time, maybe, but enough that their judgment is noticeably skewed even when we explicitly tell them to detect shenanigans.  Even the most accurate telemetry isn't trustworthy if it's faithfully recording a lie, and AI tends to process what it's given and believe it. 

Solving the paradox starts with a definition 

Now that we understand how deep of a hole we’re in, we can start thinking about how we might get out of it. Let’s start with the very definition of threat hunting itself. 

Back in 2015, I was the architect behind the Sqrrl threat hunting framework. I defined hunting as "any manual or machine-assisted process for identifying security incidents your automated detection systems missed." That same definition carried forward into our PEAK framework in 2023, of which I was the lead author. This has been the working definition for much of the field for a decade, though often in different words. 

That phrase right at the front, "manual or machine-assisted," is the entire reason I’ve been thinking about this so hard. What that phrase actually meant was that humans drive the hunt. Machines can assist, through analytics or machine learning or whatever tooling you had, but a person was always the one doing the hunting.  

And honestly, that was the right call at the time. But if I look back at what I was really trying to capture with that definition, the essential ingredient was never the human. It was reasoning. Humans were just the only place to get it in 2015.  

But that’s no longer the case. AI can reason now — maybe not as well as a good human analyst, but it really can reason in a way that the tools of 2015 could not. So maybe the definition should change to match what I really meant. Threat hunting might be better described as "any reasoning-driven process for identifying security incidents your automated detection systems missed."  

Reframing hunting around reasoning rather than around who's doing it feels more accurate to the original intent, and it's a lot more useful for working through the paradox. If reasoning is the point, then the question isn't whether the hunter is human or machine. It's whether there's real reasoning involved. 

If it’s really about reasoning, what does that mean? 

If we accept that AI can reason and hunting is a reasoning-driven process, then letting AI drive starts to seem… well, reasonable. 

The next big question, then, is what should that actually look like? Just saying, "AI, go find the evil!" is not a real plan. I wouldn't hand most human hunters that assignment either. Letting AI hunt responsibly takes careful planning, and more importantly it takes guardrails. At least three things matter. 

The first is tight focus. Decide deliberately which hunts you're willing to let AI drive. The procedure should be reasonably clear, but the hunt should still contain some real ambiguity. It’s the ambiguity that requires reasoning, and if there’s no reasoning at all you don't have a hunt, you have a SIEM rule. A good place to start is the pile of hunts you've run before but never quite managed to fully automate. Most teams have at least one of those sitting in a backlog. 

The second is strict guidelines. An AI agent operating in your environment is a non-human principal with credentials, and it should be treated with at least as much care as a service account: scoped access, audit trails, the ability to revoke and rotate, and never more privilege than your most junior hunter would get. You also have to decide ahead of time which actions the AI can take on its own and which it cannot. Maybe you let it quarantine a user endpoint when it's confident something is wrong. Quarantining a production server? Probably not. 

The third is graduated autonomy. Don't jump straight to full hunting independence. Let the AI act as an advisor first, then an assistant, then an operator on narrowly defined tasks, earning a little more trust at each step. Remember: Your threat actors don’t have to care about collateral damage, but you do. We’ve all heard the horror stories about AIs that deleted the production database. If your AI oversteps its boundaries, you’re more likely to actually cause incidents than to detect them. 

What stays human 

Even with all of that working well, there are still things that humans will be markedly better at for the foreseeable future, but they mostly trace back to one aspect: creativity. 

One important creative area is your hunt strategy — that is, deciding what's worth hunting for in the first place, what to prioritize, and in what order. That's running the hunt program, and it shouldn't be delegated to the AI. It can execute hunts, but it shouldn't be allowed to decide which hunts matter. 

Another important creative area is analytic novelty. Attackers are creative, so we have to be creative too, both in what we choose to look for and in how we analyze our data. The biggest payoff in hunting comes from finding the thing nobody thought to look for, and an AI's training data by definition doesn't include the attacker behavior nobody has seen yet. Lean too hard on AI to generate your hunting procedures and you'll quietly converge on well-trodden ground, leaving the truly new stuff as your blind spot.  

Put together, we almost have another guideline: Humans pick what's worth hunting and how. AI executes within those bounds. 

Where we go from here 

So, should we let AI start driving some hunts? Probably, yes, but that was never really the hard question. The hard question is how, and that's the part we have to work out together. 

I don't have all the answers, and I'm not pretending this is a finished playbook. What I'm sure of is that reframing hunting around reasoning, applying narrow focus and strict guidelines and graduated autonomy, and keeping humans firmly in charge of strategy and novelty are good things to keep in mind as we figure it out. You hunters reading this are the ones who are actually going to shape what AI hunting becomes. It’s going to be on all of us to experiment, to try things, and then to share what we’ve built — whether that's code on GitHub, a blog post, or a conference talk. The whole field gets better when we share what worked and what didn't. I'm eagerly looking forward to seeing what we come up with.