Companies don’t appear to need convincing that AI carries risk. They appear willing to accept quite a lot of it anyway.
TrendAI’s global study of 3,700 business and IT decision-makers found that 67% had felt pressure to approve AI despite security concerns. One in seven described those concerns as extreme but said they had still been overridden to keep pace with competitors or internal demand. Only 38% of organisations surveyed had comprehensive AI policies in place, while 31% admitted they lacked sufficient observability or auditability over their AI systems.
Rachel Jin, Chief Platform and Business Officer and head of TrendAI, puts the gap down to something more basic than ignorance. Organisations, she argues, know there are risks but don’t necessarily have the conditions to manage them. TrendAI’s research found that 44% see AI agents accessing sensitive information as their biggest agentic AI risk, while concerns also extend to malicious prompts, expanding attack surfaces and autonomous code deployment. That has far-reaching implications as AI moves beyond answering questions and starts doing things.
An agent connected to company systems can be allowed to access information, call software tools or trigger actions without asking a person for approval at every step. The permissions differ enormously depending on how the system has been configured, which is precisely where TrendAI believes traditional security approaches begin to struggle.
TrendAI itself is a relatively new name. Trend Micro renamed its enterprise cybersecurity business TrendAI in March as it expanded its focus from securing infrastructure and applications towards governing how AI systems act and interact inside organisations. The change coincided with the launch of its Agentic Governance Gateway, a product designed to sit between autonomous agents and the systems with which they interact.
The marketing around any security product deserves scrutiny, particularly when its maker has an obvious commercial incentive to persuade companies that a new category of risk requires a new layer of software. There’s one phrase in TrendAI’s description of the Gateway gets to a much larger problem: organisations should be able to introduce human oversight at “critical decision points”. But who decides what makes a decision “critical” before the damage is already done? That question takes on far-reaching implications as AI shifts from merely answering questions to autonomously executing tasks.
Who decides what the AI is allowed to do alone?
Putting a human in the loop sounds straightforward until an organisation has to decide what the human is actually there to approve.
A customer-service agent reading an account balance may be considered low risk. Changing a customer’s contact details probably deserves more scrutiny. Moving money creates another level of exposure entirely. The same problem appears in cybersecurity, software development, HR systems and any environment where an agent can progress from retrieving information to altering something in the world.
TrendAI’s Agentic Governance Gateway reflects that distinction. It is designed to discover what agents can access, observe how they interact with other systems, enforce policies around their actions and stop behaviour that falls outside those rules. Human approval can then be introduced where an organisation decides the consequences justify it.
That model makes sense operationally because asking a person to approve everything would make autonomy considerably less useful. It also moves one of the most important decisions away from the moment an agent acts. Somebody has to decide which actions don’t require approval.
TrendAI’s research suggests organisations are making those calls while their broader governance remains incomplete. Around 40% of respondents supported an AI “kill switch” capable of shutting systems down after a failure or misuse, while nearly half remained uncertain about whether such a mechanism was necessary. TrendAI reads the disagreement as evidence that businesses are moving towards autonomous systems without having agreed on how they will retain control.
There is a useful tension in that finding. A kill switch deals with what happens once somebody realises a system should stop. Governance also has to deal with everything the agent was authorised to do before anybody reached for it.
Having a human there doesn’t guarantee much
The technology industry as a whole is arriving at a similar problem.
Cisco now distinguishes between human-in-the-loop and human-on-the-loop systems. The former asks a person to participate in individual decisions, while the latter allows an agent to operate independently under higher-level human supervision. AWS, meanwhile, explicitly warns that requiring human review for every agent action can produce reviewer fatigue and rubber-stamp approvals.
Cisco and AWS are useful examples, but they don’t resolve the underlying problem. An organisation can put a person into an approval workflow and still create a system where the machine effectively makes the decision.
That risk has already shown up in practice. Anthropic found that Claude Code users approved roughly 93% of permission requests when the coding agent asked before certain actions. The company found that repeated approval requests reduced people’s attention to them and responded by automating some lower-risk permissions while putting more emphasis on limiting what the agent could access in the first place.
The implication is uncomfortable. A person clicking Approve is evidence that a human participated in a workflow. It says considerably less about whether they exercised meaningful judgement.
Human-in-the-loop AI oversight was already beginning to show these limits before the current enterprise push towards more autonomous agents. As the number of automated decisions grows, meaningful supervision increasingly depends on designing the boundaries well rather than inserting a person into every transaction.
That places TrendAI’s “critical decision points” in a different light. The important control may be the policy determining when the agent has to stop, what information the reviewer sees when it does, and what happens when nobody responds.
South African law has already met the electronic agent
For South African organisations, this isn’t a theoretical debate waiting on future AI regulation; the legal foundation has been quietly sitting on the books for over two decades. Since 2002, the Electronic Communications and Transactions Act (ECTA) has explicitly recognised the concept of an “electronic agent,” defining it as an automated system capable of independently initiating actions or responding during a transaction without human intervention.
The technology lawmakers had in mind was considerably simpler than a generative agent capable of interpreting an objective and choosing how to accomplish it, but the Act still gives automated activity legal consequences. Under section 20, an agreement can be formed through an electronic agent and, subject to specified exceptions, a party using one can be presumed bound even if that person didn’t personally review the agent’s actions or the resulting terms. Section 25 also attributes certain automatically generated data messages to the organisation behind the system unless it can be shown that the information system failed to execute its programming properly.
Those provisions don’t settle liability for modern agentic AI. Attribution of a data message doesn’t automatically decide who is civilly, criminally or regulatorily liable for everything an agent might do, and a law drafted around automated transactions shouldn’t be stretched into an AI liability statute that Parliament never wrote.
It does make one assumption difficult to sustain: the absence of a human clicking the final button doesn’t automatically put the resulting action beyond the organisation that chose to automate it.
POPIA gives the human-oversight question a more contemporary form. Section 71 restricts certain decisions based solely on automated processing of personal information where the decision has legal consequences for a person or affects them substantially and the processing is intended to create a profile. The law specifically contemplates areas such as work performance, creditworthiness, reliability, health and conduct.
The section contains exceptions, so it doesn’t create a universal requirement for a human to approve every consequential automated decision. Where an applicable exception relies on measures protecting the person’s legitimate interests, however, POPIA can require an opportunity to make representations and enough information about the underlying logic for those representations to be meaningful.
There’s already a direct connection here with the unresolved questions around South Africa’s broader AI policy. South African companies are making decisions about AI deployment while the country’s dedicated policy framework remains unsettled, leaving existing legislation and sector rules to do much of the immediate work.
TrendAI’s governance argument therefore lands differently in South Africa. Companies don’t have the luxury of treating agent controls as something to design only once dedicated AI legislation appears. Depending on what the agent touches and what decision it makes, existing privacy, electronic-transactions and sector-specific rules may already apply.
Financial services makes the accountability question harder
TrendAI’s original concern becomes particularly relevant in banking, insurance and other regulated financial services because the consequences of automated access and action can escalate quickly.
The FSCA and Prudential Authority’s Joint Standard 2 of 2024 on cybersecurity and cyber resilience establishes governance requirements for covered financial institutions. Its focus is cybersecurity rather than AI specifically, so it shouldn’t be treated as a catch-all agentic AI law. But where autonomous systems interact with security controls, sensitive information or critical infrastructure, the governance obligations become relevant. The bigger issue is how responsibility is allocated before an autonomous system enters production.
An organisation can decide that an agent may read customer information but may not change it. It can permit routine account actions while escalating unusual cases. Security teams can restrict which tools an agent is allowed to invoke and what data it can see. TrendAI’s own approach is built around making those permissions visible and enforceable before and while agents operate.
That governance work is less dramatic than the idea of an AI suddenly going rogue, but it is probably where most real accountability questions will begin.
The important human may never see the agent act
TrendAI’s research describes organisations deploying AI despite concerns they already understand, often before the policies governing those systems are complete. Agentic AI adds another decision to that process because every useful degree of autonomy represents something a human has deliberately stopped checking case by case.
That doesn’t automatically make autonomy reckless. A system that requires a person to approve every routine action can become slow enough to defeat the purpose of deploying it, while overloaded reviewers may approve requests largely out of habit. The challenge is deciding where autonomy stops being an efficiency decision and becomes a risk the organisation is no longer prepared to delegate.
TrendAI’s idea of human oversight at critical decision points therefore raises a question that software alone can’t answer. Which decisions are critical, who gets to classify them that way, and how often should those boundaries be reconsidered as the agent gains new capabilities?
Those choices are made by people before an agent moves money, changes a permission or acts on someone’s information. The employee who eventually receives an approval request may be only one part of the accountability chain.
As companies hand more decisions to autonomous software, the human who matters most may increasingly be the one who decided the AI no longer needed to ask.