AI adoption may be accelerating in Australian workplaces, but employee trust is struggling to keep pace. New research commissioned by Appian found that 60% of Australian workers have personally identified an error made by an AI tool at work, while only 14% fully trust AI-generated outputs used by their organizations. The findings point to a growing enterprise AI challenge: businesses are deploying AI into increasingly consequential workflows before employees are confident that its decisions can be questioned, reviewed or overridden.
For enterprise AI, the trust problem may be becoming as important as the technology itself.
Workers are increasingly encountering AI-generated mistakes in everyday business processes, yet many say they are not always encouraged to challenge the systems producing those outputs. In Australia, that disconnect is creating a difficult environment for organizations attempting to move generative AI and AI agents from experimentation into customer-facing and operational workflows.
Research commissioned by Appian and conducted by Zoho Research among 500 Australian workers in the third quarter of 2026 found that 60% had personally identified an error made by an AI tool at work.
Only 14% said they fully trust AI-generated outputs used within their organization.
More than one in five respondents also said employees are discouraged from challenging AI recommendations or decisions.
That combination creates a governance problem. If workers identify an AI mistake but do not feel empowered to question it, the organization can lose one of its most accessible safeguards against an incorrect automated decision.
AI errors are reaching customers and operations
The consequences of AI mistakes are not necessarily limited to minor productivity issues.
One in five Australian workers surveyed said the most significant AI error they had encountered caused a customer service issue. Another 13% reported operational disruption.
Almost half said their most significant AI error had consequences beyond minor inconvenience, including financial, compliance or reputational effects.
The findings highlight why human oversight remains important as AI systems move deeper into enterprise processes.
The greatest concern among respondents was the possibility of AI making incorrect decisions, cited by 25%. Data privacy and security risks followed at 22%, while 20% identified a lack of transparency around how AI reaches decisions as their primary concern.
These concerns extend beyond conventional generative AI chatbots. As enterprises introduce AI agents capable of interpreting data, recommending actions and executing multi-step workflows, an incorrect output can potentially move further through an organization before a human notices it.
Workers want limits on autonomous AI
Australian employees appear to be drawing a clear boundary around how much authority AI should receive.
Ninety-four percent of respondents said AI should not be allowed to make decisions affecting employees or customers without limits or human oversight.
Two-thirds went further, saying decisions affecting workers or customers should either require human review and approval or always be made by people.
Another 28% said AI should be restricted to low-risk decisions.
Trust in human leadership also remains stronger. Fifty-two percent of respondents said they place greater trust in senior management than AI, compared with 29% who said they trust AI more.
The numbers suggest that employees are not necessarily rejecting workplace AI. Instead, they appear to be asking for clearer boundaries around where AI can operate independently and where human judgment must remain part of the process.
Governance is lagging behind AI deployment
The Australian findings align with a broader enterprise governance challenge identified in research from Harvard Business Review Analytic Services, sponsored by Appian.
That global research found that 92% of organizations agree AI agents require rules-based guardrails to operate safely and effectively. Yet fewer than half, or 48%, said their organizations have actually defined those guardrails.
The gap matters as AI moves from standalone productivity tools into business processes.
A generative AI assistant summarizing documents presents a different risk profile from an AI agent making decisions about customers, employees, payments or compliance. The latter requires organizations to define permissions, escalation rules, monitoring and intervention mechanisms before the system operates at scale.
That is increasingly becoming an architectural question rather than simply an AI policy question.
AI needs to operate inside governed workflows
The research suggests that organizations cannot rely on employee vigilance as their primary AI control.
Workers can identify errors, but they need mechanisms to flag questionable outputs, escalate decisions and override automated recommendations. Without those processes, organizations risk creating a choice between two undesirable outcomes: employees spend significant time checking every AI output, or they accept questionable results because challenging the system is difficult.
This is where enterprise AI platforms and workflow automation systems are increasingly converging.
Rather than placing an AI model outside existing business processes, organizations can embed AI into workflows with defined permissions, approval steps, audit trails and human intervention points.
That model also changes how businesses should measure AI productivity.
An AI system that generates faster outputs but requires employees to manually validate every result may deliver less value than expected. Conversely, an AI agent operating inside a controlled workflow can potentially automate more work while preserving accountability.
The next phase of AI adoption will test governance
The Australian research arrives as enterprises move toward more autonomous AI systems.
Large language models are becoming components of broader enterprise applications, while AI agents can connect to business systems and execute tasks. This creates opportunities for automation across customer service, operations, HR, finance and other functions—but also increases the consequences of poor governance.
The challenge is therefore not simply whether AI can perform a task.
Organizations need to determine when AI can act, what information it can access, what rules it must follow, when a human must intervene and how the resulting decision can be audited.
Australian workers appear to be sending a clear signal on that question. They are willing to work alongside AI, but not without accountability.
For enterprise technology leaders, that makes human oversight less of a temporary safety mechanism and more of a core component of AI infrastructure.
Market Landscape
Enterprise AI is moving toward workflow-level automation, increasing the importance of governance, observability and human-in-the-loop controls. The Appian research highlights a familiar challenge: employee trust can fall when AI systems make visible mistakes without clear mechanisms for review or intervention.
The global HBR Analytic Services research cited by Appian further suggests that organizations recognize the need for rules-based AI guardrails but have not consistently implemented them.
This creates an opportunity for AI automation platforms, AI agents, enterprise AI applications and AI governance technologies that can embed model outputs inside controlled workflows rather than treating AI as an isolated assistant.
The competitive landscape includes Microsoft, Google, Amazon, Salesforce and specialized enterprise AI providers. Increasingly, differentiation will depend not only on model capabilities but also on how safely those models can operate inside business processes.
Top Insights
- AI mistakes are already affecting Australian workplaces, with 60% of surveyed workers reporting that they have personally identified an AI error at work.
- Employee trust remains low, as only 14% of respondents fully trust AI-generated outputs used within their organizations.
- Workers overwhelmingly want human oversight, with 94% rejecting unrestricted AI decision-making affecting employees or customers.
- Enterprise AI guardrails remain underdeveloped, despite 92% of global organizations agreeing that AI agents require rules-based controls.
- Workflow governance is becoming essential, as AI moves from productivity assistance into customer-facing and operational decision-making.
Power Tomorrow’s Intelligence — Build It with TechEdgeAI
