AI Agents’ Mathematics: A Complex Equation That Doesn’t Balance

In recent years, the promise of fully automated lives through AI agents has remained largely unfulfilled. Big tech firms originally touted 2025 as the timeline for a transformative AI revolution. However, expectations have now shifted to 2026 or even beyond. The central question remains: Will generative AI robots eventually perform our daily tasks and manage the world around us?

AI Agents: A Complex Equation

A paper titled “Hallucination Stations: On Some Basic Limitations of Transformer-Based Language Models” challenges the feasibility of true AI agency. This study argues that large language models (LLMs) cannot perform tasks beyond a specific level of complexity. The author team includes Vishal Sikka, a former CTO at SAP and ex-CEO of Infosys, and his son, a teenage prodigy. They claim that even advanced reasoning models won’t solve these foundational issues.

Inherent Limitations of AI Agents

• Current AI models struggle with reliability.
• Mathematics offers no pathway to develop trustworthy AI capable of managing critical infrastructures.
• AI’s utility may remain limited to simple tasks, such as document preparation.

During a discussion, Sikka emphasized the impracticality of relying on AI agents for complex operations, such as managing nuclear power plants. While AI may assist in minor tasks, it is crucial to accept that mistakes may occur.

Progress in AI Coding

Contrary to the aforementioned challenges, the AI industry highlights several achievements. For instance, coding capabilities saw significant advances in the past year. At the recent Davos meeting, Demis Hassabis, Google’s AI lead, shared breakthroughs in reducing AI hallucinations.

A notable startup, Harmonic, co-founded by Vlad Tenev, CEO of Robinhood, and Stanford-trained mathematician Tudor Achim, reports significant progress in AI code reliability. Their product, Aristotle, utilizes formal methods of mathematical reasoning to enhance AI system trustworthiness.

Mathematical Verification in AI

Harmonic focuses on ensuring the accuracy of LLM outputs through mathematical verification. This approach employs the Lean programming language, known for its robust verification capabilities. However, Harmonic has concentrated primarily on coding, leaving areas like essay writing, which lacks quantifiable verification, outside its current range.

A Shared Perspective on AI Reliability

As the conversation around AI reliability continues, both sides recognize that hallucinations will still pose challenges. A study by OpenAI scientists echoed this sentiment, revealing that even advanced models such as ChatGPT can generate inaccurate information. They found that when queried about the lead author’s dissertation, all three AI models fabricated titles and misrepresented publication years.

OpenAI acknowledged that achieving 100 percent accuracy in AI models is unattainable, emphasizing ongoing limitations in generative AI’s capabilities.

The Path Ahead for AI Agents

The future of AI agents remains uncertain. As researchers work to address the complexities, the integration of formal mathematical reasoning could pave the way for more reliable systems. However, for now, the dream of fully autonomous, error-free AI agents is still just that—a dream.