There is an old Latin phrase: Quis custodiet ipsos custodes? Simply put: Who watches the watchmen?
It was a question of power and oversight. If those entrusted with guarding society become corrupt, who ensures they are accountable? In today’s world, that same question applies not to presidents and law enforcement but to algorithms, automation, and artificial intelligence, especially in the case of agentic AI.
The Rise of the Automated Watchers
Modern systems are too vast and complex for humans to monitor alone. These complexities range from
- Microservices sprawl across Kubernetes clusters, spawning thousands of interactions per second.
- Observability tools like Datadog, New Relic, and OpenTelemetry stream terabytes of logs, traces, and metrics to surface anomalies.
- AI guardrails in platforms like LangChain, GuardrailsAI, and Azure’s Responsible AI toolkits catch unsafe or biased model outputs before they get to customers.
These systems watch everything: performance, security, compliance, and fairness. They are our first line of defense against outages, breaches, and reputational risk.
This idea came to me when I was writing a program for my robot using ROS2: What happens when the watcher itself fails, drifts, or is compromised?
The Accountability Gap
We assume watchers are infallible, but history says otherwise:
- A metrics pipeline silently dropped alerts during a network partition, and no one noticed until the customer SLA was breached.
- An intrusion detection system was itself bypassed in a supply chain attack, leaving a false sense of security
- An AI safety layer failed to catch adversarial prompts, exposing users to harmful outputs or expose a company’s sensitive data
In each case, the system built to guarantee trust became the single point of failure. The absence of alerts was misread as the absence of problems.
This is the accountability gap:Who verifies the automated verifier?
Lessons from Toyota: Jidoka and the Andon Cord
Early in my career, I had the privilege of working with Toyota as a customer, and my counterpart shared a history lesson with me. The auto industry wrestled with this decades ago. Toyota, the pioneer of lean manufacturing, introduced robots to improve efficiency. But they quickly discovered a hard truth: robots can make the same mistake perfectly, at scale.
Every incorrect weld resulted from a robotic arm’s miscalibration. If a sensor failed, the defect affected thousands of cars. Automation didn’t correct errors; if it didn’t, it made them worse.
Toyota’s solution was jidoka: “automation with a human touch.” Rather than relying solely on machines, they included human oversight in the process:
- The Andon Cord: Any worker could pull a literal cord to stop the entire assembly line if a defect was spotted.
- Layered Verification: Human inspectors and visual systems checked robotic output continuously.
- Kaizen (Continuous Improvement): Every failure was treated as a learning loop, improving both robots and oversight systems.
The lesson is timeless: automation increases both efficiency and risk. A single defect in a manual process is localized; a defect in an automated process is systemic.
The software world is no different. Observability dashboards are our Andon cords. SREs are our jidoka. And post-incident reviews are our kaizen.
Strategies for Watching the Watcher
Just as Toyota built layered accountability into its manufacturing system, we need to design resilience into our agentic AI systems. Four key strategies stand out:
- Meta-Monitoring for Microservices
- Observability tools should watch each other, not just the services.
- Example: Prometheus scrapes are validated by synthetic transactions running through the service mesh, the digital equivalent of a second inspector checking a robot’s welds.
- Audits for Observability
- Periodic “reality checks” involve comparing raw logs and traces against dashboards.
- Independent tools like Honeycomb validating a Datadog pipeline are today’s equivalent of a Toyota team double-checking machine outputs.
- Guardrails for Guardrails in AI
- Safety layers need redundancy: pre-training filters, real-time classifiers, and post-response moderation.
- Think of this as multiple Andon cords for LLMs such as OpenAI’s Evals, Anthropic’s Constitutional AI, and Microsoft’s Responsible AI dashboards, which can all act as independent cords waiting to be pulled.
- Human-in-the-Loop Escalation (Digital Jidoka)
- Automation can reduce noise, but critical thresholds must escalate to humans.
- Just as Toyota trusted line workers to stop the factory floor, we need to empower SREs, red teams, and ethics boards as the final circuit breaker.
Why It Matters
My experience with Toyota taught me, and Toyota taught the world, that automation doesn’t eliminate human judgment; it amplifies the need for it. The philosophy of jidoka, the practice of pulling the Andon cord, and the discipline of kaizen created not just efficient factories, but resilient ones.
Agentic AI needs the same mindset:
- Jidoka: Design automation with human judgment built in.
- Andon Cord: Give humans the power to halt systems when trust is in doubt.
- Kaizen: Treat every monitoring failure as a learning loop, not a one-time solution.
Juvenal’s warning still holds: unchecked power, whether in presidents, robots, or algorithms, breeds complacency.
👉 The real question for software leaders is this: will we embed jidoka for Agentic AI systems, or will we continue to trust the watchers blindly until they fail at scale?
The future of resilient software, trustworthy AI, and reliable observability depends on whether we pull the cord in time.
