{"id":10166,"date":"2026-05-04T09:56:59","date_gmt":"2026-05-04T16:56:59","guid":{"rendered":"https:\/\/calleam.com\/WTPF\/?p=10166"},"modified":"2026-05-04T10:31:10","modified_gmt":"2026-05-04T17:31:10","slug":"ai-agents","status":"publish","type":"post","link":"https:\/\/calleam.com\/WTPF\/?p=10166","title":{"rendered":"AI Agents – The Jigsaw Piece That Broke the Picture"},"content":{"rendered":"\n

Following entry is a record in the \u201cCatalogue of Catastrophe<\/a>\u201d \u2013 a list of failed and troubled projects from around the world.<\/p>\n\n\n\n

\n\n\n\n

Organization:<\/strong> PocketOS
Project type:<\/strong> AI Agent implementation
Date:<\/strong> Apr 2026
Cost:<\/strong> Not disclosed<\/p>\n\n\n\n

Synopsis:<\/strong><\/p>\n\n\n\n

The explosion in interest in Artificial Intelligence (AI) agents means that enterprises around the world are experimenting with the possibilities. Projects abound and while there is no central record of how those efforts are going, the use of any nascent technology brings with it risks. Often leveraging Large Language Models (LLM), AI Agents help organizations automate tasks that would otherwise require human intervention. Despite the risks, AI tools that leverage LLMs are everywhere and the hype has become a stratospheric frenzy of high hopes.<\/p>\n\n\n\n

Despite the promise, reality sometimes bites and the past week has surfaced one such story. PocketOS is an organization that provides software to enable car rental companies. As you would imagine, databases are the foundation of such a business and hence data is mission critical stuff. Unfortunately an AI agent the organization had created to automate staging and testing activities went awry. Reports indicate that in 9 seconds flat the agent went beyond the system staging area and permanently deleted not just the production database, but also the recent backups that were to be used for disaster recovery. There was no checking with the \u2018human in the loop\u2019 and safety measures built into the agent\u2019s script proved ineffectual. Aligning the holes in Reason’s Swiss Cheese Model of System Accidents<\/a><\/strong> the agent\u2019s unanticipated behaviour led to unimagined consequences.<\/p>\n\n\n\n

To their credit the PocketOS\u2019s CEO shared the story and the inner mechanics of how it happened (see references below). The full analysis makes for an interesting read, but in short the AI Agent had more access to systems, permissions, data and API\u2019s (Application Programming Interfaces) than was fully understood. Technically, perhaps the most significant failure was a “Principle of Least Privilege” failure (i.e. the agent had administrative rights to a production environment it didn’t need given that it\u2019s intended focus was on staging and test). That level of permission and the LLM\u2019s interpretation of the available data resulted in irreversible commands being issued that resulted in the loss. As the CEO notes every organization deploying agentic technologies should read the post and understand the full cascade of events that did the damage.<\/p>\n\n\n\n

Demonstrating how interconnected systems can be, the story accentuates that an AI agent is often a jigsaw piece within a much larger and more complex IT architecture. Letting an AI agent free in that environment can be akin to hiring a junior intern and giving them access to your most complex systems. Yes you can give them instructions on what to do, but will they ALWAYS<\/strong> interpret those instructions in the way YOU<\/strong> imagined? Will they trip over unexpected situations and then misinterpret what to do next? Such risks are of course only part of the story. Other risks include the reality that many IT systems have undocumented (or even forgotten) \u2018technical debt\u2019 lying latent in the system\u2019s dark recesses. That technical debt can leave unknown vulnerabilities that the agent might end up interacting with. It is also then worth recognizing that there are inherent limitations in current AI technologies. As we all know that when being used for text generation LLMs will sometimes hallucinate. While having your text generator hallucinate is annoying, an AI agent that hallucinates is hallucinating at the executable level! Put the risks together and unanticipated problems seem inevitable. Likely we will be seeing many more stories of organizations who were at first smitten, but ended up getting bitten!<\/p>\n\n\n\n

<\/p>\n\n\n\n

Contributing factors as reported in the press:<\/strong><\/p>\n\n\n\n

Unrecognized technical debt in system resulted in unknown vulnerabilities. IT Security \/ permissions failure. <\/p>\n\n\n\n

Reference links:<\/strong><\/p>\n\n\n\n

Claude AI agent\u2019s confession after deleting a firm\u2019s entire database: \u2018I violated every principle I was given\u2019<\/a><\/p>\n\n\n\n

An AI Agent Just Destroyed Our Production Data<\/a><\/p>\n\n\n\n

\u2018It took nine seconds\u2019: Claude AI agent deletes company\u2019s entire database<\/a><\/p>\n","protected":false},"excerpt":{"rendered":"

Following entry is a record in the \u201cCatalogue of Catastrophe\u201d \u2013 a list of failed and troubled projects from around the world. Organization: PocketOSProject type: AI Agent implementationDate: Apr 2026Cost: Not disclosed Synopsis: The explosion in interest in Artificial Intelligence (AI) agents means that enterprises around the world are experimenting with the possibilities. Projects abound and while there is no […]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[27,28,5,3,42,77],"tags":[138,97,132,129,62,93,147],"class_list":["post-10166","post","type-post","status-publish","format-standard","hentry","category-blog","category-causes-of-failure","category-private-sector","category-failed-project","category-project-management","category-why-projects-fail","tag-causes-of-failure","tag-examples-of-failed-it-project","tag-private-sector","tag-failed-project","tag-quality-management","tag-why-do-projects-fail","tag-why-projects-fail"],"_links":{"self":[{"href":"https:\/\/calleam.com\/WTPF\/index.php?rest_route=\/wp\/v2\/posts\/10166","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/calleam.com\/WTPF\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/calleam.com\/WTPF\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/calleam.com\/WTPF\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/calleam.com\/WTPF\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=10166"}],"version-history":[{"count":8,"href":"https:\/\/calleam.com\/WTPF\/index.php?rest_route=\/wp\/v2\/posts\/10166\/revisions"}],"predecessor-version":[{"id":10178,"href":"https:\/\/calleam.com\/WTPF\/index.php?rest_route=\/wp\/v2\/posts\/10166\/revisions\/10178"}],"wp:attachment":[{"href":"https:\/\/calleam.com\/WTPF\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=10166"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/calleam.com\/WTPF\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=10166"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/calleam.com\/WTPF\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=10166"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}