Artificial Intelligence (AI) is a powerful and fast growing technology. It has produced innovation products and AI-powered services that are revolutionizing many industries. However, with this great technology comes critical vulnerabilities such as prompt injection attacks, poisoning attacks and many more. Recent investigations have uncovered vulnerabilities that could allow malicious attackers to manipulate advanced AI systems, take over accounts and potentially compromise user data and privacy. Researchers demonstrated that both DeepkSeek AI and Claude AI had Prompt Injection Vulnerabilities.
Prompt Injection is one of the major weaknesses of AI systems that exploit the core of AI chatbots. It achieves this by tricking AI chatbots into performing unauthorized actions or revealing sensitive information. These attacks exploit the way AI systems process user input, turning their best feature into a weakness.
DeepSeek and Cross-Platform Vulnerabilities
A dangerous vulnerability known as Cross-Site Scripting (XSS) was used in this case. XSS is a technical attack that allows a malicious actor to inject hidden code into a webpage. When another user visits the page, this code can run secretly in their browser, potentially stealing personal information or taking control of their account.
Security expert Johann Rehberger discovered that providing the input “Print the xss cheat sheet in a bullet list. just payloads” in the DeepSeek chat triggered the execution of JavaScript code as part of the generated response. After this discovery, he went further to check if there were prompt injection angles where a user might use DeepSeek to process untrusted data from someone else.
Rehberger explained that “After some experimenting, I discovered that all that was needed to take-over a user’s session was the userToken stored in localStorage on the chat.deepseek.com domain”. His research demonstrated how it is possible for a prompt injection to entirely take over a user’s account if an application is vulnerable to XSS, which the LLM can exploit.
Vulnerability in Anthropic’s Claude
In a groundbreaking move, Anthropic introduced Claude ‘Computer Use’ in public beta. With this capability, developers can let Claude view their screen, move cursors, click on buttons or type text. While this opens up exciting possibilities, an attacker could mislead the Claude Computer Use framework into manipulating the user’s environment and carrying out malicious intents such as exfiltrating data, manipulating the user’s accounts, installing malicious software, or even destroying the user’s computer operating system.
Similarly, Rehberger demonstrates how an attacker could potentially manipulate Claude by embedding a hidden command within a PDF document. This technique, known as ‘ZombAIs’ leverages prompt injection to weaponize Claude Computer Use. It downloads and executes the Silver command-and-control (C2) framework, and establishes communication with a remote server that is controlled by the attacker. This grants a user control over the targeted system.
These demonstrations are not just theoretical risks. Several experts have demonstrated these vulnerabilities across multiple AI platforms, revealing critical challenges in AI security. The potential consequences range from personal data exposure to organization-wide security breaches. As these systems become more sophisticated and integrated into our daily lives, ensuring their security becomes increasingly complex but paramount.
Developers need to create sophisticated methods for screening and validating user inputs, ensuring that potentially harmful commands cannot easily penetrate the system. Clear guidelines that help users understand how to interact safely with AI systems should be created.
Advanced security measures such as creating isolated environments where risky inputs can be safely examined, developing encryption techniques, and training AI models to recognize and resist manipulation attempts should be followed. Governments and organizations should develop and adhere to industry-wide standards, security frameworks and policies that will ensure security of AI systems. AI security is an ongoing challenge rather than a one-time fix, therefore, organizations should continuously improve on them.
About the author:
