AI and information security - how we can counter new risks

AI makes many things easier - unfortunately also for cyber criminals. Information security has long been one of the key success factors of modern IT. However, with the spread of artificial intelligence (AI), the threat situation is changing considerably. Terms such as Shift Left Security or Security by Design make it clear that security aspects must be integrated into development and operating processes at an early stage in order to be implemented effectively and economically.

However, the advent of AI is expanding the playing field - for developers and attackers alike. While AI-supported tools are increasing productivity, the hurdles for hackers are also decreasing. New attack methods are emerging, while many users outside of traditional software development have little awareness of security.

The decisive question is therefore no longer: "AI or non-AI?", but rather: How do we face up to the safety-related consequences of this development?

New types of attack through AI

The combination of classic IT security threats and new AI-driven attacks creates a complex risk scenario:

Prompt Injection & Manipulation

AI models can be misdirected by manipulated inputs. Attackers use this to access confidential information or trigger unwanted actions.

Example: Invisible prompts are hidden in a website or document. An AI that uses this source interprets the hidden instructions as legitimate input. Mechanisms such as the Model Context Protocol (MCP) can be used to trigger unwanted actions.

Insecure output handling

Insecure processing of AI output, e.g. if generated code is adopted directly or executed as an instruction without validation.

Example: AI tools generate code that contains actively exploitable vulnerabilities. If such outputs are adopted without being checked, there is a massive risk.

(Training) Data Poisoning

If training data is manipulated, an AI system can be systematically weakened or compromised.

Supply Chain Vulnerabilities

Risks from external models, tools or libraries.

Example: AI systems generate code and automatically suggest libraries. Without blacklists/whitelists or dependency checks, there is a high risk that unsafe or manipulated dependencies will be adopted unnoticed.

Sensitive Information Disclosure

Confidential data can be exposed through careless prompts, spending or training data.

Insecure Plugin Design

Insecure design or integration of plugins/tools that give the AI additional functions, such as external API access.

Excessive Agency

If the AI is given too much autonomy and performs actions without human control - e.g. automated tool or API calls.

Overreliance

Over-reliance on AI, without critical scrutiny or human review.

Example: AI-supported phishing campaigns are not recognized because users blindly trust the generated content.

Model Theft

Theft or unauthorized use of the model itself - for example through exfiltration of weights or configuration files.

Deepfake-based identity attacks

Not directly part of the OWASP LLM Top 10, but relevant in practice: Deceptively real audio or video manipulations that enable identity fraud and undermine trust.

How to protect yourself from AI risks

Security in classic software development

In development organizations with established processes, security measures can be integrated into the software life cycle in a structured manner:

Shift Left Security & Security by Design

Embed security checks in architecture, coding and testing right from the start, not just at the end.

Using established frameworks

• OWASP LLM Top 10: Address risks such as prompt injection, data poisoning or excessive authorizations.
• Google's Secure AI Framework (SAIF) and NIST SSDF: Bridging the gap between traditional application security and AI-specific threats.

Technical protective measures

• Validate inputs to models and process them in a context-separated manner.
• Check AI outputs against fixed schemas (e.g. JSON schema).
• Operate AI agents and tools in isolated environments (sandboxing, least privilege).
• Secure library management: Black/whitelists for dependencies, automated dependency scans and signature verification to prevent the takeover of malicious libraries.

Red Teaming & Tests

Regularly test the robustness of AI applications with simulated attacks (e.g. MITRE ATLAS).

• SpecificTests for hidden prompts in documents or websites, analogous to a virus scanner, in order to detect manipulations in sources at an early stage.

Security in specialist departments & with citizen developers

However, AI applications are increasingly being created outside of traditional development, e.g. in HR, finance or marketing. The deep development standards do not apply here. Security must become simpler, more standardized and more process-based:

Provide secure platforms

Companies should offer centralized, hardened AI workspaces (e.g. internal RAG environments or audited GenAI services) instead of allowing uncontrolled tool usage.

Force data classification

Clarify which data is allowed in the model (public, internal, confidential). Tools should support this technically, for example through upload locks or automatic editing.

Prefabricated secure templates

Departments use tested prompt templates and standard workflows that have been approved by security instead of experimenting "freely".

Governance & responsibility

• Clearly define usage policies for citizen developers.
• Control access rights based on roles.
• Critical applications only with approval from IT/security.

Awareness at a minimum level

Training on basic rules: "Do not upload confidential data to public models", "Critically scrutinize AI output", "Report suspicious results".

GoalUnderstanding risks without requiring developer know-how.