arXiv: Beyond the Prompt: Jailbreaking Function-Calling LLMs via Simulated Moderation Traces

This paper, published on arXiv, details a novel vulnerability in large language models (LLMs) that use function-calling capabilities. The research demonstrates that attackers can bypass safety guardrails by injecting malicious instructions through simulated moderation traces—essentially, by tricking the model into believing it is reviewing its own output for safety, when in fact it is being manipulated to execute harmful actions. This is not a regulatory change but a newly identified technical risk that could undermine existing AI safety frameworks.

Organizations deploying LLMs with function-calling features—particularly in regulated sectors like finance, healthcare, legal services, and customer support—are directly affected. Any firm using AI agents that can access external tools, databases, or APIs should consider this a high-priority threat. The vulnerability could enable unauthorized data access, financial transactions, or system commands, potentially violating GDPR, AI Act, or sector-specific compliance obligations.

Compliance teams should immediately review their AI model deployment pipelines to ensure that function-calling LLMs are not exposed to untrusted user inputs without additional validation layers. Implement runtime monitoring for anomalous function call patterns and consider adding a separate, non-LLM-based moderation step for all tool-use requests. Update your AI risk register to include this attack vector and coordinate with security teams to test your models against this specific jailbreak technique before the next regulatory audit.