This article explores the reliability of large language models (LLMs) in detecting security vulnerabilities in code. It highlights a study comparing Anthropic’s Claude Code and OpenAI’s Codex on their ability to identify SQLi, XSS, and IDOR vulnerabilities in real-world Python applications. The research reveals that while AI can spot some security flaws, it struggles with consistent and accurate detection, especially for data flow-based vulnerabilities. Jayson DeLancey, the author, discusses the limitations of AI in security scanning and emphasizes the need for hybrid approaches combining AI with traditional static analysis tools. This article is worth reading because it provides a critical evaluation of AI’s role in secure coding practices. Readers will learn how AI models like Claude Code and Codex perform in real-world security scenarios and the challenges they face in detecting vulnerabilities.
Key facts
- A study compared Anthropic’s Claude Code and OpenAI’s Codex for detecting SQLi, XSS, and IDOR vulnerabilities in real-world Python applications.
- Claude Code identified 46 real vulnerabilities, while Codex found 21, with both having high false positive rates.
- AI models performed well at detecting IDOR vulnerabilities but struggled with SQL injection and XSS due to challenges in taint tracking.
- AI tools exhibit non-determinism, producing different results each time they scan the same codebase, which affects reliability.
- The article suggests that AI should be used as an assistant rather than a replacement for deterministic static analysis tools in security workflows.
