WormGPT: Jailbroken LLM Sold on Dark Web for AI-Powered Phishing Generation
WormGPT — a jailbroken large language model with no ethical restrictions — was made available on dark web forums for €60–100 per month, enabling anyone to generate sophisticated phishing emails, malware code, and BEC scripts without technical skill.
Background
The release of ChatGPT in late 2022 demonstrated that large language models could produce human-quality text. Criminal entrepreneurs immediately recognised the potential to strip ethical guardrails and sell access to LLMs specifically tuned for malicious use. WormGPT was the first widely publicised example.
The Attack
WormGPT was based on an open-source LLM (GPT-J) fine-tuned on malware-related data and made available without safety restrictions. Advertised in a popular hacking forum, it allowed users to generate highly convincing phishing email text, BEC scripts tailored to specific companies, malware code in multiple languages, and social engineering scripts. The barrier to creating convincing phishing campaigns dropped from requiring writing skill and English fluency to simply describing the target. WormGPT could generate personalised phishing in multiple languages with appropriate cultural context.
Response
SlashNext security researchers discovered and published analysis of WormGPT in July 2023. The research triggered significant media coverage and government attention. The original WormGPT operator shut down the service under pressure but similar services (FraudGPT, DarkBard, PoisonGPT) appeared within weeks. OpenAI invested in content controls on official ChatGPT.
Outcome
WormGPT demonstrated the dual-use problem of LLMs: the same technology that generates helpful content can generate malicious content when restrictions are removed. Security researchers noted a measurable increase in phishing email quality and volume following the emergence of criminal LLMs.
Key Takeaways
- AI-generated phishing is grammatically perfect and culturally appropriate — spelling mistakes are no longer a reliable phishing indicator
- Security awareness training must update to reflect that AI-polished phishing is indistinguishable from legitimate emails
- Technical phishing controls (email authentication, URL scanning) are more important than ever as human detection degrades
- Open source LLMs with no guardrails will be fine-tuned for criminal use — this is a permanent feature of the threat landscape
How to Prevent This
All guidesConduct quarterly phishing simulations with immediate personalised coaching
Phishing simulations that simply report click rates produce marginal improvement. The most effective programmes deliver immediate, personalised coaching at the moment of failure: when an employee clicks a simulated phishing link, they see an explanation of what they missed and why it was deceptive. Studies show that this just-in-time training reduces repeat click rates by 50–70% compared to annual awareness videos. Simulate a range of attack types: credential harvesting pages, malicious attachments, BEC requests, and SMS phishing. Include AI-generated phishing emails that are grammatically perfect, since poor spelling is no longer a reliable indicator.
Treat grammatically perfect emails as potential AI-generated phishing
For decades, poor grammar and spelling were reliable indicators of phishing emails. WormGPT, FraudGPT, and general-purpose LLMs have eliminated this signal. AI-generated phishing emails are indistinguishable from legitimate emails by grammar alone. Train employees that the absence of spelling mistakes is no longer a safety signal. Shift phishing recognition training to focus on: unexpected requests (even in well-written emails), urgency combined with an unusual ask, mismatched sender domains (visible in email headers), and links that do not go to the expected domain. Technical controls — DMARC, email authentication, URL scanning — become more important as human detection degrades.