Detecting fraud in the age of AI: Why benchmarking and testing are key

We’re in the midst of an AI-fraud arms race. On the one hand, fraudsters are leveraging AI to their own advantage – whether that’s through deepfakes, synthetic identities, or the creation of phishing messages. The numbers speak for themselves: Deepfakes are now linked to 1 in every 5 biometric fraud attempts, and there has been a 4,151% surge in malicious phishing messages since the launch of ChatGPT in November 2022.

Key takeaways:

• AI is enhancing current fraud trends, including deepfakes, synthetic identity generation, and fake job applicants
• AI is transforming fraud detection, and benchmarking and standardization are playing an increasing role in testing products
• Entrust’s solution achieved 0.0% FAR when tested against the IDNet dataset (the largest publicly available dataset for benchmarking fraud detection solutions)
• Follow best practices for businesses when assessing vendors and combating fraud using AI

Current Fraud Trends Enhanced by AI

Fraudsters are using AI tools to scale, automate, and advance their attacks. As these tools have become more accessible online, this is increasing the surface area of fraud, offering more fraudsters access to more advanced techniques.

AI-driven fraud trends include:

1. Deepfake impersonation scams
   • New account fraud: Fraudsters use deepfakes (often submitted via an injection attack) to bypass biometric verification during onboarding.
   • CEO/Executive impersonation: Attackers use deepfakes to simulate executive employers during real-time calls. One finance worker at a multinational firm was tricked into paying out $25 million after a fraudster used deepfake technology to pose as the company’s chief financial officer in a video conference call.
   • Romance scams: Fraudsters convince victims into handing over money by posing as a romantic interest – one romance scammer used AI-generated images of Brad Pitt to con a French woman out of $850,000.
2. Fraudulent documents and synthetic identity creation
   • Fraudsters can use GenAI to help them create fake IDs, either entirely from scratch, or by editing images of existing documents.
   • Synthetic identity fraud is projected to generate at least US$23 billion in losses in the U.S. alone by 2030.
3. Fake job applications
   • Many job applicants are using AI in their application process, which can include falsifying information or even deepfaking themselves for interviews, making it harder for employers to evaluate candidates’ true abilities and identities. Gartner predicts that by 2028, 1 in 4 candidate profiles will be fake.
   • North Korea IT agents are also using real-time deepfakes and other AI tools to pose as legitimate remote workers, apply for jobs, and infiltrate companies, putting businesses at risk of espionage and theft.
4. AI-enhanced phishing scams
   • Tools like ChatGPT (or copycats like FraudGPT or WormGPT) can help attackers draft tailored phishing emails or develop malware with minimal coding skills.
5. Bolstering organized crime
   • Organized crime groups and fraud rings can outsource AI tools to execute high-value fraud, or profit from sharing what they know via fraud-as-a-service methods on the dark web.
   • Bodies like FinCEN, FTC, FS-ISAC, Europol, and INTERPOL have all released alerts on deepfake-enabled fraud, romance scams, and investment schemes.

Identity verification continues to evolve as threats like social engineering and device emulation accelerate. This shift is transforming IDV and organizations worldwide are responding with smarter, scalable solutions. Discover key fraud trends and insights to help you protect identity and maintain trust in a digital-first world.

How AI Is Transforming Fraud Detection

Fraudsters might be using AI to their advantage, but defense systems also leverage AI in their own ways to detect fraud. This includes using AI to:

• Analyze government-issued IDs (like passports or driver’s licenses) to verify its authenticity
• Compare facial features from a selfie or video to the photo on the ID to check the ID belongs to the person presenting the document
• Perform liveness detection checks across biometrics (such as a selfie or video) for signs of spoofing (e.g. masks or deepfakes)
• Using AI to generated fraud samples to help train machine learning algorithms

The benefits of leveraging AI for defense is that AI systems improve over time through a continuous learning loop. This approach, which combines testing, training, and updating models to detect new fraud vectors, is essential to keep pace with the evolution of fraud tactics.

Challenges in Combating AI-Driven Fraud

One of the main challenges in the industry when it comes to combating AI-driven fraud is data. This includes both 1) having access to enough data to accurately train fraud preventing models, and 2) having standardized datasets that the industry uses for benchmarking.

Benchmarking is vital in the IDV industry because it provides an objective comparison across solutions. While document-based KYC checks are robust in theory, actual implementations vary significantly. And when vendors routinely use words like "AI-powered detection" and "AI-learning loop,” it’s hard to unpack the buzzwords from what this actually means for product performance.

Without standardized testing, performance claims often rely on limited datasets that don't reflect real-world fraud. This is why we often see performance metrics presented by vendors that function more as marketing plays rather than realistic reflections of real-world applications.

Moving Toward Standardization: The IDNet Dataset

Today, there is no standardized and objective dataset that the industry uses to benchmark. However, in 2024 a group of researchers at the U.S. Department of Homeland Security published IDNet – a dataset comprising over 800,000 AI-generated synthetic documents, including passports and driver’s licenses. It’s one of the only publicly available datasets designed for training and benchmarking and marks a shift in the industry to improve standardization and benchmarking.

The full 800,000+ dataset spans 20 document types from 10 U.S. states and 10 European countries and include the following attack vectors:

• Copy and move: Fraudsters take a legitimate biometric image and copy parts of it to another image or document to create a fraudulent identity
• Face morphing: Fraudsters use two or more facial images and digitally blend them into one composite image
• Face replacement: Fraudsters replace the face in an image with another person’s face using advanced editing or AI-based tools
• A combination of all: Fraudsters use a combination of all of the above

Testing and Benchmarking Our Solution

The IDNet dataset offers IDV and fraud detection providers an opportunity to test their solutions against third-party data.

Entrust’s Fraud Lab tested our identity verification technology against the IDNet dataset and achieved a 0.0% false acceptance rate (FAR) across 835,834 documents.

FAR measures instances of false acceptance – in other words, any instances when the technology incorrectly accepts a fraudulent document as genuine.

The results tell us that our fraud detection technology can detect a large range of synthetic attack vectors and AI-generated fraudulent documents, offering strong protection against some of today’s most prevalent attack vectors.

However, we aren’t satisfied with stopping there. As we know, fraud never stands still for long, and one of the downsides of the IDNet dataset is that it only contains four attack vectors (copy and move, face morphing, face replacement, and a combination of all three).

In comparison, our in-house Fraud Lab has identified and reproduced more than 100 different attack vectors across fraudulent documents. The IDNet dataset is therefore far less representative of what customers actually face in real-world scenarios. This lack of breadth leaves significant blind spots when benchmarking performance.

This is why today, our Fraud Lab uses a dual approach to test our solution:

• External, publicly available datasets: Such as IDNet or third-party labs, which provide a baseline but remain limited in scope and scale.
• In-house fraud generation: For more than three years we have been building one of the most comprehensive fraud libraries and generation platforms in the industry, covering both document and biometric attacks. This enables us to simulate production-like fraud more accurately and to measure performance against scenarios that are closer to what our customers see day to day.

We also address the domain gap – the mismatch between synthetic fraud samples and real-world fraud – by designing generation methods that narrow this gap, resulting in data that is more difficult to identify for an automated system and better predicts live performance.

Best Practices for Business to Combat Fraud With AI

When evaluating vendors that leverage AI in their detection, buyers should look beyond marketing numbers and ask more probing questions. Instead of simply accepting an answer to "what is your fraud performance?", dig deeper with "on what fraud samples was this measured?" and "how do these tests reflect real-world conditions?"

Specific questions to ask include:

• Adversarial testing protocols: Do you employ red teams or adversarial testing to challenge your systems? How sophisticated are these testers, and what percentage of attacks do they successfully execute?
• False positive distribution: Beyond aggregate false positive rates, can you demonstrate the distribution of errors by geos or document type?
• Transparency in failure cases: Can you share examples of where your system fails and explain why? Vendors who can't discuss failure modes likely don't fully understand their system's limitations or where verification responsibilities end and customer risk assessment begins.
• Temporal drift: How do you measure and mitigate performance degradation over time as fraudsters adapt? What's your retraining cadence and how quickly can you respond to new attack vectors?
• Third-party verification: Has your performance been independently verified by an unbiased third party using blind testing methodologies?