Trust and Safety AI: Moderation and Fraud for Marketplaces

Two-sided marketplaces and platform businesses carry a problem that most software companies don't: they host content and transactions created by people they don't employ and can't fully vet in advance. Listings, reviews, messages, and payment behaviour all need to be screened at a volume that no human team can review line by line. This is where AI earns its keep — but it is also where the limits of AI are most visible, and where getting the human-AI boundary wrong has real consequences for users and for the business.

This guide walks through where AI genuinely helps with content moderation, listing fraud, and account abuse detection on marketplace platforms, where human review has to stay in the loop, and why this problem looks different from the customer-support chatbot use case many teams default to when they think about "AI in the product."

What is trust and safety AI?

Trust and safety AI is the set of machine learning and rules-based systems that detect, prioritise, and in some cases act on content or behaviour that violates a platform's policies — spam listings, counterfeit goods, fake reviews, account takeover, collusive buyer-seller fraud, and harassment in messaging. It is a classification and risk-scoring problem, not a conversational one.

The systems typically combine several layers: rules and heuristics for known abuse patterns, supervised classifiers trained on historically actioned content, anomaly detection for behaviour that doesn't match any known pattern, and graph-based analysis that looks at relationships between accounts, devices, and payment instruments rather than single listings in isolation.

How is this different from a customer-support chatbot?

A support chatbot is built to be helpful to the person in front of it and is judged on resolution quality and user satisfaction. Trust and safety AI is built to be adversarial by design — it is screening people who are actively trying to evade detection, and false negatives and false positives both carry cost, just in different directions (a missed scam versus a wrongly suspended legitimate seller).

That adversarial framing changes the engineering approach in a few concrete ways. Trust and safety models need to be retrained more frequently because bad actors adapt once they learn what gets flagged. They need strong audit trails and explainability, because every action can be appealed or disputed. And they generally need a tiered review workflow rather than a single automated response, because the cost of an incorrect enforcement action (removing a legitimate listing, banning a genuine account) is usually higher than the cost of a delayed one. A support chatbot rarely needs any of this — it optimises for response speed and accuracy on a known, cooperative user base.

Where does AI genuinely help with content moderation?

AI is well suited to high-volume, pattern-detectable triage: flagging likely-prohibited listings for review, detecting duplicate or templated scam content, clustering fake reviews, and scoring message threads for known harassment or circumvention patterns (e.g. attempts to move a transaction off-platform). It is a prioritisation and filtering layer that makes a fixed human review team far more effective, not a replacement for the judgement calls at the edge.

What AI is not well suited to is nuanced policy interpretation — satire versus harassment, cultural context in a review, a borderline listing that technically complies but tests the spirit of a policy. These decisions carry reputational and sometimes legal weight, and they benefit from a documented human escalation path.

Where does human review remain essential?

Human review stays essential wherever a decision is contestable, policy-ambiguous, or carries meaningful consequences for a user's livelihood or safety. Automated systems should generally handle the high-confidence, high-volume end of the distribution — the obvious spam, the exact-duplicate scam listing — while routing genuinely uncertain cases to trained reviewers with full context and an appeal mechanism.

A practical way to think about the split is confidence-based tiering: very high-confidence violations can be auto-actioned with monitoring, mid-confidence cases get queued for fast human review, and low-confidence or policy-novel cases get escalated with more context and slower turnaround. This keeps the automation doing the volume work it's good at while keeping accountability for consequential decisions with a person.

Use case	AI fits well	Human review needed
Duplicate/templated spam listings	Yes — high-volume pattern match	Rarely, except appeals
Counterfeit or prohibited goods	Yes, as a triage signal	Yes, for borderline or novel items
Fake review detection	Yes, for clustering and velocity signals	Yes, for contested individual cases
Account takeover / credential stuffing	Yes — behavioural and device signals	Yes, for account restoration decisions
Collusive buyer-seller fraud rings	Partially — graph analysis surfaces candidates	Yes — investigation and action
Harassment or policy-ambiguous messaging	Limited — can flag, rarely should auto-action	Yes, almost always
General customer support queries	Yes — different problem entirely, optimised for resolution speed	Only for escalations

How should a marketplace approach fraud and abuse detection technically?

The technical foundation matters more than the choice of model. Fraud and abuse detection depends on having reliable, joined-up data — account history, device and session signals, transaction patterns, and prior enforcement actions — accessible in near real time to whatever scoring system sits in front of a listing or account action. Many marketplaces find that their biggest blocker isn't model accuracy, it's that the underlying data is scattered across systems that were never built to talk to each other. That's a data infrastructure problem before it's an AI problem, and it's worth solving in that order.

Once the data foundation is solid, the model layer itself usually benefits from starting simple: rules and lightweight classifiers can catch a meaningful share of abuse before more sophisticated graph or anomaly-detection models are justified. This mirrors the general pattern we see across AI product strategy work — the right starting point is rarely the most sophisticated model available, it's the one that matches the actual volume, risk profile, and review capacity of the team operating it.

What should Australian platforms consider from a governance standpoint?

Australian organisations deploying AI systems — including trust and safety classifiers — operate against a backdrop of voluntary national guidance rather than AI-specific legislation at present. The Department of Industry, Science and Resources (DISR) is the lead Australian Government agency for AI policy, and the government's earlier Voluntary AI Safety Standard has been succeeded by Guidance for AI Adoption, which sets out expectations around transparency, accountability, and human oversight for AI systems. Trust and safety AI is a strong candidate for applying this guidance deliberately, since enforcement decisions directly affect users' access to a platform and their livelihoods. Documenting model decisions, keeping a human appeals path, and maintaining audit trails aren't just good practice here — they align directly with the oversight expectations the guidance describes.

Should you build, buy, or combine approaches?

Most marketplaces end up combining a commercial content-moderation API for baseline coverage (image and text classification for clearly prohibited content) with custom-built fraud and abuse models tuned to their specific marketplace dynamics, since off-the-shelf fraud detection rarely understands the particular ways abuse shows up in a given category or geography. The decision of whether to build that custom layer in-house, buy a specialised SaaS fraud product, or engage an embedded team to build and hand over a system you own is really a delivery-model question as much as a technical one — worth thinking through deliberately rather than defaulting to whichever vendor your sales team spoke to first.

If you're building or extending a custom detection layer, it needs to be engineered with the same production rigour as any other core system — monitoring for drift as fraud patterns shift, clear ownership of false-positive and false-negative rates, and a maintainable codebase your team can run without the original builders. That's the focus of our AI engineering work, and it's a different conversation from standing up a support chatbot.

For more on the build-versus-buy decision generally, and on getting the data foundations right before model work starts, have a look through our insights.

If you're weighing up how to approach content moderation, fraud detection, or account abuse for your platform, we can help — starting with an honest assessment of where automation will genuinely reduce risk and where it won't.