Glossary
Trust & Safety (TnS)
Trust & Safety (TnS) is the discipline, and in a chat platform the server-side system, that protects users and the operating business from harm that happens through messaging: abusive language, unsafe content, fraud, and leaks of personal or regulated data. A TnS system scans messages in real time and blocks, censors, quarantines or reports them according to severity.
General definition
Trust & Safety is the function inside online platforms that is responsible for keeping the product safe for the people who use it and for the company that runs it. It started as an operations role in marketplaces and social networks and is now a formal requirement under regulation such as the EU Digital Services Act, the UK Online Safety Act and, for services used by children, COPPA in the United States. In regulated industries the same function is what stops protected health information, card numbers or client data from leaking through a chat window.
The scope of Trust & Safety is wider than content moderation. A complete programme covers:
- Content moderation: profanity, hate speech, harassment, sexual or violent content, spam
- Data loss prevention: detecting personal, financial or medical data that should not be shared in the channel
- Account integrity: bots, impersonation, underage accounts, fraud patterns
- Enforcement policy: a ranked scale that defines what happens at each level of severity
- Reporting and audit: a record of what was detected and what action was taken, available to compliance and legal teams
In a messaging product the hard part is that all of this has to happen on a live stream of messages. A rule that adds noticeable latency, or that only runs on a sample of traffic, is not usable at scale. That is why mature chat platforms implement Trust & Safety at the server and protocol level rather than in the client app.
How a Trust & Safety system works
A TnS system has three moving parts: detection, enforcement and reporting.
Detection usually combines three methods, each with a different cost and accuracy profile:
- Stop words and regular expressions. Fast, deterministic matching for blacklisted phrases, profanity and structured sensitive data such as card numbers, phone numbers, national ID formats or patient identifiers. Runs in microseconds and is the first line of defence.
- AI auto-moderation. An LLM or classifier evaluates the message against a policy written as a system prompt. This catches context, tone and intent that keyword lists miss, and the policy can be updated by the operator without a code release. See AI content moderation for the trade-offs.
- Human review. User reports and administrator tools handle appeals and the cases automation cannot decide. A good system routes only the ambiguous fraction of traffic here.
Enforcement is a ranked scale. The same detection can trigger a different response depending on the severity assigned to the rule:
- Blocking: the message is stopped before delivery and never reaches other participants
- Censoring: the offending fragment is masked or removed and the rest of the message is delivered
- Quarantining and reporting: the message is held for review and an alert is sent to the host application or a moderator
- Reporting only: the message is delivered but the event is logged for internal compliance review
Reporting connects the system to the rest of the enterprise stack. Events are pushed to a message bus, an API endpoint, a SIEM or an email digest, and written to an audit log so that compliance teams have evidence of both the violation and the response.
In the Ethora ecosystem
The Ethora Trust & Safety system (TnS) is a configurable business-logic layer built into the Ethora chat server. It is applied in real time at the chat server and messaging protocol level, so every message passes through it before delivery, and it is optimised for high-load processing: scanning is automated and anonymised, and it does not depend on the client app doing the right thing.
When a rule matches, the system applies one of the four enforcement outcomes above according to the severity you configured: block, censor, quarantine and report, or report only. Detection uses stop-word lists, regular expressions and AI auto-moderation with an operator-defined policy prompt, so community guidelines and data-handling rules can be changed without waiting for a vendor release. Suspected violations can be reported in real time to your own systems through a message bus, an API call or an email report, and recorded in the compliance audit trail.
Typical configurations by industry:
- Healthcare: if a patient or clinician shares sensitive personal data by accident, the TnS system can remove it straight away or after a configurable delay so it is not retained in the message archive or logs, supporting HIPAA and GDPR obligations. See the healthcare chat SDK.
- Finance and insurance: block card numbers, account identifiers and other regulated data from leaving the permitted channel, combined with role-based access control on the room and user level.
- Gaming, communities and youth media: profanity and toxicity filters, blocklists and reporting hooks that feed your obligations under COPPA and the UK Online Safety Act.
- Marketplaces and customer support: stop off-platform contact details, scams and abusive language in buyer-seller and customer-agent conversations.
Because the TnS system runs at the server level, customers on dedicated and self-hosted Ethora chat servers can configure it to their own policies and regional laws. Paired with a self-hosted LLM for the AI moderation step, neither the rules nor the content being classified leave your infrastructure.
