Tokenization

General definition

Tokenization has three distinct meanings, and search results mix them freely. In natural language processing and large language models it is the step that converts a string of text into a sequence of tokens, the units the model actually reads, counts and generates. In data security it is the replacement of a sensitive value with a token that has no exploitable meaning. In blockchain it refers to representing an asset as a digital token, which is outside the scope of this entry.

Modern language models do not read words. A tokenizer built with Byte Pair Encoding (BPE, used by the GPT family and Llama), WordPiece (BERT) or SentencePiece (many multilingual models) learns a vocabulary of frequent character sequences, typically 32,000 to 200,000 entries, and splits any input into them. Common English words become one token; rare words, code, numbers and non-Latin scripts break into several. A useful rule of thumb for English is about four characters or three quarters of a word per token, so 1,000 tokens is roughly 750 words. Tokens are the unit for everything downstream:

• The context window is measured in tokens, so long documents and long chat histories must fit inside it
• Model providers price input and output per token, so verbose prompts and retrieved passages cost real money
• Speed is quoted in tokens per second, and each generated token is one model step
• Embeddings and retrieval chunking operate on token counts, not characters

Security tokenization solves a different problem: how to keep a sensitive value out of systems that do not need it. A card number, national ID or health record number is stored in a hardened vault and replaced everywhere else by a random or format-preserving token; only the vault can map back. PCI DSS guidance allows properly tokenized card data to be treated as out of scope for many systems, and in healthcare the same technique pseudonymises protected health information (PHI) before it reaches analytics, logs or an AI model. In chat products this is the pattern behind masking a shared card number in a message, or de-identifying transcripts before they are used for training or evaluation. It is not encryption: a token cannot be decrypted, only looked up, which is why it is often preferred for data that must never be recoverable outside the vault.

In the Ethora ecosystem

Both meanings show up in the Ethora AI SDK. On the language side, each agent has a model choice and a knowledge base; retrieval selects only the relevant passages for a question, which keeps the tokens sent to the model, and therefore cost and latency, under control, and a per-room transcript memory gives the agent context between turns. Running a self-hosted model through an OpenAI-compatible endpoint removes per-token provider billing altogether, at the cost of your own hardware.

On the security side, the Trust and Safety system can catch sensitive values in messages with stop words, regex or AI moderation and block, censor or quarantine them, removing accidentally shared data immediately or after a delay, so a card number or a patient identifier need not sit in the archive in clear form. Access to chat and the API is by short-lived JWT chat tokens and app-level API tokens, a third, unrelated use of the word: those tokens are credentials, not substitutes for data.