← Back to The State of Conversational AI, Q3 2026

Figure 3.1: The Five-Layer Anatomy of a Conversational AI System (Q3 2026)

Figure 3.1 from The State of Conversational AI, Q3 2026. Vector version below; a PNG version is available to download.

The five-layer anatomy of a conversational AI systemFive layers shown left to right: 1 Interface (apps such as ChatGPT, Claude and Gemini; terminal or CLI; messengers such as WhatsApp, Slack and Teams; widgets and SDKs) - where a user meets the system. 2 Orchestration (agents, tool use and routing, guardrails, evaluation) - the application logic and harness around the model. 3 Model (a cloud API such as OpenAI, Anthropic, Google, Azure or Bedrock, or an open-weight model such as self-hosted Llama, Qwen, Mistral, DeepSeek or Gemma) - the language model. 4 Memory (RAG retrieval, vector database, short-term and long-term memory) - what the model knows. 5 Infrastructure (cloud PaaS or IaaS, on premise or edge, and a serving stack such as Ollama, vLLM or llama.cpp) - where it runs.

The five layers, described

Every conversational AI product is assembled from the same five layers, shown left to right:

  1. Interface - where a person meets the system: apps (ChatGPT, Claude, Gemini), the terminal or CLI, messengers (WhatsApp, Slack, Teams), and widgets or SDKs embedded into existing software.
  2. Orchestration - the application logic and harness around the model: agents (prompts, pipelines, workflows), tool use and routing, guardrails, and evaluation.
  3. Model - the language model itself: a cloud API (OpenAI, Anthropic, Google, Azure, Bedrock) or an open-weight model run yourself (self-hosted Llama, Qwen, Mistral, DeepSeek, Gemma).
  4. Memory - what the model knows beyond its training: RAG (retrieval), a vector database, and short-term and long-term memory.
  5. Infrastructure - where it runs: cloud (PaaS or IaaS), on premise (own data centre or edge), and the serving stack that hosts open models (Ollama, vLLM, llama.cpp).
← Back to The State of Conversational AI, Q3 2026