Glossary

Tensor Parallelism

Tensor parallelism slices each layer of a model across several GPUs so a model too big for one card can run, and a model that fits can run faster. It is the standard way to serve 70B-class models.

General definition

Tensor Parallelism (TP) splits individual layers of a model across GPUs. Instead of giving each GPU whole layers, the large weight matrices in the attention and feed-forward blocks are sliced by rows or columns; each GPU multiplies its slice, then the partial results are combined with a collective operation (an all-reduce) before the next layer begins. The approach was popularised by NVIDIA’s Megatron-LM work in 2019 and is built into every major inference server.

The first reason to use it is memory. A 70B-parameter model in 16-bit precision needs about 140 GB for its weights alone, before the KV cache, which no single GPU holds. Split four ways, each GPU needs about 35 GB plus cache. The second reason is speed: token generation is limited by memory bandwidth, and TP adds the bandwidth of every GPU, so even a model that fits on one card can generate faster on two.

  • Pipeline parallelism assigns whole layers to different GPUs and passes activations down the chain; less communication, but GPUs idle while waiting for each other unless micro-batches keep the pipeline full
  • Data parallelism runs full copies of the model on each GPU and splits the requests between them; it raises throughput but does not help a model fit
  • Expert parallelism places the experts of a mixture-of-experts model on different GPUs
  • Servers combine them: TP inside a machine over a fast interconnect, pipeline or data parallelism across machines

The practical rules: TP works best within one machine on a fast GPU interconnect, because the all-reduce happens in every layer; the TP degree is normally a power of two that divides the number of attention heads; and quantisation is the alternative to check first, since a 4-bit 70B model fits in roughly 40 GB and may need no splitting at all. In vLLM it is one flag (--tensor-parallel-size); the hard part is having the GPUs. For inference on 7B to 30B models a single GPU is almost always enough, and the model serving stack hides the detail from the application either way.

In the Ethora ecosystem

Tensor parallelism is a serving-side concern, and Ethora stays on the application side: agents call a model through an OpenAI-compatible endpoint configured with four environment variables, so whether the model behind it spans one GPU or eight is invisible to the chat layer. That separation is deliberate. The self-hosted AI agent stack can start on a CPU box with a 3B model (6 to 8 seconds per short reply in the official Ollama example) and later move to a tensor-parallel 70B deployment on vLLM without changing agents, prompts or the pgvector knowledge base.

Where it comes up: regulated customers who want a large open model inside their own healthcare or finance network, where a 70B model on several GPUs is the price of keeping data in. The model endpoint is service-wide, so one such deployment serves every agent in the installation. A quantised model on a single GPU is usually worth testing first, and for a dedicated Ethora server the messaging stack can run in the same cloud account where the GPUs live.

Get started

Build AI agents on your own stack

Ethora’s AI SDK brings agents, RAG and self-hosted LLMs into your product. Talk to our team.

Start Free
Free tier available Enterprise SLA No vendor lock-in