# Can one AI workstation serve 50 people?

> Yes, if you size the model for the peak of parallel requests instead of the headcount. Check GPU memory and load test before you buy.

- URL: https://www.rebels.software/guide/what-one-ai-workstation-can-run
- Published: 2026-11-12
- Author: Bart Szczepański
- Tags: Private AI, Hardware, Local LLM

Yes, if you size the model for the peak of parallel requests instead of the headcount. One workstation with a single large GPU can serve a private assistant to a 50-person company when the jobs are document questions, summaries and drafts. It will not run the largest open models at long context for everyone at once, and it is not a training cluster.

What goes into the index is covered in [What belongs in a private AI knowledge base](/guide/what-belongs-in-private-ai-knowledge-base), and the cleanup before indexing in [Preparing documents for private AI search](/guide/prepare-documents-for-private-ai-search).

## What decides the answer

Three numbers decide whether one box is enough: the model's weights, the memory each open conversation needs, and how many requests arrive at the same moment.

Weights are simple arithmetic. A model stored at 16 bits needs about 2 bytes per parameter, at 8 bits about 1 byte, at 4 bits a little over half a byte. An 8-billion-parameter model fits in roughly 5 GB at 4 bits. A 32B model needs around 20 GB. A 70B model needs around 40 GB before anyone asks a question.

Every active conversation then holds a key-value cache that grows with the length of the context. A long document pasted into a chat costs memory for as long as that request runs. Serving engines such as [vLLM](https://docs.vllm.ai/) allocate that cache in small blocks ([PagedAttention](https://blog.vllm.ai/2023/06/20/vllm.html)) and batch requests together, which is why one GPU can serve several people at once instead of one after another.

Headcount is the wrong input. Fifty people rarely press enter in the same second. A useful planning number is the peak count of requests running in parallel, and in most offices that is a handful, not fifty.

## Step by step

1. **List the jobs people will do.** Write down what people will actually do in the first three months: questions over procedures and contracts, meeting or ticket summaries, email drafts, maybe transcription. Each one has a typical input length. A question with five retrieved passages is short. Summarizing a 60-page tender is long.

2. **Measure peak concurrency.** Run a pilot with 5 to 10 people for two weeks, or read the logs of a tool you already use. Count requests per minute at the busiest hour and how long each one runs. Multiply out to 50 people. That number, not 50, goes into the sizing.

3. **Pick the smallest model that passes your test set.** Write 30 to 50 real questions with known answers from your documents. Try a small model first, then a mid-size one. With good retrieval, a well-tuned model in the 8B to 32B range often answers document questions as well as a much larger one. Move up only when the answers fail, not because a benchmark says so.

4. **Do the memory math for the peak.** Add the weights at the chosen precision, the cache for your peak number of conversations at your typical context length, the embedding model for search and some headroom. Roughly: a 24 to 32 GB card handles models up to about 14B comfortably, or a 32B model at 4 bits with short contexts and few users. A 48 GB card fits a 32B-class model with room for context and several parallel requests. 70B-class models at 4 bits want around 96 GB, either one large card or two.

5. **Serve it with an engine built for many users.** Tools like Ollama are good for a pilot on one desk. For a team, use a server that batches requests, such as vLLM, behind your own login and logging. Run the embedding model and the vector database on the same machine. They are small next to the language model.

6. **Load test before you buy, or before you announce.** Replay your pilot questions at the measured peak, then at twice the peak. Watch time to first token and tokens per second per user. If the first word takes more than a few seconds at peak, people will stop using it. Fix that with a smaller model, a shorter context, or more GPU memory, in that order.

7. **Plan the parts around the GPU.** You need enough system RAM to load models without swapping, fast NVMe storage for the models and the index, a UPS, backups of the index and configuration, and monitoring of GPU memory and queue length. Decide what happens when the box is down for maintenance: a queue, a second machine, or a cloud model for non-sensitive work.

## An illustrative example

An engineering company with 50 people wants answers from its procedures, past offers and project notes, plus help with email drafts. A two-week pilot with eight people shows a busy hour of about 40 requests, with at most four running at once. Most prompts are a question with retrieved passages, under 8,000 tokens.

A 70B model would need two cards and still feel slow at that peak. The test set shows a 32B model at 4 bits answers 46 of 50 questions correctly, the same as the larger model on these documents. It runs on one 48 GB card with vLLM, together with the embedding model and the vector store. At twice the measured peak, the first word arrives in about a second. Summaries of long tenders run as background jobs, so they do not block the chat.

## Common mistakes

- **Sizing for headcount.** Fifty people is not fifty parallel requests. Measure the peak.
- **Buying for the biggest model on the leaderboard.** A smaller model with clean documents and good retrieval often wins on your questions.
- **Forgetting the context cache.** The weights fit, then three long documents arrive at once and the server runs out of memory.
- **Using a desktop chat app as the server.** Single-user tools queue requests. Use an engine that batches them.
- **Skipping the load test.** A demo with one user tells you nothing about Monday at 9:00.
- **Leaving downtime unplanned.** One machine means one point of failure. Decide the fallback before people depend on it.

## Next step

Pick one job, run a two-week pilot with a handful of people on a single GPU, and log every request. Those logs give you the peak load, the context length and the model size for sizing the workstation for the whole company.

Rebels Software builds private AI on infrastructure you control, as part of the platforms and applications we deliver. In a Discovery workshop we map the jobs and the documents, then prove one case as a PoC on your hardware or a rented GPU, with a load test at your real peak. After deployment, Rebels Care keeps the models, the index and the machine running. If that matches where you are, book a call on this page or start from the [Rebels Software homepage](/).

We start with your biggest problem and the data behind it. [Book a free call](https://calendly.com/bartlomiej-jacyno/intro)
