LLM Token Limits Explained: Context Windows, Token Budgets, and Cost Control
tokenscontext windowsLLM costsprompt optimizationAI engineering

LLM Token Limits Explained: Context Windows, Token Budgets, and Cost Control

QQBot Editorial Team
2026-11-08
7 min read

Understand LLM token limits, context windows, token budgets, truncation, and a practical method for estimating AI costs.

LLM token limits affect whether a request succeeds, how much conversation history an application can retain, and what each API call costs. This guide explains context windows, input and output budgets, truncation, and a repeatable worksheet for estimating usage before deploying a chatbot, RAG workflow, or other conversational AI feature.

Overview

A token is a unit of text processed by a language model. It may represent a whole word, part of a word, punctuation, or formatting. Token counts therefore do not map exactly to word counts. English text is often estimated using a rough characters-to-tokens or words-to-tokens conversion, but the result varies with language, code, numbers, markup, and the tokenizer used by the model.

An LLM request normally has at least two token categories:

  • Input tokens: the system instruction, user message, conversation history, retrieved documents, tool definitions, and other content sent to the model.
  • Output tokens: the response generated by the model.

The context window is the maximum amount of input and output context the model can handle in one request. It is not necessarily the same as the maximum response length. A request can fail even when the requested answer is short if the system prompt, chat history, and retrieved material already consume most of the available window.

Token limits are separate from rate limits. A context limit concerns the size of one request. A rate limit concerns how many requests or tokens an account may process within a period. Both should be monitored during AI deployment.

TermWhat it meansWhy it matters
Context windowTotal token capacity available to a requestDetermines how much history and supporting material can fit
Input limitMaximum tokens accepted before generationControls prompt, history, tools, and retrieved content
Output limitMaximum tokens the model may generatePrevents unexpectedly long or incomplete responses
Token budgetYour planned allocation of input and output tokensCreates predictable cost and response behaviour
TruncationRemoving or shortening content to fit a limitCan silently reduce answer quality if poorly designed

Do not assume that a provider will truncate content in the way your application expects. Some systems reject oversized requests; others may apply a framework-level history policy. Your own application should decide what to keep, summarise, retrieve, or discard.

How to estimate

Start with a per-request budget rather than a model’s advertised maximum. A practical calculation is:

Estimated request tokens = fixed instructions + conversation history + retrieved content + tool definitions + current user input + reserved output

For cost estimation, separate input and output because providers may price them differently:

Estimated cost = (input tokens ÷ pricing unit × input rate) + (output tokens ÷ pricing unit × output rate)

Keep the rates as variables in your worksheet. Provider pricing, model versions, billing units, cached-input rules, and tool charges can change. The worksheet should therefore record the date, provider, model identifier, currency, and pricing page used for each calculation.

For a production chatbot, calculate three scenarios:

  1. Typical: the normal system prompt, average history, average retrieval result, and expected answer.
  2. Large: a long user message, more retrieved documents, and a fuller answer.
  3. Worst practical case: the maximum history and retrieval content your application will permit, not an unlimited theoretical maximum.

Leave a safety margin. Context limits can be affected by structured messages, tool schemas, formatting, and tokenisation differences. A budget that uses every available token is difficult to operate reliably.

Inputs and assumptions

Create a token-budget worksheet with one row for each component of the request. The following structure works for a customer support chatbot, website AI assistant, or RAG chatbot:

ComponentEstimated tokensControl to apply
System instructionsMeasure from the actual promptRemove repeated rules and unnecessary examples
Conversation summarySet a fixed allowanceSummarise older turns rather than retaining all messages
Recent messagesSet a message or token capKeep the turns most relevant to the current task
Retrieved passagesCount after chunk selectionUse relevance thresholds and a maximum passage budget
Tool definitionsCount the complete schemaSend only tools available for this task
Current user inputMeasure or cap itReject, shorten, or route unusually large inputs
Reserved outputChoose an explicit maximumUse a format and length appropriate to the task

Record assumptions that could change the estimate. These include the tokenizer, language mix, average number of conversation turns, document chunk size, number of retrieved chunks, expected answer length, and whether tool calls add another model request. A single user interaction may involve several calls: classification, retrieval, answer generation, and a follow-up tool request. Estimate the workflow, not only the visible final answer.

For a RAG workflow, do not fill the context window with every plausible document. Retrieve fewer, more relevant passages and instruct the model how to use them. If the source material is long, consider a map-and-reduce summary, section-level retrieval, or a staged answer process. The RAG chatbot guide covers retrieval and evaluation decisions that directly affect token use.

Worked examples

Example 1: A support conversation

Assume a support request has 700 tokens of system instructions, 1,800 tokens of recent conversation, 500 tokens of a conversation summary, and 300 tokens in the user’s new message. You reserve 600 tokens for the answer.

Estimated context use = 700 + 1,800 + 500 + 300 + 600 = 3,900 tokens.

This is a planning figure, not a provider limit. Before deployment, measure the serialised request with the tokenizer associated with the selected model. If the conversation can grow indefinitely, define a trigger that summarises older turns or removes low-value messages before the next request.

Example 2: A retrieval request

Assume the fixed prompt uses 900 tokens, the user question uses 120, and five retrieved passages use 700 tokens each. You reserve 700 tokens for the answer.

Estimated context use = 900 + 120 + (5 × 700) + 700 = 5,220 tokens.

If retrieval returns ten passages instead, the same request would add another 3,500 input tokens. That may increase cost without improving the answer. Test a smaller number of passages, deduplicate overlapping text, and log whether the selected passages were actually cited or used.

Example 3: Cost planning with variables

Suppose an application averages 4,000 input tokens and 600 output tokens per request, and receives 2,000 requests in a planning period. Keep the provider’s input and output rates as variables rather than copying a price into application code.

Total input tokens = 4,000 × 2,000 = 8,000,000.
Total output tokens = 600 × 2,000 = 1,200,000.

Apply the current rates to these two totals, then add any separate charges or additional model calls. Run the same calculation for the large and worst-practical scenarios. This makes it easier to see whether cost is driven by user volume, long context, retrieval, output length, or workflow design.

For ways to reduce unnecessary prompt growth, review your business prompt templates and remove repeated instructions, verbose examples, and irrelevant history. Token reduction should not remove safety constraints or information required for a correct answer; the objective is a smaller, more relevant request.

When to recalculate

Revisit your token budget whenever a model, provider, prompt, retrieval pipeline, or user behaviour changes. In particular, recalculate when:

  • the provider changes model limits, tokenizer guidance, billing units, or pricing;
  • you add tool calling, structured output, browsing, or another model step;
  • conversation history begins causing errors, shortened answers, or slow responses;
  • retrieval returns larger documents or more passages than before;
  • users begin submitting more code, tables, transcripts, or non-English text;
  • your observed input or output token distribution moves away from the original estimate;
  • you change the system prompt, answer format, or maximum output setting.

As a maintenance routine, review the worksheet at least when your application undergoes a material change and on a regular schedule, such as twice a year. Compare estimates with logged usage using percentiles rather than only averages. Averages can hide a small number of oversized requests that cause failures or disproportionate costs.

When a request is too large, troubleshoot in this order: measure every component, confirm the model’s current limits, identify the largest contributor, cap or summarise history, reduce retrieval volume, remove unused tools, and set an explicit output budget. Then test typical and worst-practical cases again. If the application depends on long-running conversations, store durable facts outside the context window and retrieve only what the current task requires. This approach makes token limits an engineering input to manage, rather than a surprise discovered in production.

Related Topics

#tokens#context windows#LLM costs#prompt optimization#AI engineering
Q

QBot Editorial Team

AI Deployment Editor

Senior editor and content strategist. Writing about technology, design, and the future of digital media. Follow along for deep dives into the industry's moving parts.