Performance & Evaluation

Throughput

How many requests an AI system can handle per unit of time.

Definition

Throughput measures the volume of AI requests a system can process — queries per second, documents per hour, or tokens per minute. This is critical for planning AI infrastructure at scale. A model that handles one document per minute fine in testing may become a bottleneck when an entire legal team is using it simultaneously. Understanding throughput requirements and planning infrastructure accordingly is essential for production AI deployments.

Related Terms

Latency

The time between sending a prompt and receiving a response.

Model Serving

The infrastructure that makes a trained model available to receive and respond to requests.

Scalability

The ability of an AI system to handle increasing numbers of users or requests.

Heard enough terminology — ready to talk outcomes?

We translate AI concepts into measurable business results. No upfront fees — you pay only when independently verified results are delivered.

← Back to glossary

Disclaimer

This definition is provided for educational and informational purposes only. It represents a general explanation of a technical concept and does not constitute professional, technical, or investment advice. Artificial intelligence is a rapidly evolving field; terminology, techniques, and capabilities change frequently. Coaley Peak Ltd makes no warranty as to the accuracy, completeness, or currency of the information provided. Nothing on this page should be relied upon as the sole basis for commercial, technical, legal, or investment decisions without independent professional advice.

Document reference: ISO_webpage_knowledge-base_glossary_v1

Last modified: 29 March 2026

Knowledge Base·Performance & Evaluation·Throughput