Inference

A model’s life has two phases. In training, it learns from data, an expensive process done rarely. In inference, it is put to work answering, millions of times a day.

When you pay per token through an API, you are paying for inference. Companies that serve models, whether or not they train their own, are called inference providers.

Why the term comes up

Cost, latency and where the model runs are all inference questions. Running an open-weight model on your own server, for example, means doing inference yourself.