Models and usage

The models come from the catalog registered with ai.AddModels(models, providers, defaultModelId), which the selector reads (IBitAiModelCatalog), grouped by family or provider. Picking another one moves the conversation onto a new session for that model; the old session only has to stop.

The choice reaches the transport in the session's RequestOptions, a BitAiRequestOptions sent with every request (BitAiRequest.Options): here its ModelId and Temperature. It also carries ProviderId, TopP, MaxOutputTokens, StopSequences, Seed, SystemPrompt, the ToolChoice (Tool calling), the ResponseFormat (Structured output) and an Extra dictionary for provider settings with no field of their own. Every one is optional, and a transport ignores what its provider cannot express. A session's Options can also be replaced between turns, when the transport itself stays the same.

The transport reports back what it used: a BitAiMetadataEvent with the model, the provider, the context window and the temperature, merged into the reply's metadata, and a BitAiUsageUpdatedEvent with the input, output and cached tokens and the cost. Usage is cumulative within one stream - a later report replaces an earlier one, so a provider's running totals can be forwarded as they come - and the streams of one turn add up, one per tool round trip.

Every field of the accounting is optional, because providers report different subsets of it. The components do the arithmetic they can: throughput from the output tokens over the streaming window, cost from the rates in the model catalog (BitAiModel.EstimateCost), and the share of the context window the conversation uses from the window the transport or the model reports. ShowUsage, ShowModelName and ShowTimestamps of BitAiUiOptions decide what a message shows.

• BitAiModelSelector - the picker, in the composer's prefix.
• BitAiContextMeter - the context used, in the composer's hint.
• BitAiTokenUsageView - the tokens, cost and speed of each answer.
• BitAiMessageActions - the row under an answer that carries the model, the usage and the time.