TinyServe: how an AI model answers

Teams rolling out AI keep asking: why does a long prompt compute fast, while the answer comes out slowly? Why does graphics card memory run out on a longer conversation? Slides do not answer that. TinyServe shows these mechanics live, on a real model, with no server and nothing to install.

Prefill vs Decode: two phases shown separately: a long prompt computes fast and in parallel, while the answer comes out one token at a time. The time of every token is measured. KV Cache: a chart of memory climbing with every token, against the budget the context window allows. The size is calculated from the model's config, not read from the graphics card. Attention: an attention heatmap for the chosen layer. It is a stand-in, not a readout: the runtime does not expose real attention tensors yet, so the pattern is modeled and labeled as such in the app. The strip under the map shows the model's measured uncertainty at every token. Speculative Decoding: a small model proposes a token, the large model accepts or rejects it. The target model runs for real, while the propose-and-verify cycle is simulated. Paged Memory: a simulation: paged memory allocation (as in vLLM) next to contiguous allocation, which fragments. Up to eight requests at once, each named R1 to R8. Models: five models side by side: layers, sizes and benchmark scores with the source and the date it was read. These are the model makers' numbers, not our measurements, and bars marked ~ are labeled estimates. Sampling: one step's full scorecard: the probabilities the model was choosing between, at the eight steps where it was least sure. In a live run you change the temperature with a slider and the bars recompute exactly.

A run tape sits under the stage: one bar per token (height is the token's time, the tick under it is the model's uncertainty), and pins lead to the eight forks. The tape's cursor picks the step that views 01, 02, 03 and 07, the step card and the readouts show. On a first visit a recorded Qwen3 0.6B run plays at once, before the model is downloaded: the text types itself token by token and the counter in the header reads "n of 101" (recorded on Apple Metal-3, played slower than recorded).

5 models you load one after another (Qwen3 0.6B and 1.7B, Llama 3.2 1B, LFM2.5 1.2B Think, DeepSeek-R1 1.5B) and compare in the Models view. Switching models reloads the page. 579 MB is the smallest model (Qwen3 0.6B): the size is read from huggingface.co before the download (measured: 578.9 MB), not estimated. A second visit starts from the browser cache. 0 requests to a server to produce the answer: the model computes it on your graphics card. The page asks only huggingface.co for file sizes (7 requests, none carrying a prompt; counted with Playwright in Chromium with a simulated WebGPU adapter).

The answer is computed on your graphics card, not on a server. Measured: the time of every token and the next-token probabilities. Calculated from a formula: the KV cache size. Simulated or approximate: the attention map, speculative decoding and paged memory. Model makers' numbers: the benchmark scores in the Models view, with the source beside every model.

The app after the start panel is closed: the view list on the left, the time-per-token chart in the middle, readouts and the transcript on the right, and the run tape underneath, paused at step 58, one of the eight forks.