The first time I built a chat UI on top of an LLM, I did the obvious thing. I called the API, awaited the response, and rendered it.
It worked. It also felt terrible. Users stared at a spinner for 8-15 seconds, and then a wall of text appeared all at once. Nobody thought it was broken, but everybody thought it was slow.
That typewriter effect in ChatGPT isn’t an animation. It’s the actual response arriving token by token over a stream. Once you stream, time to first visible text drops from “however long the whole answer takes” to “however long the first token takes”, which is usually well under a second or two.
Here’s how to build it properly, and the bugs I hit along the way.
The big picture
Three pieces:
- Your server calls the LLM provider with streaming turned on, and re-emits each chunk to the browser as Server-Sent Events (SSE).
- The... premium content.
