The natural way to build an AI feature is request-response. User asks, you call the model, you stream the answer back. It is how every chat interface works and it is the right shape for most things.
It was the wrong shape for Dreamtime, and working out why changed how we think about latency generally.
The constraint was not technical
Dreamtime generates a personalised bedtime story for a child every evening — written to their age, illustrated, narrated.
Generating a story with illustrations is not fast. Even when everything works, you are looking at a meaningful wait. And the moment that wait happens is bedtime: a parent at the end of a long day, with a child who is tired and has been promised a story.
A spinner in that moment is not a slow feature. It is a broken routine. The entire product promise is “there is a story ready every night” — and “ready” is doing all the work in that sentence.
No amount of optimisation fixes this. You cannot make image generation fast enough that waiting for it at bedtime is acceptable, because the acceptable wait is roughly zero.
So we stopped generating at request time
Each afternoon a cron enqueues one job per household. A queue consumer picks it up, drafts the story against that household’s children and chosen values, requests the illustrations, and writes the finished thing into the library.
By the time anyone opens the app, the story has been sitting there for hours.
The queue runs on pgmq — inside the same Postgres that holds everything else. No
separate broker, no extra service to operate. For a workload measured in jobs per
household per day, a queue in the database you already have is more than enough, and it
is one less thing that can be down at 5pm.
What it costs
This trade is not free, and pretending otherwise would be dishonest.
You generate things nobody reads. Not every household opens the app every night. Some stories are produced, illustrated and never seen. That is real money spent on nothing, and at scale it is the dominant cost of the approach.
You commit earlier. A story generated at 3pm cannot react to something the parent decides at 7pm. Anything user-driven has to either wait for tomorrow or fall back to on-demand generation — so you end up maintaining both paths anyway.
Failures move. They no longer surface as an error in front of a user; they surface as an absence. That is better, but only if you are watching for it. A job that silently fails at 3pm and a parent opening an empty library at bedtime is a worse outcome than an honest error message, so the monitoring has to be genuinely good.
We took the trade because the cost is money and the alternative cost is the product promise. That is usually an easy call, and it was here.
The generalisable bit
The useful question is not “how do we make this faster”. It is when does this actually need to exist?
If the answer is “some time before the user needs it” rather than “the instant they ask”, you have a scheduling problem rather than a latency problem — and scheduling problems are far easier. Retries are invisible. Rate limits stop mattering. A slow model becomes an implementation detail rather than a user-facing number.
Lots of AI features have this shape and get built request-response out of habit. Daily digests, generated reports, recommendations, summaries of things that changed yesterday. Anything on a cadence rather than an impulse.
If you know roughly when a user will want something, the best latency optimisation available is to have finished before they ask.