DevFlow Conf 2027

Caching, batching, routing and quantisation, measured end to end on a real workload. Includes the changes that looked clever and made things worse. Our inference bill grew faster than our traffic for two quarters straight, and the obvious answers — a smaller model, a cheaper provider — were off the table for quality reasons. So we went looking for the money everywhere else: in the cache we were not reusing, in the requests we were sending one at a time, and in the long tail of prompts nobody had read since launch. In this hands-on session we rebuild that programme from scratch against a live workload. You will add a prompt cache and measure the real hit rate rather than the hoped-for one, batch a stream of requests and watch the latency percentiles you just traded away, route by difficulty so the expensive path only handles the requests that need it, and trim context that has been dead weight since the second sprint. We finish with the cost model we now review monthly — cost per resolved request rather than cost per token — plus the two optimisations we rolled back and the graphs that told us to. Bring a laptop; every exercise ships with a workload you can run locally.