Track: Infrastructure & ScaleFormat: TalkLevel: Intermediate
Prompt Caching, Batching and Other Boring Wins
Tuesday, May 11: 9:00 AM – 9:30 AM
Mission Hall
Unglamorous optimisations with outsized effects, benchmarked on production traffic rather than a synthetic loop.
Speakers (1)
- KBKwame Boateng
Staff Engineer, Anansi Systems
Kwame works on inference runtimes and open-model deployment on hardware that was never meant for it.