--- title: Leveraging KV Cache for Efficient Programming with LLMs description: Discover how to optimize programming with large language models using KV Cache, enhancing efficiency, reducing costs, and improving response times. image: https://tech.univio.com/hubfs/KV%20Cache%20cover-1.png --- [Skip to content](https://tech.univio.com/leveraging-kv-cache-for-efficient-programming-with-llms#main-content) [![tech\_univio\_blog\_COLOR+BLACK](https://tech.univio.com/hs-fs/hubfs/tech_univio_blog_COLOR+BLACK.png?width=850&height=300&name=tech_univio_blog_COLOR+BLACK.png)](https://tech.univio.com/) - [GitHub](https://github.com/Univio-Tech) - [PL \>](https://tech.univio.com/pl/kv-cache-jak-wykorzysta%C4%87-kluczowy-mechanizm-llm-w-pracy-programist) ![](https://tech.univio.com/hubfs/raw_assets/public/univio-tech-blog/images/icons/search.svg) Search the blog - There are no suggestions because the search field is empty. # Leveraging KV Cache for Efficient Programming with LLMs Table of contents 1. How Does an LLM Remember Context? 2. Is Caching Available in Every Model? 3. Putting KV Cache Knowledge into Practice 4. How KV Cache Impacts Token Costs 5. Benchmarks: KV Cache in Action 6. Key Takeaways Caching is a concept every developer knows well: we store the result of an operation so we don't have to recompute it on the next request. A similar mechanism works with equal success in large language models. In the LLM world, the counterpart to cache memory is **KV Cache (Key-Value Cache)**. While it operates under the hood, understanding it helps you run models faster, more efficiently, and at a significantly lower cost. ## How Does an LLM Remember Context? Let's start with a bit of theory. When a query hits a Transformer-based LLM, it passes through two main phases (assuming the model runs on a GPU): Phase I: Prefill. This involves processing the entire input prompt using the full compute power of the GPU. During this phase, Key-Value (KV) states are calculated and saved in memory for every token. Put simply: K (Key) answers what a given token represents, while V (Value) carries what content it holds. The memory used here is the GPU's VRAM. Processing in this phase happens in parallel. Phase II: Decoding. This is the stage where the model generates its response token by token. As it generates each new token, it reaches into the previously allocated VRAM to read back the existing KV pairs. This reveals a clear difference in execution profile: the prefill phase is heavily GPU-bound, whereas the decoding phase is memory-bound. To visualize this read/write flow, take a look at the concept below: [![uproszczony schemat konwersacji z modelem](https://tech.univio.com/hs-fs/hubfs/Gemini_Generated_Image_uuuqajuuuqajuuuq.jpeg?width=1365&height=768&name=Gemini_Generated_Image_uuuqajuuuqajuuuq.jpeg)](https://tech.univio.com/hubfs/Gemini_Generated_Image_uuuqajuuuqajuuuq.jpeg?hsLang=en) The diagram above illustrates a simplified conversation flow with a model. In turn 1, the model must process all input tokens during the prefill phase and write them into the KV Cache. In turn 2, it begins generation right from token #5, fetching the prior conversation (including its own previous outputs) directly from cache memory. ## Is Caching Available in Every Model? KV Cache is a foundational component of every Transformer-based LLM system. While you can influence how it behaves, your degree of control depends on how you interact with the model: - Directly via API: when manually constructing the system prompt, tool definitions, and user prompts. - Through ready-made agentic frameworks: such as Claude Code. Different model providers offer varying levels of control. Anthropic is a great example: when querying their models directly via API, you can pass a cache\_control parameter to explicitly mark which blocks of context should be cached. This provides far more granular control than relying on high-level agentic wrappers. [![KV Cache cover](https://tech.univio.com/hs-fs/hubfs/KV%20Cache%20cover.png?width=605&height=235&name=KV%20Cache%20cover.png)](https://tech.univio.com/hubfs/KV%20Cache%20cover.png?hsLang=en) \[Code demonstrating prompt caching using Anthropic API block-level caching. Source: https://platform.claude.com/docs/en/build-with-claude/prompt-caching\] ## Putting KV Cache Knowledge into Practice Let's look at a practical example using Claude Code and custom AI harness development. The context window passed on every request typically consists of three main parts: - System prompt – primary instructions and tool definitions. - Project context – files like CLAUDE.md and project-specific guidelines. - Conversation history – the most dynamic section. Each part has a distinct lifecycle. In an ideal world, everything would remain static, resulting in a 100% cache hit rate on every request. In reality, that's impossible—conversation history naturally grows with every turn. That is why keeping the system and project layers as static as possible is crucial. Avoid embedding dynamic data into these early sections—such as real-time timestamps or dynamically generated directory trees. Any change in the prefix automatically invalidates the entire downstream cache. Beyond prompt edits, specific environment actions will also clear your cache: - Changing models or adjusting the reasoning depth (Effort level). - Adding or removing MCP servers (which mutates tool definitions in the system prompt). - Toggling plugins on or off. - Executing commands like /clear or /compact (which reload project rules). - Uploading large attachments (images, PDFs) that exceed provider caching thresholds. It's also worth noting how caching works across multi-session setups on a single machine. Sessions running in the same working directory share the same cache. If you spin up sub-agents, each sub-agent maintains its own isolated cache—it doesn't pollute or automatically inherit from the parent agent. Only explicit commands like /fork will copy the parent's cache state over to a new sub-agent. Finally, cache has a shelf life. Extended periods of inactivity mean the provider eventually drops it from memory. This window is governed by a TTL (Time To Live) parameter, which usually ranges from 5 minutes to 1 hour depending on the provider and tier. ## How KV Cache Impacts Token Costs Your context memory strategy directly impacts your API bills. The rate for an initial call (a full prefill with zero cache hits) is higher than subsequent requests that hit the cache. [![Obraz2 artykuł](https://tech.univio.com/hs-fs/hubfs/Obraz2%20artyku%C5%82.png?width=907&height=214&name=Obraz2%20artyku%C5%82.png)](https://tech.univio.com/hubfs/Obraz2%20artyku%C5%82.png?hsLang=en) \[https://platform.claude.com/docs/en/about-claude/pricing\] Looking at the pricing structure above, writing to the cache (`Prompt Caching / Write`) incurs a slight premium over standard base tokens. However, the real savings hit when a query lands a cache hit reducing the cost to a fraction of the base rate ($0.25). The 5-minute and 1-hour write options simply correspond to different TTL retention settings. ## Benchmarks: KV Cache in Action To verify the theory in practice, I ran a benchmark measuring response times on repeated requests across two setup types: 1. **Bielik-4.5B** – running locally via Ollama on CPU. 2. **GPT-5-mini** – queried via an enterprise AI Gateway. Both tests track TTFT (Time To First Token) using a fixed prefix containing a question. The prefix length must exceed 1,024 tokens; below this threshold, caching overhead outweighs performance gains. | Parametr | Test 1 — Bielik (lokalnie, Ollama/CPU) | Test 2 — GPT (gpt-5-mini) | | --- | --- | --- | | Turn 1 cold (TTFT) | 45 501 ms | 5 257 ms | | Turn 2 hot (TTFT) | 2 833 ms | 3 112 ms | | Różnica czasu | −42 668 ms (szybciej o ~93,8%) | −2 145 ms (szybciej o ~40,8%) | | Hit 1 (cached\_tokens) | brak danych | 0 | | Hit 2 (cached\_tokens) | brak danych | 17 920 (99,4% prefiksu) | Both tests demonstrate a significant drop in TTFT when the prompt prefix is reused. With GPT-5-mini, we get explicit confirmation via the cached\_tokens metric (17,920 tokens read straight from cache). Ollama doesn't expose this metric directly, but the performance leap speaks for itself. The high initial latency for Bielik stems from CPU inference limitations—that massive 93.8% drop reflects how painfully slow an un-cached CPU Prefill phase is, rather than proof that local caching is inherently superior. ## Key Takeaways Although the low-level caching engine remains hidden, you can directly influence how effectively it works: - Minimize dynamic elements in static context zones (system prompts, project setups) to maximize cache hit rates. - Project configuration files (like CLAUDE.md) affect cache invalidation just as much as main prompt text. - Direct API calls grant much tighter control over cache break points than out-of-the-box agentic frameworks. - Design workflows around TTL limits so you don't lose cached states during long pauses between requests. Tags: - [Backend & Data](https://tech.univio.com/tag/backend-data) - [AI & ML](https://tech.univio.com/tag/ai-ml) Don’t forget to share this post! - - - ## Author ![Michał Pawlik](https://tech.univio.com/hs-fs/hubfs/IMG_4909.jpeg?width=125&height=125&name=IMG_4909.jpeg) [Michał Pawlik](https://tech.univio.com/author/micha%C5%82-pawlik) Software engineer and cloud architect specializing in C# and JavaScript, with a strong focus on leveraging AI tools to boost productivity. He also possesses in-depth expertise in Large Language Models (LLMs) and their associated system infrastructure. ## Check out other Blog Posts ![](https://tech.univio.com/hubfs/raw_assets/public/univio-tech-blog/images/icons/arrow-left.svg) - ![](https://tech.univio.com/hs-fs/hubfs/unity_220606_unity_team_of_teams_v1-kopia-2%20(1)-1.png?width=1320&height=770&name=unity_220606_unity_team_of_teams_v1-kopia-2%20(1)-1.png) ### [Performing SaaS application background tasks with AWS Batch and AWS Step Functions](https://tech.univio.com/platforming-saas-application-background-tasks?hsLang=en) Tomasz Dudek & Dawid Branicki Oct 6, 2020 - ![](https://tech.univio.com/hs-fs/hubfs/wadki_AI_v1-2.png?width=1320&height=770&name=wadki_AI_v1-2.png) ### [The Art of Choosing the Right Model. Why We Created the Commerce & Enterprise Benchmark (CEB)](https://tech.univio.com/the-art-of-choosting-the-right-model?hsLang=en) Łukasz Biegaj Oct 9, 2026 - ![](https://tech.univio.com/hs-fs/hubfs/unity_230509_unity_omnichannel_illustration_v3@2x%20(1)-1-1.png?width=1320&height=770&name=unity_230509_unity_omnichannel_illustration_v3@2x%20(1)-1-1.png) ### [Quality Assurance vs Quality Control](https://tech.univio.com/quality-assurance-vs-quality-control?hsLang=en) Tomasz Jański Feb 14, 2019 ![](https://tech.univio.com/hubfs/raw_assets/public/univio-tech-blog/images/icons/arrow-right.svg) [Follow us on Facebook](https://www.facebook.com/UnivioIT) [Follow us on LinkedIn](https://www.linkedin.com/company/univio-it-company/) [Follow us on Instagram](https://www.instagram.com/univio_it) [Check YT Channel](https://www.youtube.com/@univioITcompany) © 2026 Univio