Univio - Tech blog

The Art of Choosing the Right Model. Why We Created the Commerce & Enterprise Benchmark (CEB)

lukasz.biegaj@univio.com (Łukasz Biegaj) — Fri, 09 Oct 2026 13:36:53 GMT

Choosing an AI model for production systems rarely is rarely just a matter of down to a simple comparison of charts on public leaderboards. Vendor marketing promises and academic tests rarely answer the key questions: how a given LLM will handle our specific tasks, how much it will cost to execute them, and what the quality of the output will be. Evaluation only becomes valuable when it reflects the actual environment in which the system is supposed to operate, tackling problems and datasets aligned with our needs.

The Commerce & Enterprise Benchmark (CEB for short) addresses these challenges in a single evaluation suite. It provides a reproducible way to test the capabilities and limitations of AI systems using tasks relevant to Commerce & Enterprise solutions.

Why did we create our own AI benchmark?

CEB is a tool we created for our own needs to reproducibly test the real-world capabilities of models, agents, and evaluation harnesses in our daily engineering work at Univio. Today, we are opening it up and sharing our insights publicly.

A large portion of existing benchmarks relies on synthetic data and public datasets. Unsurprisingly, models are trained on this exact same data. In many cases, models rank high on leaderboards simply because they memorized the questions during training (i.e., data contamination, meaning the leakage of test data into training sets). As a result, public benchmarks often measure a model's memory rather than its reasoning capabilities.

A stellar score in SWE-bench, MMLU, or GSM8K does not translate to whether the model will correctly decline a Polish surname, work efficiently with Jira and HubSpot, or maintain document consistency after tens of edits.

Public leaderboards and CEB operate in entirely different domains. The former frequently measure universal academic competencies. We measure the models' effectiveness in solving problems encountered in e-commerce and enterprise organisations.

What is CEB?

It is a closed set of tasks selected from real-world problems encountered by Univio employees and our partners. The test architecture is based on Inspect AI from the UK AI Safety Institute.

Coding tasks are run in isolated sandboxes.

What exactly do we evaluate?

We apply a range of evaluation dimensions:

Linguistic tasks: declension, spelling, punctuation, and vocabulary comprehension, verified independently for Polish, English, and German.
Long context and editing consistency: multimodal operations and handling tasks that exceed the allowable context window size.
Working with data, business tools, and technical integrations: e.g., Jira, Confluence, HubSpot, Excel.
Working with structured data and data contracts.
Fact validation: knowledge tests identifying the model's knowledge cutoff and specialized domain expertise.
Coding and automation: currently analyzed in JS, Python, and PHP.

Thanks to the private nature of the benchmark, our test data does not end up in public training corpora. We do not use synthetic data; we rely on real problems that have been anonymized and encapsulated into test cases.

In addition to scoring the results, we also calculate the number of tokens used and their cost. We record the execution time of the test, but we do not factor it into the rankings—we don't consider it a reliable metric when inference is handled over a network.

Our goal is an objective measurement of a model's true competencies in commercial applications.

What does CEB not do?

We do not treat our benchmark as a universal ranking for every use case. We are also aware of its current limitations; for example, the task database for the German language is currently too small to draw conclusive results, but we are constantly expanding it.

We are also not looking for a perfect model for everything. We are looking for optimal solutions for specific use cases and budgets.

We regularly test AI model performance across various use cases and sectors. Curious how the ones you use at work are doing? We’d gladly compare our findings with your industry and needs. Contact us!

Leveraging KV Cache for Efficient Programming with LLMs

Michał Pawlik — Fri, 09 Oct 2026 13:16:47 GMT

Caching is a concept every developer knows well: we store the result of an operation so we don't have to recompute it on the next request. A similar mechanism works with equal success in large language models. In the LLM world, the counterpart to cache memory is KV Cache (Key-Value Cache). While it operates under the hood, understanding it helps you run models faster, more efficiently, and at a significantly lower cost.

How Does an LLM Remember Context?

Let's start with a bit of theory. When a query hits a Transformer-based LLM, it passes through two main phases (assuming the model runs on a GPU):

Phase I: Prefill.
This involves processing the entire input prompt using the full compute power of the GPU. During this phase, Key-Value (KV) states are calculated and saved in memory for every token. Put simply: K (Key) answers what a given token represents, while V (Value) carries what content it holds. The memory used here is the GPU's VRAM. Processing in this phase happens in parallel.

Phase II: Decoding.
This is the stage where the model generates its response token by token. As it generates each new token, it reaches into the previously allocated VRAM to read back the existing KV pairs. This reveals a clear difference in execution profile: the prefill phase is heavily GPU-bound, whereas the decoding phase is memory-bound.

To visualize this read/write flow, take a look at the concept below:

The diagram above illustrates a simplified conversation flow with a model. In turn 1, the model must process all input tokens during the prefill phase and write them into the KV Cache. In turn 2, it begins generation right from token #5, fetching the prior conversation (including its own previous outputs) directly from cache memory.

Is Caching Available in Every Model?

KV Cache is a foundational component of every Transformer-based LLM system. While you can influence how it behaves, your degree of control depends on how you interact with the model:

Directly via API: when manually constructing the system prompt, tool definitions, and user prompts.
Through ready-made agentic frameworks: such as Claude Code.

Different model providers offer varying levels of control. Anthropic is a great example: when querying their models directly via API, you can pass a cache_control parameter to explicitly mark which blocks of context should be cached. This provides far more granular control than relying on high-level agentic wrappers.

Putting KV Cache Knowledge into Practice

Let's look at a practical example using Claude Code and custom AI harness development. The context window passed on every request typically consists of three main parts:

System prompt – primary instructions and tool definitions.
Project context – files like CLAUDE.md and project-specific guidelines.
Conversation history – the most dynamic section.

Each part has a distinct lifecycle. In an ideal world, everything would remain static, resulting in a 100% cache hit rate on every request. In reality, that's impossible—conversation history naturally grows with every turn.

That is why keeping the system and project layers as static as possible is crucial. Avoid embedding dynamic data into these early sections—such as real-time timestamps or dynamically generated directory trees. Any change in the prefix automatically invalidates the entire downstream cache.

Beyond prompt edits, specific environment actions will also clear your cache:

Changing models or adjusting the reasoning depth (Effort level).
Adding or removing MCP servers (which mutates tool definitions in the system prompt).
Toggling plugins on or off.
Executing commands like /clear or /compact (which reload project rules).
Uploading large attachments (images, PDFs) that exceed provider caching thresholds.

It's also worth noting how caching works across multi-session setups on a single machine. Sessions running in the same working directory share the same cache. If you spin up sub-agents, each sub-agent maintains its own isolated cache—it doesn't pollute or automatically inherit from the parent agent. Only explicit commands like /fork will copy the parent's cache state over to a new sub-agent.

Finally, cache has a shelf life. Extended periods of inactivity mean the provider eventually drops it from memory. This window is governed by a TTL (Time To Live) parameter, which usually ranges from 5 minutes to 1 hour depending on the provider and tier.

How KV Cache Impacts Token Costs

Your context memory strategy directly impacts your API bills. The rate for an initial call (a full prefill with zero cache hits) is higher than subsequent requests that hit the cache.

Looking at the pricing structure above, writing to the cache (Prompt Caching / Write) incurs a slight premium over standard base tokens. However, the real savings hit when a query lands a cache hit reducing the cost to a fraction of the base rate ($0.25). The 5-minute and 1-hour write options simply correspond to different TTL retention settings.

Benchmarks: KV Cache in Action

To verify the theory in practice, I ran a benchmark measuring response times on repeated requests across two setup types:

Bielik-4.5B – running locally via Ollama on CPU.
GPT-5-mini – queried via an enterprise AI Gateway.

Both tests track TTFT (Time To First Token) using a fixed prefix containing a question. The prefix length must exceed 1,024 tokens; below this threshold, caching overhead outweighs performance gains.

Parametr	Test 1 — Bielik (lokalnie, Ollama/CPU)	Test 2 — GPT (gpt-5-mini)
Turn 1 cold (TTFT)	45 501 ms	5 257 ms
Turn 2 hot (TTFT)	2 833 ms	3 112 ms
Różnica czasu	−42 668 ms (szybciej o ~93,8%)	−2 145 ms (szybciej o ~40,8%)
Hit 1 (cached_tokens)	brak danych	0
Hit 2 (cached_tokens)	brak danych	17 920 (99,4% prefiksu)

Both tests demonstrate a significant drop in TTFT when the prompt prefix is reused.

With GPT-5-mini, we get explicit confirmation via the cached_tokens metric (17,920 tokens read straight from cache). Ollama doesn't expose this metric directly, but the performance leap speaks for itself. The high initial latency for Bielik stems from CPU inference limitations—that massive 93.8% drop reflects how painfully slow an un-cached CPU Prefill phase is, rather than proof that local caching is inherently superior.

Key Takeaways

Although the low-level caching engine remains hidden, you can directly influence how effectively it works:

Minimize dynamic elements in static context zones (system prompts, project setups) to maximize cache hit rates.
Project configuration files (like CLAUDE.md) affect cache invalidation just as much as main prompt text.
Direct API calls grant much tighter control over cache break points than out-of-the-box agentic frameworks.
Design workflows around TTL limits so you don't lose cached states during long pauses between requests.

Performing SaaS application background tasks with AWS Batch and AWS Step Functions

Tomasz Dudek — Tue, 06 Oct 2020 13:16:50 GMT

Even the tiniest web application has to perform a lot of background tasks, be it image resizing, product/stock imports and report or sitemap generation. In addition, if your product is distributed in SaaS model, then most of them have to be performed for each of your customers.

That, depending on how popular your product is, may in turn require tons of computing power. You will have to find a way to quickly add and maintain all kinds of jobs, without adding too much technical debt to the project.

Oh, and without blowing your application up whenever a heavy job is running.

At first, it may be tempting to just let the web application be their executor and scheduler. Add an @Scheduled annotation, if you live in Java’s world, or a node-cron and voila – your application has a background tasks mechanism.

It works fine… but it also causes some fundamental problems. As soon as you run a second instance of your application, you have to synchronize your state between apps, otherwise you schedule the same task twice. Even worse – any fluctuations of resource usage (a “busy” task or a sudden spike in requests) can heavily impact the performance of the application. Your service could end up unreachable for end users, because a very CPU heavy task has consumed all the resources. This is nothing short of a nightmare, especially in the SaaS world with “noisy neighbors”.

Of course, you may add a synchronization mechanism between the tasks (and also code all the failure handling, retries or internal queues), schedule them to process only at night and so on. This is often the fastest method, but it leaves you with huge technical debt and does not scale too well.

Can you do better? Well, you may split your app into an app-web and app-worker and detach the latter from handling the web requests. Various frameworks like Django support this, although it requires running queues like Celery or RabbitMQ. Now, the failures and memory errors of your tasks no longer affect end users. Surprise traffic spikes don’t suddenly blow up your background task. This is way better, but still requires you to manage and scale the worker cluster on demand. There is also a lot of shared code between the app-web and app-worker, which could both confuse developers and further grow the technical debt.

Can you do (even) better? Well, why not let the web applications do the web part only? Let the worker applications be truly workers? Bind your tasks into lightweight, abstract containers that are ran on-demand, when necessary, with all the metadata collected from the outside. This is a common pattern in the data engineering/science world, where a huge DAG (directed acyclic graph) of various connected tasks is scheduled and executed on a daily basis. Using the right tools, you don’t need to be a data engineer to bring such quality into our web applications – you just need to use the cloud to its full extent.

To achieve this, two major pieces are needed:

First, an easy to use container environment, which automatically provides compute resources and runs containerized jobs.
Second, something to orchestrate the entire process. Thankfully, AWS has got us covered, specifically with the two technologies mentioned in the article’s title (a surprise huh?).

AWS Batch is, as the name suggests, a batch processing service. It is able to run hundreds of containers, providing as many compute resources as necessary. With its built-in queue all you need to do is define and trigger a job. AWS Batch will handle the execution part for you.

AWS Step Functions is an orchestrating service, based on a state machine. We define what should happen and in what order. The “what” could be a Lambda Function, a Fargate container, an AWS Glue job or an AWS Batch job (among few others). Step Functions (SFN) trigger all necessary “whats”, pass data between them, are able to run tasks in parallel, have some branching if/else logic and can be scheduled as needed.

The know-how

The components

The architecture below is generic and works in most of our tasks. You will be able to abstract the logic concerning your particular task and reuse literally 90% of your Infrastructure as Code (IaC) whenever you encounter a similar problem. As always, CDK or Terraform is great and you should definitely use it.

The general idea is:

Chunk the job into smaller, parallelizable parts (this is a piece of cake in SaaS applications – you could just run the same job for each tenant)
Run each of them in a separate container
Have the entire process scheduled, if necessary

Metadata Lambda

As a starting point to our workflow, we have a Lambda Function for deciding how many tasks AWS Batch needs to run. This entirely depends on your use case – if the task is to resize images then “a portion of the job” could be an array with a thousand ids. Each container would get thousands of photos to process – all unique to that respective container, too. If a sitemap is to be generated or products are to be imported, then one task could handle exactly one tenant.

This is also a good place to handle any special logic – maybe some of your tenants require more parameters? What if a few of them should be ignored? Was the last execution for a given tenant successful and further computations are unnecessary for now? Answer similar questions yourself and code the logic accordingly. Do also note that this could be a container instead of a Lambda if 15 minutes of execution time is not enough to decide how big your job should be.

The metadata Lambda generates n sets of parameters (each set corresponds to “a portion of the job”) and saves them to an S3 bucket. We could also let the Step Functions pass the output from Lambda directly to AWS Batch (with no S3 involved), but there are various limits on how big (in kilobytes) the Step Function’s state can get. Our solution is more future-proof.

Batch Job and the Container

Batch Job Definition is rather straightforward – all you need to specify is how much vCPU or memory will be used by a single ECS Task, as well as the its specific Docker image. The type of the job we are using here is called Array Job – n parallel independent containers are to be run on the Compute Environment.

The container is a tiny application and its sole purpose is to run the parameterized task it was designed to perform, exiting after the work is done. Consequently, the app often resembles more of a script-ish file than a fully-fledged application.

Furthermore, the code inside has to be as generic and reusable as possible – no hardcoding allowed! The first step is to read the S3 Bucket file using the AWS_BATCH_JOB_ARRAY_INDEX environment variable to get exact parameters – each container instance has its own unique number. That file, created by Lambda, should contain all the information required by the task – such as the tenant name, tenant bucket name and a thousand image IDs to process.

Batch Compute Environment and Job Queue

After the Batch Job is scheduled, the tasks are put into the Job Queue, where they wait be executed on a Compute Environment. These are pretty much ECS Clusters mixed with ASG/Spot Fleet and come in different colors and sizes types – you can either use your own ECS Cluster or let Batch manage the ECS Cluster for you. We have been successfully using Batch-managed clusters with Spot Instances to save some pennies. You only need to specify the minimum and maximum number of vCPUs. If you set the minimum to 0, the cluster will automatically scale its resources to nothing if no jobs are running. Pay as you go!

You can also indirectly define the maximum parallelization of jobs here – if a container requires 4 vCPU, your cluster is using a maximum of 2 EC2 instances with 4 vCPU each, then at any given moment, at most, 2 tasks will be running. The rest of them will gently wait in the Job Queue to be picked up and processed at a later time. Similar reasoning can be applied with Memory. This is a great way to ensure you don’t kill your database or other resources.

It might be worth noting that a Job Queue can have multiple Compute Environments assigned. You may try running your job on spot instances first (using arbitrary Compute Environment A), but if no resources are available fallback to on demand ones (using Compute Environment B). You can also specify multiple EC2 instance types a Compute Environment is allowed to run. Depending on the workload AWS Batch will try to acquire an instance suited to the current demand in the Job Queue.

Step Functions workflow

Step Function governs the entire process, gluing other components together. With its rich integrations with other AWS Services you are able to trigger a Lambda function (or a Fargate container) and schedule a Batch Job. You define the entire workflow and its order of execution here, in the State Machine. Should the workflow fail at some point, you could also define an error handler. Our example is rather simple, but you might for example want to store all failures and successes in a database for historic reasons, perform more tasks after the Array Job is finished, send an email to each tenant or trigger analytic jobs in AWS Glue. Possibilities are nearly endless.

Other Batch functionalities

We have shown a rather simple workflow – an array of parallel, independent jobs. Batch itself provides more capabilities. Jobs can be dependent of each other and pass parameters within AWS Batch itself.

N to N jobs

In a common scenario where a lot of data for each of our tenants has to be imported, we might be interested in splitting the job into even smaller chunks. Import is not only about getting metadata from external resources and saving it in to database. The process often contains many smaller steps, such as data validating, fetching additional attachments (documents or images) or transforming to required model data or format in a specific order. If we have to import images, we generally want to scale them, change the format and perform a few other operations. These seemingly simple operations can consume a lot of resources in our system (and bring headaches to the Site Reliability Engineering (SRE) team if it was performed in the web application serving the requests).

Thankfully, AWS Batch allows us to split our long process into smaller chunks. Instead of one long running container performing all the operations, we can process data in multiple, smaller containers with their responsibilities narrowed down. The first container pulls the data into our system, the second one fetches the images and the third rescales them. If rescaling fails at some points, we do not need to redownload the images again – we can simply retry the container with the rescaling part. This also makes the development and testing of such little applications easier.

Using a Batch Array N to N Job, we are able to define a linked-list of dependencies. Job A has to finish first, passing the data to Job B. After that one finishes, Job C is started. The entire process is also an Array Job, so it can be performed for each tenant independently.

Note that it may be tempting to use Step Functions instead and design that kind of workload as follows:

However, there is a subtle difference – if you define it as N to N, then each sequence of jobs runs separately. One of your tenants with a little data can finish the entire sequence quickly, while the others are still at the “Job A” stage. If you use Step Functions instead, all “Job A” tasks must finish before any “Job B” starts. Unless you need it to function that way, you’re often better off with N to N because the workload finishes faster and parallelizes slightly better.

Multi-Node Parallel Jobs

Another interesting capability is the multi-node parallel job, which allows you to run a single job on multiple EC2 instances. This differs from a standard job, which occupies only a single EC2 instance. Multi-node jobs are used for large-scale, high-performance computing applications and distributed GPU model training. This is fully compatible with most machine learning frameworks supporting IP-based, internode communication, such as Apache MXNet, TensorFlow, or Caffe2.

If you want to use multi-node parallel jobs on AWS Batch, your application must be prepared to run on multi-node infrastructure, so the code must use frameworks suitable for distributed communication.

Each multi-node parallel job consists of the main node, which is launched at the beginning. After that node is up, the child nodes are launched and started. Forget about compute environments that use Spot Instances, though – multi-node parallel jobs don’t support them. Regardless, you should look into AWS Batch if you’re looking for easy to use and disposable data science compute environments.

The quirks we have noticed

Batch gives little to no insight from its UI

Unfortunately, we haven’t been able to extract much information from the UI. If something blows up, you are forced to juggle between Batch, ECS, CloudWatch and Step Functions to find out what exactly happened and why. The community has so far made projects like batchiepatchie to mitigate the issue and provide a different AWS Batch UI. No moves or official plans from AWS, however.

Batch reacts at a snail’s pace

You might quickly become annoyed by how slow Batch reacts to events. Everything there seems eventually consistent:

You schedule a job and you wait until it gets picked up and put in the Job Queue.
Then you wait until the Compute Environment notices there are tasks in the Queue – but not enough vCPUs to handle it.
Then you wait until it adjusts the desired capacity.
Then you wait until Spot Fleet/ASG actually gets the signal to scale up.
A few more “then you wait“ moments later… the task is finally running.

Nothing here seems instant – which of course isn’t much of a problem in background long-running tasks, but might be irritating, especially while developing the solution. The feedback loop is unrushed.

Step Functions are a challenge to debug (and develop)

Well, the definition of a State Machine is just a horrendous JSON file. When you make a mistake there (or even better – mix it with String interpolation from Terraform), then no IDE will save you. Even the built-in editor in AWS Console fails to provide meaningful error messages. Tests? Forget about it.

Also, no IDE helps you with the input/output processing syntax of SFN. All those $ and .$ take some time to get used to.

A great and useful rant about Step Functions can be read here on Reddit – so just know that you’re not alone on this!

Launch Configuration of managed Compute Environment with Spot Instances

There is some magic happening behind the scenes whenever you create Managed Compute Environments with Spot Instances. During creation, you may specify a Launch Template to be used by the EC2s. If Spot Instances are also selected, Batch creates a Spot Fleet Request on your behalf (you have attached AmazonEC2SpotFleetTaggingRole, have you?).

Because Spot Fleet Request requires Launch Templates as well, the one from Compute Environment gets copied. But then, any changes applied on the Compute Environment’s Launch Template are not propagated to the Spot Fleet Request’s Launch Template. Our IaC tool of choice (Terraform) was not able to help us there and we spent some time debugging and asking “why doesn’t the cloud-init script change even though the Launch Template does?”.

Content-Type of Launch Template

This is of course written somewhere in the docs – make sure any Launch Template is a MIME multi-part file (this is shown just below)!

Authentication in a private repository

ECS is able to pull images from private registries using a handy repositoryCredentials in task definition. Unfortunately, Batch does not support that parameter. If you want to use a protected, private repository you have to ensure the EC2 on which the ECS Agent run has configured ECS_ENGINE_AUTH_DATA. You may either bake that into the AMI or add a script similar to this one in the Launch Template of the Compute Environment. This one pulls credentials from AWS Secrets Manager – make sure your instance will have permission to perform that API call!

The data engineering struggles

This is our current go-to template for batch processing and performing background tasks in one of the SaaS projects we maintain and develop.

Can you (still) do better? Well, the real question at this point is would you use Step Functions as a fully-fledged workflow orchestrator with Batch as an execution engine? Are they good enough for data engineering projects?

As a starting point, in the MVP phase of an arbitrary project or product – without a doubt. Step Functions are easy to pick up, have enough integrations with other AWS services (like Glue or Batch – very handy in the data world) and work really well in not-so-complicated environments. Issues arise when connections and dependencies between tasks are no longer not-so-complicated.

The idea behind Step Functions is a state machine – we observe the current state of the flow and decide what should be done next. Basically, a Step Function is an overcomplicated if/else machine. In data engineering, it is way more convenient to think in Directed Acyclic Graphs (DAGs) – do the first task, then simultaneously those two, combine the results in the next task and so on. Unfortunately, this type of thinking is not supported by SFN and there is no other AWS product which could be used instead.

What about the other types of Batch Jobs we have mentioned? Couldn’t they simulate the DAG? Well, if you can fit into their requirements, they could work for a while. Sooner or later though, your branching logic will get more complicated and Batch will no longer be enough. Fighting with a framework (either SFN or Batch) won’t be easy.

There are also several red flags – when one of your SFN tasks blows up, you can’t just patch it and pick up where you left off. The entire job is marked as a failure and you have to restart the entire graph. This (along with a not-so-pleasant debugging/testing experience and parallelization/branching limits) automatically rules SFN out of data engineering projects, where computations often take hours to complete. There are of course retry mechanisms but they are not enough. We would recommend using Apache Airflow or a similar tool as a DAG scheduler for your data engineering needs. You may run it on AWS EKS, if you wish, and fully integrate with your other AWS tools.

The judgement

Thankfully, the majority of web applications’ background tasks aren’t that complicated and don’t depend too much on other jobs. Resizing images, generating sitemaps or reports all can be done independently. In the world of SaaS, they can also be parallelized for each tenant separately. This makes a great use case for SFN and Batch – as shown above.

Is it the best solution on the market? Like most things in IT, the answer depends on the scale and requirements of your project. You could use the standard periodic job provided by the framework your web application uses. The implementation will probably be swift, with no interference in the project’s infrastructure.

However, looking at the tradeoffs, this will become dangerous and overload your system, especially if your business model is SaaS. A slight increase in number of tenants would directly “contribute” to the application’s performance and technical debt. The last thing your business needs is an application blowing up because one of your customers has a lot of images to process.

With various AWS technologies already buried deep into the code and the product getting traction, the decision was trivial for us. The combination of CloudWatch, Step Functions, Batch, Lambda and S3 allows us to process large amounts of data with no impact on our system. If you are in similar place right now, you should definitely give it a shot.

Quality Assurance vs Quality Control

Tomasz Jański — Thu, 14 Feb 2019 13:16:40 GMT

So much has been written about Quality Assurance that we should ask ourselves if we haven’t lost our understanding of this term. I can’t resist the assumption that each IT project is said to have an elite Quality Assurance unit, composed of software testers only, responsible for ensuring software quality in the project. In addition, HR units only add to this impression when they search for: “QA Specialists”, offering them jobs that focus on reporting tests and reporting defects.

Differences between the Quality Assurance and Quality Control processes

What’s the source of these differences? First and foremost, Quality Assurance covers all activities carried out under a project. This starts with cooperation with the client (transparent communication), defining the roles and responsibilities in the team (defining the modus operandi, communication, defining the software development process), and ends with the definition of requirements and their life cycle.

The process includes all project participants, giving them the opportunity to influence the shape and definition of quality. This is based on the project’s context and its business value. We should remember about the possibility of measuring and assessing the quality of software at the same time.

The QA process is a preventive measure oriented largely toward the entire production process, both in the company and in the project at hand.

Now, Quality Control focuses largely on the search for errors/ irregularities in the functioning of the application by testing the software at various test levels and using various test types. These are activities that verify and validate the product in terms of compliance with the user’s requirements and expectations. Much of the responsibility lies with the software tester. This is a detective activity.

The illustration below shows the proportion between the activities around software testing and the entire quality assurance process.

A different look at the Quality Assurance process

This chart can be used as a “StarterPack” for understanding the Quality Assurance process and the role of the tester. It won’t fit every situation, but it can help you see quality assurance through the prism of increased activity in the project.

Let’s start from working with requirements. (Find out more about the term “3 Amigos” here).

In the context of software quality, we must bear in mind the beginning of a close cooperation between the 3 Amigos. We have three roles here as presented below:

1. Business analyst –this role may vary from one case to another, but this must surely be a person from the business, e.g. Product Owner.
2. Programmer.
3. Software tester.

In the model described, the three people fulfilling the above functions work together to define the Requirements (Story). This is an important moment when each party presents its point of view and how it understands individual functionalities (a confrontation of visions about the Requirements). A quick feedback loop and mutual understanding appear at this stage. “We don’t talk after implementing the functionality, we talk before that”. Such actions have a major preventive effect on the project and often reveal many risks that haven’t materialized yet, so the cost of their mitigation might be lower.

Example questions, e.g. from the perspective of a software tester:

- What does the client need this functionality for?
- What are the mock-ups, if there are any? 🙂
- Does the new requirement affect other components in the application?

Understandably, each party will have a different set of questions. Consequently, there should be a common understanding as to the product/ value to be delivered.

Once we know the expected deliverables and outputs, we move smoothly to the stage of implementation of the requirements. At this stage, we can use many methods to improve the software quality, ranging from good practices in relation to design, through to application writing and a set of test-oriented activities:

- unit test coverage of a % of the code,

An interesting part of this stage of work is the inclusion of the pyramid of automated tests, which might slightly influence the shape and scope of our tests.

The next steps in the process are the activities that are usually assigned to broadly understood software testers. This is a set of activities related to software verification/ validation, in terms of the above requirements, as well as areas related to quality characteristics (e.g. performance and security).

What’s next?

Quality Assurance is more than just software testing. Keep that in the back of your head and share this responsibility with the whole team. By inviting and engaging your team members in defining activities around software, make sure that your clients can subscribe to Gucci’s motto: “Quality is remembered long after the price is forgotten”.

If you want to learn more about quality management in IT, you should take a look at our publications on agile and software development practices.