Archive for July, 2009

Who needs acceptance criteria?

July 9, 2009

I meet a lot of testers (collective noun for test managers, test coordinators, test engineers, and test process improvers), so it’s inevitable that I regularly end up in a discussion about requirements versus acceptance criteria. A long time ago I held the view that you didn’t need acceptance criteria. The reasoning behind this was something like:
– You should write clear requirements (many people talk about SMART requirements).
– If your requirements are clear, you don’t need acceptance criteria.
– If your requirements are not clear, you haven’t done a good job of writing requirements.

Many people pointed out that it is hard to write clear requirements, and many testers therefor suggested that you always need acceptance criteria. At the very least you could use them as an intermediate step to get from vague requirements to clear requirements. While I accept that acceptance criteria can be a helpful aid in improving the quality of requirements, I do not think they are always needed. Particularly for function requirements (or “functional requirements” if you insist), a good requirements process, requirements structure, requirements checklist and the necessary skills should do to get clear requirements (I prefer not to use SMART, but that is perhaps a topic for a different post).

I am indebted to some of my requirements colleagues for helping me understand that there is a need for acceptance criteria in relation to certain types of requirements. I have not yet got a good definition which types of requirements – lets say quality requirements for now, but I think it is only a subset of quality requirements for which it is really needed. Let me explain with an example.

One quality aspect which is important to many IT systems is availability. This is often expressed in “up-time”, as in the percentage of time in which the system is “up”, i.e. available. A requirement such as “Availability of 99,8%.” is not precise enough on its own. You must be clear on what must be available – this can be done by tracing to the functions to which this applies. In addition, you must specify to whom it must be available. Also important is to define what the 100% is, to understand what 99,8% is. That is, you must specify if e.g. “scheduled down-time” is included or not and you must identify the “sample frequency”, i.e. how often you measure and check. All this can be done, and you would then have a clear requirement. Congratulations!

But wait! Even though the requirement may be clear, there is still a problem. To validate this requirement you may have to test the system for a very long time. This depends on the definitions, but usually availability is related to time periods of months or years and thorough validation would require testing it for multiples of this time period. Rarely is the customer prepared to wait that long, and this is where acceptance criteria come into play. An acceptance criteria allows you to specify less stringent conditions for acceptance without changing the requirement itself. So the customer could agree to accept the system if during 2 weeks of availability testing the system has an availability of 99,99%, for example.

To me that is the essence of acceptance criteria: they allow you to specify alternate requirements (less or more stringent) for the sole purpose of accepting the delivered system. The “proper” requirements remain valid when the system is in operation.

As you can probably tell, my thinking on acceptance criteria is still developing. I would really appreciate any thoughts, additions and challenges you have on this topic.