Comprehensive requirements testing and quality assessment – the ultimate in shift left.

Requirements Quality Reporting

Requirements Testing

User Story Testing

Requirements testing reduces rework and waste. Automated (or manual) requirements testing can ensure that requirements quality is achieved early in the software development lifecycle. High quality requirements help teams become more successful on software projects. Let us explore why this is so. On average 15% – 35% of all production defects are caused by poor or missing requirements (Source Capers Jones and Accenture). If the requirements are low quality, there is little opportunity of achieving a successful result. This article introduces automated requirements testing.

First we need to understand what makes a good quality requirement or set of requirements. We need to understand the attributes of good quality requirements and then test our requirements against those attributes.

Many Agile software teams are aware of, and may use the INVEST pneumonic. INVEST is a vague and incomplete set of quality attributes for good software requirements.  Instead we recommend a more comprehensive and unambiguous set of quality attributes.

Requirements Quality Attributes

  • Clear – unambiguous functional intent
  • Concise – as few words and details as necessary to describe functional intent
  • Measurable – Has functional size (clear statements of data movement)
  • User-oriented – Each requirement has a clearly identified user
  • Design-free – no specification of how the functionality will be achieved
  • Valuable – each requirement should deliver value to the user/business
  • Testable – Can be tested for correct behaviour
  • Complete – includes all the functionality needed for a user interation
  • Consistent – use the same terms for objects and users throughout
  • Unique – no duplicate functional requirements in a set

These are our recommended 10 quality attributes for good software requirements.  The first 7 apply within the context of an individual requirement and the last three look at the quality of each requirement in the context of a set of requirements. No requirement (or user story) should be examined in isolation.

Software Requirements Testing by ScopeMaster®

If you are fed up with unclear requirements this is what you need, requirements static analysis:

Requirements testing for Critical requirements quality attributes

ScopeMaster performs both static and dynamic testing of requirements.  It will assesses the quality of individual requirements and sets of requirements.  ScopeMaster is the only requirements analyser that assesses your backlog.  ScopeMaster’s tests cover 9 of the 10 essential quality categories shown above.  Overall it will find (and help you fix) over 50% of the defects in your requirements.

Automated Requirements Testing by ScopeMaster®

350+ Static Requirements Tests

Checking that your requirements are compliant with these attributes is rather tedious.  At ScopeMaster, we have made the work of checking requirements quality a breeze.  We have automated it.  ScopeMaster® tests each requirement, performing 350+ static tests covering good practice in writing requirements.  It’s rather like having the experts (Linda Westfall, Karl Wiegers and Robin Goldsmith) check your work as you go.  It assigns a quality grade and offers an explanation for how to fix each problem found.

100-2000+ Dynamic Tests per Requirement

ScopeMaster then runs further tests to cross-reference all the requirements with each other to determine quality attributes relating to the set of requirements. 

In total, ScopeMaster typically performs about 1000 tests per requirement covering 9 out of the 10 categories above.

Requirements Testing is the Ultimate in Shift Left Testing

Requirement Quality Grade

The ScopeMaster analyser determines a quality grade for each individual requirement.  This is determined by the results of static tests and contextual tests. Each of the 350+ static tests has a weight associated with it.

requirements testing, recommendations and quality score by ScopeMaster®
Each Requirement is Tested

Quality attributes such as clarity, user-oriented, conciseness, user-oriented, testable and sizeable are all attributes that can be assessed when examining each individual requirement or user story.

Consistency, completeness and uniqueness can only be assessed when examining, cross-referencing and comparing sets of requirements or user stories.  ScopeMaster performs hundreds of static tests on each requirement (or user story)

Requirements Set Quality Grade

ScopeMaster® automatically tests the set of requirements giving an overall quality grade for the set.  This is a very useful indicator of project readiness for investment.

ScopeMaster assigns a score for a set of requirements quality
ScopeMaster assigns a score for a set of requirements quality
Sets of Requirements are Tested

Consistency, completeness and uniqueness can only be assessed when examining, cross-referencing and comparing sets of requirements or user stories.  For a given set of user stories (or requirements) ScopeMaster will perform up to several million contextual tests on sets of user stories.

Other Techniques for Finding Requirements Defects

First we need to ensure that we know the actual requirements and then we need to be sure to articulate them well, below are lists of techniques for addressing both of these concerns:

Discovering the Real Requirements?
  • Workshops
  • Prototypes
  • Three amigos
  • Backlog refinement
  • Requirements Inspections
  • Simulations and walkthroughs
Articulating the Requirements Well
  • Textual analysis
  • Use case modelling
  • Data modelling
  • User flow modelling
  • CRUD analysis

Defect Potential

Every software artefact has a predictable quantity of defects when it is created. When you think about it, this makes sense. For every hour a knowledge worker spends on something (code, design, requirements etc) they are likely to make a mistake or two each hour. Defect potentials in requirements are the defects that are likely to be in them when they are created.  The metric for requirements defect potential that we favour is a normalised metric of defect potential per COSMIC Function Point. For example, studies by Capers Jones shows that the average defect potential in requirements vary between 1 – 2 defects per FP in requirements. Knowing this is an important starting point for finding and fixing these defects – if we now know how many requirement defects we need to find we are part of the way to fixing them.

Tracking Quality

ScopeMaster Tracks Requirements Quality over time.

As you use ScopeMaster to help refine your user stories, it will track the defect counts and quality grade over time.  This is helpful for QA managers and project managers to assess requirements readiness for development.

Defects Found

Comprehensive examination and testing of the requirements reveals:

  • potential defects within a requirement, and
  • potential defects across a set of requirements
Screenshot - results of automated requirements testing

Defect Density

The defect density is an indication of the concentration of defects within the requirements. Whilst you could use an indicator such as defects found per requirement or defects per use case, we recommend defects per CFP as this is a standardised and consistent metric.