Requirements testing for Critical requirements quality attributes

Requirements testing is a quick way to improve the quality of your software requirements – we have identified the key attributes that matter and have automated as much as we can.

Requirements Quality

Requirements quality is key to successful software projects.  On average 15% – 35% of all production defects are caused by poor or missing requirements.   If the requirements are low quality, there is little opportunity of achieving a successful result.  This article introduces automated requirements testing.

First we need to understand what makes a good quality requirement or set of requirements.  We need to understand the attributes of good quality requirements and then test our requirements against those attributes.

Many Agile software teams are aware of, and may use the INVEST pneumonic. INVEST is a vague and incomplete set of quality attributes for good software requirements.  Instead we recommend a more comprehensive and unambiguous set of quality attributes.

Requirements Quality Attributes

  • Clear – unambiguous functional intent
  • Concise – as few words and details as necessary to describe functional intent
  • Measurable – Has functional size (clear statements of data movement)
  • User-oriented – Each requirement has a clearly identified user
  • Design-free – no specification of how the functionality will be achieved
  • Valuable – each requirement should deliver value to the user/business
  • Testable – Can be tested for correct behaviour
  • Complete – includes all the functionality needed for a user interation
  • Consistent – use the same terms for objects and users throughout
  • Unique – no duplicate functional requirements in a set

These are our recommended 10 quality attributes for good software requirements.  The first 7 apply within the context of an individual requirement and the last three look at the quality of each requirement in the context of a set of requirements. No requirement (or user story) should be examined in isolation.

Automated Requirements Testing by ScopeMaster®

350+ Static Tests

Checking that your requirements are compliant with these attributes is rather tedious.  At ScopeMaster, we have made the work of checking requirements quality a breeze.  We have automated it.  ScopeMaster® tests each requirement, performing 350+ static tests covering good practice in writing requirements.  It’s rather like having the experts (Linda Westfall, Karl Wiegers and Robin Goldsmith) check your work as you go.  It assigns a quality score out of 100 and offers an explanation for how to fix each problem found.

100-2000+ Dynamic Tests

It then runs further tests as it cross-references all the requirements details to determine quality attributes relating to the set of requirements. 

In total, ScopeMaster typically performs about 1000 tests per requirement covering 9 out of the 10 categories above.

Requirements Testing is the Ultimate in Shift Left Testing

Requirement Quality Score

The ScopeMaster analyser determines a quality score for each individual requirement.  This is determined by the results of static tests and contextual tests. Each of the 350+ static tests has a weight associated with it.

requirements testing, recommendations and quality score by ScopeMaster®

Requirements Set Quality Score

ScopeMaster® automatically tests the set of requirements giving an overall quality score out of 100.

ScopeMaster assigns a score for a set of requirements quality

Defect Potential

Every software artefact has a predictable quantity of defects when it is created. When you think about it, this makes sense. For every hour a knowledge worker spends on something (code, design, requirements etc) they are likely to make a mistake or two each hour. Defect potentials in requirements are the defects that are likely to be in them when they are created.  The metric for requirements defect potential that we favour is a normalised metric of defect potential per COSMIC Function Point. For example, studies by Capers Jones shows that the average defect potential in requirements vary between 1 – 2 defects per FP in requirements. Knowing this is an important starting point for finding and fixing these defects – if we now know how many requirement defects we need to find we are part of the way to fixing them.

Defect Density

The defect density is an indication of the concentration of defects within the requirements. Whilst you could use an indicator such as defects found per requirement or defects per use case, we recommend defects per CFP as this is a standardised and consistent metric.

Defects Found

Comprehensive examination and testing of the requirements reveals:

  • potential defects within a requirement, and
  • potential defects across a set of requirements
Screenshot - results of automated requirements testing
Test Scenario - autogenerated by ScopeMaster, illustrated alternative flows

Other Techniques for Finding Requirements Defects

First we need to ensure that we know the actual requirements and then we need to be sure to articulate them well, below are lists of techniques for addressing both of these concerns:

Discovering the Real Requirements?
  • Workshops
  • Prototypes
  • Three amigos
  • Backlog refinement
  • Requirements Inspections
  • Simulations and walkthroughs
Articulating the Requirements Well
  • Textual analysis
  • Use case modelling
  • Data modelling
  • User flow modelling
  • CRUD analysis

All Features