Here are some insights into the value of finding requirements defects  before design or coding.

To a software development team it is very valuable to find defects early.  Even better still, is to prevent defects altogether.  Excellent requirements are an effective form of defect prevention and rework prevention.   An ambiguous requirement can lead to substantial wasted effort.

Defects Arise before Coding Starts

Software requirements refinement - finding requirements defects earlyDefects arise in requirements and architecture before coding even starts.  Let us imagine we are in an insurance business, and the business folks have decided that a new (or changed) system is needed.  They set out to describe what they want and pass it on to IT to estimate.  Such requirements are often  written as user stories, typically this is done in a consistent format:  “As a “….”I need to “…. “so that” …

As soon as pen is put to paper, ambiguities and omissions appear, it is inevitable.   These mistakes in written requirements (if left undiscovered) can lead to costly rework later on in the project.  The cost of such defects is non-trivial,  the UK is set to lose £37Bn on failed (Agile) IT projects (reference).  To avoid these avoidable costs, we need to ensure that the requirements are as unambiguous and as complete as possible.  Even the most skilled requirements authors will make unintentional mistakes, getting them right is not so easy.  This is where automation can now help.

The cost of poor quality software requirements in generating rework

Benefits of Finding Requirements Defects Early

Valuing an early requirements defect discovery is neither straightforward nor definitive.  The main reason is that it depends on how late (in the development process) those defects might have otherwise been found.  If a defect is not found until a much later stage, it will take significantly more effort to repair, costing thousands, if it just found one stage of the SDLC sooner, then perhaps a few hours of effort can been saved.   On average we estimate a modest 4 staff hours saved for every requirements defect discovered early – approximately $200.

Typical effort of finding and fixing and verifying the fix of:

a requirements defect found during the requirements stage:  20 minutes – 4 hours

a requirements defect found during the design stage:   1-10 hours

a requirements defect found during the development stage:  2 – 15 hours

a requirements defect found during the system test stage:  4 hours upward

a requirements defect found during the production stage:  8 hours upward

Note this only represents the additional work to resolve the problem, it does not address the consequences of the defect.

Cost of Requirements Defects found in Production

When a requirements defect is found in production it is usually reported by a user, and we must consider:

  • How many users were affected before it was reported to us.
  • How many more users will be affected until it can be fixed.
  • What are the costs of providing a workaround to those who have been affected.
  • What is the consequential loss of this defect on customers and our organisation.
  • The knock-on affect (opportunity cost) of diverting attention to this problem vs other opportunities.

The sum of these costs usually outweigh the cost to repair the fix, by a significant margin.  In some extreme cases, businesses survival can be threatened by a single defect in a major customer facing application.  e.g. a banking system that lost track of account balances

Security Defects – a special case

Security defects not found until production, will always be expensive to fix, can often involve legal claims against the organisation and are typically measured in millions of dollars.  Finding security defects early (especially in the requirements phase) can therefore potentially avoid incidents that might cost millions of dollars.

Value of ScopeMaster Finding Requirements Defects

Let us turn to the value provided by ScopeMaster in exposing requirements defects early.  There are in fact, multiple benefits of using ScopeMaster on your requirements:

  1. Very fast way to quality assure requirements
  2. Finds (some) potential missing requirements – automatically.  (Remember: a missing requirement is a defect).
  3. By improving requirements, there will be fewer defects elsewhere, schedules and development and testing costs will reduce.
  4. Improving requirements clarity, reduces potential discord amongst team members caused by ambiguous requirements.
  5. Generates an automated functional size metric without manual effort, which can be used to give more control of the project.
  6. Improves the speed of functional size measurement by 200-400% (for those projects that do measure functional size)
  7. Changes the language used by requirements authors such that over time will prevent requirements defects from being introduced.

These are not all of equal value, and their relative importance will vary from project to project.  In many cases, perhaps the last point is the most important.  If you can prevent defects by specifying requirements more precisely and completely, then there will be less defects to be looking for later.

Whilst the potential benefits of using ScopeMaster on a project could be in the millions,  a modest $200 / defect is very conservative estimate of the value of exposing the defect early.

Advice recently shared with a client:
“During the requirements phase, finding defects takes more time than fixing them, combined we usually allow 1 hour to find and fix (or 4-10 minutes with ScopeMaster).
If the requirements defects are found during system test, finding, fixing and verifying are ALL very time-consuming. (4-8 hours total would be a fair range).  So expect each defect to cost least 4x more costly to fix later.  This project is not like building an app where the devs can take a high-level conversation about user need and then guess the details; these requirements describing who get’s paid what and when need to be specific and correct.
Effort to find and fix early
  • Finding requirements defects in the requirements phase is normally done with the 3-4 amigos discussing them; thereafter it’s a few minutes to fix.  15 minutes of amigos discussion is 1 hour of staff time. (Many of these could be found and fixed in 4-10 staff minutes with ScopeMaster.)

Finding and fixing requirements defects in system testing

  • Finding the requirements defects in the system test phase may still only take 1 hour but fixing will take considerably longer because:
  • When a (requirements defect) is found in system test, it must be copiously documented, then requires extraordinary coordination of SME’s testers and devs to triage and verify as an actual requirements defect, then finally the requirement corrected.
  • The implementation of the fix may be modest configuration work but the new testing layers around this defect, and all related functionality that must also regression tested again is non-trivial.
  • Late finding of a requirements defect means that there is a higher probability of bad fixes.  ie. one defect causes one or two more (each with it’s own documentation, triage and fix-verify cycle).
  • There is a human tendency not to question the requirements at later stages.  Testers tend to accept the requirements as correct and they are more likely to challenge coding/config compliance with the requirements,  not the requirements themselves.  The consequence is that defects (caused by poor requirements) are more likely to end up in production, with very bad consequences.”