Software measurement is important because it is hard to manage what you don’t measure. Software work is knowledge work. It is one of the costliest endeavours undertaken by mankind, with more than 50 million people across the world working in highly paid software roles, and for most of them, their efforts are unmeasured. Many people don’t realise that software size can be measured. Most other forms of human labour have standardised measures of size and productivity. But with software work, although suitable metrics exist, they are seldom used. Industry leaders, educators, executives and even governments have so far been relaxed about this phenomenon, but with the increasing importance of software on corporate survival, more transparency and appropriate software measurement is needed. Introducing software measurement with COSMIC sizing will help CIOs manage more effectively.
Software Measurement helps answer key questions
When trying to answer the questions of how much effort is needed to deliver a software project or when will it be finished, most organisations rely on the opinions of experts. These software estimates tend to be unscientific and prone to manipulation. What they need is a consistent means of software measurement. They are an unsound basis for dependable management decisions, and can often lead to counterproductive behaviours. However, with standardised, valid software estimates and software metrics based on standardised functional sizing in COSMIC Function Points (CFPs), CIOs can significantly improve the visibility, manageability, productivity, and value for money of the software work in their organisation. CFP-based software metrics and corresponding project metrics are also appropriate for board-level reporting. This 15-minute read will be sufficient to get you started on this journey to higher levels of software management.
"When will it be done?"
This article shows how these questions CAN be answered, by introducing just one simple approach to software measurement.
Software Measurement of Size Matters.
Let us consider some general characteristics that are particular to software work:
- Software work is almost entirely driven by human effort.
- There are many factors that determine the effort it will take to develop some software. The most significant factor is functional size.
- Functional size can be estimated and in some cases even measured before coding starts.
- There are dozens of other factors affecting how much effort it will take to develop software, including: team competency, complexity and executive support, but size is the most significant.
- There is a high variability in productivity between developers. 10x or more between least and most productive – and the same for teams.
- Software work does not scale. There are diseconomies of scale, mostly caused by the high proportion of time spent communicating amongst team members. A common language and knowledge of a project’s size can help contain this problem.
Ever since software became an industry, software sizing has been a topic of endless debates amongst developers and managers. Given the number of people involved in the software industry, it is surprising that executives and boards have allowed this to happen.
Whilst there are many factors affecting the cost and duration of a software project, size is the most significant factor. If you have analysed the requirements and determined the functional size in CFPs, you can predict, with reasonable confidence, how long it will take and how much it should cost. These are vital questions that responsible managers need to answer to plan effectively. Armed with the knowledge of functional size early in a project’s lifecycle can help to improve the quality of decisions that are made by managers and hence can reduce the risk, cost and duration of the project. Industry benchmarks are available or organisations can establish their own benchmarks for productivity in CFP, in order to answer the “when will we be done?” question.
Software Measurement and Software Engineering
All other forms of engineering adopt standardised (ISO) measures of size. Software Engineering is the only Engineering “discipline” that has failed to do so. Software engineering should embrace standardised size measurement.
Standardised Software Sizing
Many people think that software size measurement can only be performed after the work has been completed. This is not so. Software can be measured performed before the coding has even started.
Most people involved in the industry are unaware that software size measurement is possible, and easy to do, even before any coding work has started. Functional size can be determined just by examining the user requirements.
You can count lines of code after software has been written, but the line of code count (LOC) is of little value except when considering the work to maintain what has already been written. Also, consider two developers writing some functionality, dev A can write it in just 4 lines of code, whereas it takes dev B 40 lines to achieve the same functionality. Which is bigger? Which is more productive? Which is more error prone? This shows us that we cannot reliably use LOC as a predictor of size, effort or time estimation.
Software Measurement of Functional Size leads to Data-driven Decisions about Software
As a society we are moving rapidly towards data-driven decision-making. Even for decisions at home, we look at reviews and technical data before making a purchase decision. For this approach to be effective, we need to be using valid, consistent and trustworthy data. Having examined core measures in the software space for many years, we have concluded that, COSMIC size based metrics are the most reliable.
- Size or scope (CFP),
- Productivity metrics (CFP/time),
- Software quality metrics (Defects/CFP) and
- Resources (resource/CFP).
What is Functional Size
CFP in a Nutshell
A piece of software can be sized in CFPs before, during or after it has been coded. You first establish some boundaries for “the count”, then you can determine the number of CFP in the software by adding up the data movements: Entry, Exit, Read, Write. See the sizing guide. COSMIC sizing allows size to be determined without doing detailed design work up front. Detailed knowledge of attributes and constraints are generally not required. This is important for agile software development where design is often left to the “last responsible moment”.
COSMIC sizing is a refined version of the original idea of functional sizing that dates back to the 1970’s. COSMIC is easier to learn, more suited to agile and more consistent than its predecessor.
Move to Good Practice Sizing of Software
Measuring and estimating software has been a challenge since the software industry first began. We’ll skip the history lesson and jump straight to the current state of play. Most agile software teams estimate effort using unscientific, inconsistent “effort-equivalent” indicators of story points or T-shirt sizes. Story points and T-shirt sizes are unreliable, non-linear and highly game-able proxy guesses for how long it will take to do a chunk of work. The number of story points allocated by an estimator or team will be influenced by the motives of whoever is declaring the size. “Story points” are often the only indicator that managers are offered to help them plan and manage the work, so these indicators become overused and misused as some kind of reliable metric, when they are far from it. In some cases story count is used as an indicator of progress and size. Our studies of over 250,000 user stories have shown a variance in story size of 2-120 CFP, evidencing that story size is inconsistent and therefore story count is unreliable.
The solution to this problem has been available for a quite few years now, it is the ISO standard measurement of software functional size known as the COSMIC Function Point (CFP). COSMIC sizing is a free methodology that is a stable, international standard, governed by a non-profit organisation https://www.cosmic-sizing.org/. The COSMIC standard is an adaptation of the original concept of functional sizing that was invented by IBM-er Alan Albrecht in 1974.
The COSMIC Function Point (CFP) is a unit of measure of functional size that can be applied to ANY software project, DevOps work, software maintenance or package implementation. It is a consistent and universal size measurement that can be used irrespective of the coding methods or languages used. It is gradually becoming recognised as the leading standard for sizing and is being taught by a growing number of computer engineering courses throughout the world.
COSMIC sizing allows you to estimate or measure functional size (the number of CFPs) within a chunk of software. It represents a view of size from a user’s perspective, not a technical perspective. (Note that a “user” can be another system actor, so this methodology can also be used for embedded systems). It is agnostic to the technology or architecture used to deliver the software and so can be used across an entire portfolio of software and maintenance work. As a CIO you can measure every one of your software projects and software maintenance activities. You will benefit from valid, consistent software metrics upon which to make well-informed decisions. You can even measure past projects to provide benchmarks for productivity.
We want to know when will it be finished
This is a common challenge faced by CIOs. And it us normally very hard to answer. But when you use CFPs as a standard size metric, and you track the output of the development teams in CFP per month, or CFP per sprint, then the end date becomes predictable. Measuring software size is the cornerstone to valid software metrics, and answering the crucial question of how long will it take.
CFP size and it’s Relationship with Effort
The CFP size happens to be an excellent predictor of effort, and therefore cost. Independent studies have shown that CFP is a more consistent predictor of cost than a team’s own estimate in story points (see charts below). Productivity rates vary from team to team, but any given team, with all other things being equal, is likely to deliver a consistent number of CFP per sprint or per month. So, you can quickly determine the cost per CFP.
Suitable for Contracts
Since CFP is a consistent and independent measure, it can also be used in software contracts. Rather than be forced into a “time and materials” contract with a development vendor, CIOs can request cost per CFP based contracts. This can achieve more certainty of cost, even if you don’t yet know all the requirements. Outsourced software contracts based on story points are likely to be costing you 10% more, and taking about 10% longer, than those based on a firm fixed price per CFP. In such contracts they should also include targets for productivity (e.g. CFP/month) and quality (limits on defects/CFP found in acceptance testing, by criticality).
Learning to Measure Size
The technique of sizing software functionality in CFP takes about one day to learn the basics and about three weeks to become proficient and certified. With experience, you’ll be able to determine the equivalent size of a staff-year of development work in less than a day of sizing. If you choose to use our automated sizing tool, ScopeMaster, CFP sizing can be performed even faster and with little effort.
Standardised Software Sizing in Your Organisation
As in all things, the use of standardised weights and measures, leads to a fairer, more transparent market. Standardised software sizing will do the same for the software services market – it’s only a matter of time. Until then, there are winners and losers. The USA alone spends $500 billion per annum on enterprise software. We estimate that over 30% of this could be saved with the adoption of standardised software size measurement.
If you want to know why humans do what they do, it is always worth following the money trail. There are multi-billion dollar institutions that are thriving on “time and materials” contracts and whose commercial interests will be threatened by the introduction of standardised sizing. Nowadays most software work is commissioned on a time-and-materials basis, with little commercial risk to the supplier. Once purchasing organisations start to insist on paying for output instead of effort on software work, the balance of commercial risk will shift slightly from vendor to buyer.
Also, many consultants, agile coaches, framework vendors and even developers benefit from the lack of transparency that comes from not formally sizing software.
By using CFP sizing, buyers will be able to procure software services including developers, testers, agile coaches, architects, even consultants on a cost per CFP. In due course, wide adoption of CFP sizing is inevitable because those buying the software services will discover that not only is it more transparent and measurable, but with proper sizing, you can deliver more software, faster. Eventually this will lead towards commoditisation of software work.
The ability to deliver software change projects faster and better than competitors is becoming a key determinant of corporate survival. By using valid measurement, you can determine which teams, suppliers and projects are actually successful vs those who just pretend to be. For example, if team A delivers 1000 CFP in 12 months for $1m dollars, whilst team B delivers 700 CFP in 20 months for $2m you can clearly see who is doing a better job. This is not possible without CFP sizing.
There will be resistance to adoption of transparent sizing of software work. Transparency and measurability can be threatening to those who benefit from the absence of it. You should expect pushback from your development suppliers, and even your own in-house development teams, who currently enjoy a degree of freedom that comes from being unmeasured.
You’ll hear pushback like “we don’t have the time to do that”. For a typical refined user story, it will take up to 2 staff hours of aggregated conversation to shape, analyse and estimate in story points. Whereas CFP can be determined in just a few minutes.
Another reaction you’ll encounter is “we have plenty of metrics already”. In response to this, you’ll need to challenge your teams on the validity of the “metrics” that they currently provide. Here are some commonly used, so-called metrics, and their failings vs CFP-based equivalents. These are commonly used indicators, they may be helpful but they are NOT valid metrics based on a metric system.
- Story points – these are subjective, gameable, unscientific and inconsistent within and between teams. Story points are essentially a proxy for an estimate of hours of effort. Instead use CFP.
- T-Shirt Sizes– are also subjective, gameable, unscientific and inconsistent indicator of effort.
- Story Counts – stories vary in absolute size, typically ranging from 2 – 120 CFP, median of 6 CFP, so counting stories of inequivalent size is an unreliable indicator of progress.
- Agile Velocity – quoted in stories or story points over time. See Story Points, this should be considered an indicator of productivity. Instead use: CFP per sprint or month`
- Sprint Burndown or weekly burnup– based on story points or story counts. Instead use CFP delivered.
Nb. Cycle time, lead time, defect fix rate, code complexity, defect discovery rate, and test coverage are examples of valid metrics that we encourage.
Using CFP as a Foundation Metric for Controlling Projects
As well as using CFP for sizing your software portfolio and projects, you can use CFP-based metrics as the basis for controlling project progress:
- productivity (CFP/month),
- quality (defects found/CFP)*
- scheduling human resources such as work allocation (in CFP) per individual.
- scope (CFP), scope change and scope volatility
*this can be further broken down to defects by criticality and defects by source – all per CFP.
These are all vital measures for effectively controlling software projects. You’ll quickly start to see which projects are going well and which ones need help.
CFP and Quality
Let us take a look at the relationship between software quality and sizing or knowledge of size (in CFP).
The process of sizing helps with quality
The process of sizing encourages us to think deeply about the functional requirements needed to deliver a valuable business capability. This thinking helps to flush out potential misunderstandings early. These misunderstandings would otherwise manifest themselves as defects. So the process of CFP sizing is, in effect, the “ultimate in shift left testing”.
Knowledge of size helps improve quality
Knowing the size can help us (when using benchmarks) predict the number of defects that we might expect to find in a piece of software. This forecaste, in turn, will help us determine how much testing is needed? By working with defects per CFP, our appreciation of quality improves.
Automated analysis helps improve quality too
By choosing to use ScopeMaster® to analyse and test requirements, you will immediately discover about 50% of the latent requirements problems that you can then rapidly and early. This represents about 8% of all the defects on a project, and even more on larger ones.
Vendor Quality Management with CFP
When you use a contract that insists on a maximum level of acceptable defects per CFP, you can encourage vendors to ensure quality. Something that on T&M contracts, they may be less motivated to do.
Limitations of Functional Size Measurement with CFP
CFPs are a measure of functional size from the user’s perspective. They measure data movements of a software system. They cannot measure computational size of a software transaction, nor can they be used to measure “non-functional” aspects of software work, such as performance, maintainability, usability (and other -ilities). This is not just a shortcoming of CFP but a failing of the entire industry, nobody has yet devised valid, consistent measures of these two dimensions (computational size, and NFRs) of software.
Even with those limitations, the correlation of CFP to the effort taken to do the work is exceptional, meaning that knowing the size in CFP will give you a very strong indicator of how much effort is needed and therefor cost and timelines – the two pieces of planning knowledge that project managers and development managers need to make sound decisions.
When Can you Estimate or Measure Size
Estimating CFP can be performed at any time in the software development life cycle, even right at the beginning, as soon as the business requirements start to emerge. The early estimates can become more refined and accurate as your knowledge of the user functionality increases. As the requirements evolve, the certainty of the estimates increases and eventually the estimates can become an accurate measurement. To perform an accurate measurement, you just need to know all the data movements associated with each functional requirement (or functional user story). With CFP you can size any given set of requirements, you do not need to know all the requirements up front, nor do you need to know the design to determine the CFP size. At the very latest, you should measure the CFP size of a user story just before it is accepted into a sprint. You can even estimate or measure past projects in order to build up reference data for internal benchmarks of productivity.
The Disadvantages of Adopting CFP
There are many people in the world of software whose livelihood benefits from the absence of consistent size measurement. Amongst the biggest beneficiaries are those organisations that sell software development services on a time and materials basis. This includes some very large and influential organisations. Also, many consultants, agile coaches, framework vendors and even developers benefit from the lack of transparency that comes from not formally sizing software. Mistakes can be hidden, productivity is hard to assess and this may suit their interests. On the other hand, for anyone responsible for buying or managing software work, there really is no downside to measurement with CFP. It takes a trivial amount of effort to do the measurement. Both the knowledge of the size and the act of CFP sizing can help reduce problems of scope, improve quality and improve productivity on any software project.
CFP is Perfect for Agile and Scaled Agile
The full requirements of a software project are rarely known at the project outset. The Agile workflow allow you to handle this early uncertainty. Agile projects are characterised by evolving awareness of requirements. Fortunately, CFP can be measured precisely even before all of the requirements for a system are known. You simply measure just what you do know. You can also potentially estimate what is not yet known for sure. With experience you can even estimate early when only the broad capabilities or epics are known. As a project progresses, epics are broken down into functional user stories, and those initial estimates become more precise measures of CFP with a decreasing margin of error. Think of requirements certainty as a spectrum ranging from “we know everything” to “we know very little”. Although it is easier to size a project if you know all the requirements, it is surprising how much is knowable early on. In general we just don’t work hard enough at the outset to uncover that which is knowable.
In organisations with multiple software teams, senior managers need to know the productivity of different teams, in order to help improve underperforming teams and to learn from highly productive ones. CFP resolves the problem of a lack of consistent sizing between teams. By adopting CFP across all teams, productivity rates can be compared. Productivity rates can also be compared between internal and external teams. This helps provide software executives with the data that they need to make better decisions. One should never compare teams with story points.
It is Time to Get a Grip on Software Size
After 70 years of guesswork and non-scientific approaches, the time has come for responsible CIOs and software leaders to insist on the adoption of proper engineering metrics for size. By introducing and insisting on CFP adoption, you will enjoy better visibility and predictability of your software activity. There will be naysayers in your organisation and amongst suppliers, so you’ll need to be ready to encourage them to see the merits of valid size measurement. Software size measurement is good practice. It leads to better measurability and ultimately more effective management. In due course you’ll be able to share metrics of CFP delivered, CFP maintained as part of your board-level metrics. Forward thinking CIOs who want to be able to improve performance through measurement will benefit significantly from adopting CFP.
Automation is Here to Help
Our ground-breaking work at ScopeMaster, provides an automated consistent and valid sizing of functional user stories, or written functional user requirements effortlessly. ScopeMaster will estimate the number of CFP in a user story by analysing the language of the story. Most agile projects contain less than 500 user stories. These can be automatically sized within an hour using ScopeMaster. The user stories can then be refined (improved in quality) and the estimate will become a measurement, this typically takes a tenth of the time using ScopeMaster® than without.
Estimation vs Measurement
Estimation is about anticipating the size, duration and effort before a software undertaking has begun. Whereas measurement is a determinant of absolute size, achieved by following a standard sizing methodology, size measurement is a matter of fact.
Story Points are Misleading
When thinking of sizing software, most software managers turn to expert opinions about how much effort or how long it will take to develop and release a piece of software. Commonly used indicators are staff-days (or proxies thereof such as), story points, t-shirt sizes or numbers of stories. These approaches may provide some indication of size, and therefore effort, but they are not reliable measures. So they are unsuitable for dependable management metrics. This article will show that there is a better way, a consistent and objective way to measure software size, and that by doing so, as a CIO you will enjoy greater visibility and productivity of software work in your organisation.
Next Steps.
If you’d like to know more about how to measure your software portfolio, learn about valid software metrics and measure your development productivity to bring greater certainty to your software projects here are some suggested next steps:
- Encourage your management team to discover COSMIC sizing in CFP, we recommend the Guide to Software Measurement
- Encourage your PMO to introduce CFP sizing as a metric in portfolio management
- Ask your vendors to report on CFP delivered per sprint or per month
- Ask your quality managers to look into reporting on defects found per CFP.
- Ask your contracts team to investigate fixed price contracts per CFP, and ask your suppliers to quote in cost per CFP.
- Anticipate pushback from within your organisation and from suppliers, be ready to counter it with your own need for improvements in transparency of size, productivity, value for money and quality measurement.
Reach out to us at ScopeMaster to explore our automated sizing tool and guidance on adopting CFP in your organisation.
Knowing the Size
If you are looking to dive deeper into how to measure size with CFP, you may find it helpful to read this guide by the co-founder of COSMIC
About Colin Hammond
Colin Hammond is an expert in software sizing, project assurance and automated software requirements analysis. He regularly speaks at conferences of project management, software testing, business analysis and cost analysis. Colin is the founder and CEO of ScopeMaster who provide the award-winning automated functional sizing tool ScopeMaster and professional services on how to adopt and use functional sizing.
Thanks to
I am very grateful to Kirk Bryde (Agile Coach) and Lonnie Franks (Software Project Assurance Expert) for their valuable editorial input to this article.
References
McConnell, Steve. “Software Estimation, Demistyfing the Black Art”
Boehm, Barry et al 2000. “Software Cost Estimation with COCOMO II”
Garmus, David and Herron, David. “Function Point Analysis: Measurement Practices for Successful Software Projects.”
Jones, Capers. “Applied Software Measurement: Assuring Productivity and Quality”
Symons, Charles “A Guide to Software Size Measurement”
