Move to Good Practice Sizing of Software
Measuring and estimating software has been a challenge since the software industry first began. We’ll skip the history lesson and jump straight to the current state of play. Most agile software teams estimate effort using unscientific, inconsistent “effort-equivalent” indicators of story points or T-shirt sizes. Story points and T-shirt sizes are unreliable, non-linear and highly game-able proxy guesses for how long it will take to do a chunk of work. The number of story points allocated by an estimator or team will be influenced by the motives of whoever is declaring the size. “Story points” are often the only indicator that managers are offered to help them plan and manage the work, so these indicators become overused and misused as some kind of reliable metric, when they are far from it. In some cases story count is used as an indicator of progress and size. Our studies of over 250,000 user stories have shown a variance in story size of 2-120 CFP, evidencing that story size is inconsistent and therefore story count is unreliable.
The solution to this problem has been available for a quite few years now, it is the ISO standard measurement of software functional size known as the COSMIC Function Point (CFP). COSMIC sizing is a free methodology that is a stable, international standard, governed by a non-profit organisation https://www.cosmic-sizing.org/. The COSMIC standard is an adaptation of the original concept of functional sizing that was invented by IBM-er Alan Albrecht in 1974.
The COSMIC Function Point (CFP) is a unit of measure of functional size that can be applied to ANY software project, DevOps work, software maintenance or package implementation. It is a consistent and universal size measurement that can be used irrespective of the coding methods or languages used. It is gradually becoming recognised as the leading standard for sizing and is being taught by a growing number of computer engineering courses throughout the world.
COSMIC sizing allows you to estimate or measure functional size (the number of CFPs) within a chunk of software. It represents a view of size from a user’s perspective, not a technical perspective. (Note that a “user” can be another system actor, so this methodology can also be used for embedded systems). It is agnostic to the technology or architecture used to deliver the software and so can be used across an entire portfolio of software and maintenance work. As a CIO you can measure every one of your software projects and software maintenance activities. You will benefit from valid, consistent software metrics upon which to make well-informed decisions. You can even measure past projects to provide benchmarks for productivity.
We want to know when will it be finished
This is a common challenge faced by CIOs. And it us normally very hard to answer. But when you use CFPs as a standard size metric, and you track the output of the development teams in CFP per month, or CFP per sprint, then the end date becomes predictable. Measuring software size is the cornerstone to valid software metrics, and answering the crucial question of how long will it take.
CFP size and it’s Relationship with Effort
The CFP size happens to be an excellent predictor of effort, and therefore cost. Independent studies have shown that CFP is a more consistent predictor of cost than a team’s own estimate in story points (see charts below). Productivity rates vary from team to team, but any given team, with all other things being equal, is likely to deliver a consistent number of CFP per sprint or per month. So, you can quickly determine the cost per CFP.
Suitable for Contracts
Since CFP is a consistent and independent measure, it can also be used in software contracts. Rather than be forced into a “time and materials” contract with a development vendor, CIOs can request cost per CFP based contracts. This can achieve more certainty of cost, even if you don’t yet know all the requirements. Outsourced software contracts based on story points are likely to be costing you 10% more, and taking about 10% longer, than those based on a firm fixed price per CFP. In such contracts they should also include targets for productivity (e.g. CFP/month) and quality (limits on defects/CFP found in acceptance testing, by criticality).
Learning to Measure Size
The technique of sizing software functionality in CFP takes about one day to learn the basics and about three weeks to become proficient and certified. With experience, you’ll be able to determine the equivalent size of a staff-year of development work in less than a day of sizing. If you choose to use our automated sizing tool, ScopeMaster, CFP sizing can be performed even faster and with little effort.
Standardised Software Sizing in Your Organisation
As in all things, the use of standardised weights and measures, leads to a fairer, more transparent market. Standardised software sizing will do the same for the software services market – it’s only a matter of time. Until then, there are winners and losers. The USA alone spends $500 billion per annum on enterprise software. We estimate that over 30% of this could be saved with the adoption of standardised software size measurement.
If you want to know why humans do what they do, it is always worth following the money trail. There are multi-billion dollar institutions that are thriving on “time and materials” contracts and whose commercial interests will be threatened by the introduction of standardised sizing. Nowadays most software work is commissioned on a time-and-materials basis, with little commercial risk to the supplier. Once purchasing organisations start to insist on paying for output instead of effort on software work, the balance of commercial risk will shift slightly from vendor to buyer.
Also, many consultants, agile coaches, framework vendors and even developers benefit from the lack of transparency that comes from not formally sizing software.
By using CFP sizing, buyers will be able to procure software services including developers, testers, agile coaches, architects, even consultants on a cost per CFP. In due course, wide adoption of CFP sizing is inevitable because those buying the software services will discover that not only is it more transparent and measurable, but with proper sizing, you can deliver more software, faster. Eventually this will lead towards commoditisation of software work.
The ability to deliver software change projects faster and better than competitors is becoming a key determinant of corporate survival. By using valid measurement, you can determine which teams, suppliers and projects are actually successful vs those who just pretend to be. For example, if team A delivers 1000 CFP in 12 months for $1m dollars, whilst team B delivers 700 CFP in 20 months for $2m you can clearly see who is doing a better job. This is not possible without CFP sizing.
There will be resistance to adoption of transparent sizing of software work. Transparency and measurability can be threatening to those who benefit from the absence of it. You should expect pushback from your development suppliers, and even your own in-house development teams, who currently enjoy a degree of freedom that comes from being unmeasured.
You’ll hear pushback like “we don’t have the time to do that”. For a typical refined user story, it will take up to 2 staff hours of aggregated conversation to shape, analyse and estimate in story points. Whereas CFP can be determined in just a few minutes.
Another reaction you’ll encounter is “we have plenty of metrics already”. In response to this, you’ll need to challenge your teams on the validity of the “metrics” that they currently provide. Here are some commonly used, so-called metrics, and their failings vs CFP-based equivalents. These are commonly used indicators, they may be helpful but they are NOT valid metrics based on a metric system.
- Story points – these are subjective, gameable, unscientific and inconsistent within and between teams. Story points are essentially a proxy for an estimate of hours of effort. Instead use CFP.
- T-Shirt Sizes– are also subjective, gameable, unscientific and inconsistent indicator of effort.
- Story Counts – stories vary in absolute size, typically ranging from 2 – 120 CFP, median of 6 CFP, so counting stories of inequivalent size is an unreliable indicator of progress.
- Agile Velocity – quoted in stories or story points over time. See Story Points, this should be considered an indicator of productivity. Instead use: CFP per sprint or month`
- Sprint Burndown or weekly burnup– based on story points or story counts. Instead use CFP delivered.
Nb. Cycle time, lead time, defect fix rate, code complexity, defect discovery rate, and test coverage are examples of valid metrics that we encourage.
Using CFP as a Foundation Metric for Controlling Projects
As well as using CFP for sizing your software portfolio and projects, you can use CFP-based metrics as the basis for controlling project progress:
- productivity (CFP/month),
- quality (defects found/CFP)*
- scheduling human resources such as work allocation (in CFP) per individual.
- scope (CFP), scope change and scope volatility
*this can be further broken down to defects by criticality and defects by source – all per CFP.
These are all vital measures for effectively controlling software projects. You’ll quickly start to see which projects are going well and which ones need help.
CFP and Quality
Let us take a look at the relationship between software quality and sizing or knowledge of size (in CFP).
The process of sizing helps with quality
The process of sizing encourages us to think deeply about the functional requirements needed to deliver a valuable business capability. This thinking helps to flush out potential misunderstandings early. These misunderstandings would otherwise manifest themselves as defects. So the process of CFP sizing is, in effect, the “ultimate in shift left testing”.
Knowledge of size helps improve quality
Knowing the size can help us (when using benchmarks) predict the number of defects that we might expect to find in a piece of software. This forecaste, in turn, will help us determine how much testing is needed? By working with defects per CFP, our appreciation of quality improves.
Automated analysis helps improve quality too
By choosing to use ScopeMaster® to analyse and test requirements, you will immediately discover about 50% of the latent requirements problems that you can then rapidly and early. This represents about 8% of all the defects on a project, and even more on larger ones.
Vendor Quality Management with CFP
When you use a contract that insists on a maximum level of acceptable defects per CFP, you can encourage vendors to ensure quality. Something that on T&M contracts, they may be less motivated to do.
Limitations of Functional Size Measurement with CFP
CFPs are a measure of functional size from the user’s perspective. They measure data movements of a software system. They cannot measure computational size of a software transaction, nor can they be used to measure “non-functional” aspects of software work, such as performance, maintainability, usability (and other -ilities). This is not just a shortcoming of CFP but a failing of the entire industry, nobody has yet devised valid, consistent measures of these two dimensions (computational size, and NFRs) of software.
Even with those limitations, the correlation of CFP to the effort taken to do the work is exceptional, meaning that knowing the size in CFP will give you a very strong indicator of how much effort is needed and therefor cost and timelines – the two pieces of planning knowledge that project managers and development managers need to make sound decisions.
When Can you Estimate or Measure Size
Estimating CFP can be performed at any time in the software development life cycle, even right at the beginning, as soon as the business requirements start to emerge. The early estimates can become more refined and accurate as your knowledge of the user functionality increases. As the requirements evolve, the certainty of the estimates increases and eventually the estimates can become an accurate measurement. To perform an accurate measurement, you just need to know all the data movements associated with each functional requirement (or functional user story). With CFP you can size any given set of requirements, you do not need to know all the requirements up front, nor do you need to know the design to determine the CFP size. At the very latest, you should measure the CFP size of a user story just before it is accepted into a sprint. You can even estimate or measure past projects in order to build up reference data for internal benchmarks of productivity.
The Disadvantages of Adopting CFP
There are many people in the world of software whose livelihood benefits from the absence of consistent size measurement. Amongst the biggest beneficiaries are those organisations that sell software development services on a time and materials basis. This includes some very large and influential organisations. Also, many consultants, agile coaches, framework vendors and even developers benefit from the lack of transparency that comes from not formally sizing software. Mistakes can be hidden, productivity is hard to assess and this may suit their interests. On the other hand, for anyone responsible for buying or managing software work, there really is no downside to measurement with CFP. It takes a trivial amount of effort to do the measurement. Both the knowledge of the size and the act of CFP sizing can help reduce problems of scope, improve quality and improve productivity on any software project.
CFP is Perfect for Agile and Scaled Agile
The full requirements of a software project are rarely known at the project outset. The Agile workflow allow you to handle this early uncertainty. Agile projects are characterised by evolving awareness of requirements. Fortunately, CFP can be measured precisely even before all of the requirements for a system are known. You simply measure just what you do know. You can also potentially estimate what is not yet known for sure. With experience you can even estimate early when only the broad capabilities or epics are known. As a project progresses, epics are broken down into functional user stories, and those initial estimates become more precise measures of CFP with a decreasing margin of error. Think of requirements certainty as a spectrum ranging from “we know everything” to “we know very little”. Although it is easier to size a project if you know all the requirements, it is surprising how much is knowable early on. In general we just don’t work hard enough at the outset to uncover that which is knowable.
In organisations with multiple software teams, senior managers need to know the productivity of different teams, in order to help improve underperforming teams and to learn from highly productive ones. CFP resolves the problem of a lack of consistent sizing between teams. By adopting CFP across all teams, productivity rates can be compared. Productivity rates can also be compared between internal and external teams. This helps provide software executives with the data that they need to make better decisions. One should never compare teams with story points.
It is Time to Get a Grip on Software Size
After 70 years of guesswork and non-scientific approaches, the time has come for responsible CIOs and software leaders to insist on the adoption of proper engineering metrics for size. By introducing and insisting on CFP adoption, you will enjoy better visibility and predictability of your software activity. There will be naysayers in your organisation and amongst suppliers, so you’ll need to be ready to encourage them to see the merits of valid size measurement. Software size measurement is good practice. It leads to better measurability and ultimately more effective management. In due course you’ll be able to share metrics of CFP delivered, CFP maintained as part of your board-level metrics. Forward thinking CIOs who want to be able to improve performance through measurement will benefit significantly from adopting CFP.
Automation is Here to Help
Our ground-breaking work at ScopeMaster, provides an automated consistent and valid sizing of functional user stories, or written functional user requirements effortlessly. ScopeMaster will estimate the number of CFP in a user story by analysing the language of the story. Most agile projects contain less than 500 user stories. These can be automatically sized within an hour using ScopeMaster. The user stories can then be refined (improved in quality) and the estimate will become a measurement, this typically takes a tenth of the time using ScopeMaster® than without.