Sunday, March 13, 2011

SDM Best Thesis analyzes costs of software system complexity - SDM Pulse Spring 2011

By Andrei Akaikine, SDM ’09

Andrei Akaikine
SDM ’09
Editor’s note: Andrei Akaikine was awarded the SDM Best Thesis Prize in October 2010 for his thesis, “The Impact of Software Architecture on Product Maintenance Efforts and Measurement of Economic Benefits of Product Redesign.”
“If it ain’t broke, don’t fix it” is the basic principle used to manage complex socio-political and engineering systems. For many engineers, the risk of upsetting a system’s stability outweighs the benefits of attempting to improve it when there is no evident flaw.
However, maintaining a complex system at a “good enough” level of functionality comes at a cost. In systems where stability is paramount, personnel dealing with small emergencies traditionally rely on “Band-Aid” fixes that increase complexity and heighten the risk of destabilizing the whole system with each new modification, however minor.
In most cases, a product’s lifetime maintenance cost overshadows the cost of initial product development by a large margin. Based on the empirical data for the computer software industry, the cost of a fix grows exponentially between each phase of the product’s life cycle. Some sources estimate that activities during the maintenance phase may consume as much as 75 percent to 90 percent of total product lifetime expenses.
Measuring costs associated with maintaining complex computer systems is one of my long-time interests. Having spent more than five years working in product support and sustained engineering at Microsoft, I have seen the effects of product architecture complexity on maintenance costs and overall software system stability. Combining my interests with the knowledge that I acquired in MIT’s System Design and Management Program (SDM), I focused my SDM thesis research on the following three questions:
•          Is there a relationship between engineering efforts required for a basic maintenance task and the overall software system complexity?
•          Are any components more susceptible than others to change during system maintenance tasks?
•          Can the redesign of a system be economically justified?
To answer these and other questions regarding the effects of system complexity on the cost of software maintenance, I designed an experiment based on data from the industry. The core idea was to demonstrate a correlation between some numerical measure of a system’s complexity and engineers’ productivity.
I studied two versions of a mature software product.
The chosen product had undergone a significant redesign between these two versions that involved rewriting an estimated 70 percent of the product’s code. However, most of product’s functional requirements did not change. This provided a basis for an experiment with few externalities.
In the experiment, I measured the effort engineers spent making a code modification associated with a corrective change request. For each version of the product, I tallied about 400 corrective changes performed during the first 30 months after release. I then used the time that engineers reported working on each change to estimate their productivity.
To measure the initial product architecture complexity of the product, I chose the methodology developed by Alan MacCormack, MIT visiting associate professor of technological innovation, entrepreneurship, and strategic management (see article, page 4), and Carliss Baldwin, the William L. White professor of business administration at Harvard Business School. This methodology is based on the application of design structure matrices (DSMs) to map dependencies between system components. I used DSMs to compute the design structure complexity measures that were most suitable to assessing software system maintainability (see Figure 1).

Figure 1: Visibility matrices of the product before and after the redesign.
Visibility matrix for the old product is much denser, indicating the high propagation cost.
The first complexity metric I used measures the degree of “ripple effect” propagation through the chain of dependencies across system elements. Propagation cost predicts the percentage of system components that can be affected, on average, when a change is made to a random design element. Propagation cost is a single numerical measure that characterizes the interconnectedness of the elements that comprise the product system.
For each design element, two other complexity measures can be defined: 1) a measure of dependencies that flow into an element—fan-in visibility; and 2) a measure of dependencies that flow out of the element—fan-out visibility. Based on fan-in visibility and fan-out visibility measures, components of the system may be characterized into four types. MacCormack et al. define these types as core, peripheral, shared, and control (see Figure 2). These few measures captured the important complexity attributes of the hypotheses I tested in my research.
My experiment showed a clear correlation between initial product complexity and engineers’ productivity. A decrease in the product’s propagation cost (or complexity) from 38 percent to 11 percent resulted in 10 percent to 14 percent improvement in engineers’ productivity. Besides engineer productivity, such metrics as amount of rework and effort needed for a single module change improved by 16 percent to 25 percent.
Figure 2: Characterization of components
by visibility measures.
Modules of the core type were found to be more likely to be touched during a random corrective code change.
Out of 542 unique modules that were changed, 404 were of the core type. The frequency with which core modules were modified is disproportionate to their share in the whole system: 85 percent of modules that were modified more than once during the studied period were core modules. Some of them were modified 10-12 times.
I concluded that product architecture redesign that reduces the structural complexity of the software system may become economically viable and desirable as the product support cycle grows in length and the pressure to lower maintenance costs increases. Considering that in some cases, maintenance can consume up to 90 percent of the total product life-cycle cost, an improvement of 10 percent to 15 percent in productivity may result in savings almost as large as the initial cost of product development. Of course, any redesign may have downsides that need to be considered before the decision to redesign an existing product can be made.
It is essential for a development organization to perform a comprehensive net present value (NPV) analysis before investing resources into redesign to avoid any NPV-negative initiatives.
That said, it’s clear that the economic benefits of controlling the complexity of product architecture have significant managerial implications. Product development organizations should measure the complexity of new and legacy software products to control their total life-cycle costs.

No comments:

Post a Comment