Product Line Engineering

Moving targets – Change Management for Software Product Lines

Posted in Article by danilo on October 21, 2009

This article is dedicated to change management in software product line development. This is a challenging topic and in many cases more critical for the success of a product line than configuration of products. A software product line is similar to an living organism. It keeps changing continuously. Many who have heard a little about software product line development believe it to be far more complicated than the normal software development they are used to. Surprisingly software product line development is very similar to developing in a conventional way in many aspects.  The main differences are caused  by the number of products you have to care for simultaneously, which, no doubt about this, adds complexity and challenges.

Sources of Change

There are quite a number of different sources for change requests and changes to a product line. One source are defects found in products. Defects need to be removed quickly and it’s very important to estimate the impact for affected products relevant for the business, i. e. products already released or currently under development.  Speed is the key here: only a quick (and accurate) impact  estimation allows for proper decisions regarding when, how and for which products of the product line the defect is fixed immediately, later, or never.

A second source for change are new product ideas. Often marketing, product management or customers come with ideas such as a mobile phone with a GPS receiver to support navigation applications. Sometimes it requires new features to be integrated into the product line (changed hardware and a piece of driver software in this case) to support the product idea. However product ideas often just require a new combination of existing features and requirements (a business phone with mp3 player and fm radio applications).

Technology trends (like touch screen interaction in mobile phones) are different in this respect: there should be more time allocated to think integrating new technology trends into a product line since changes coming from technology trends might have a huge impact on the features available in the product line, the requirements and other information as well as the solution space. So a single technology trend can require a multitude of changes in the existing information about the product line.

Change Management Challenges

Where are the main challenges in product line change management? Well, it is obvious that the impact of a change is higher than in a single product development. Simply because a change potentially affects products already shipped to the market or being in development. A single problem introduced with a change might affect many products. However, the reverse is true as well: A single fix also fixes problems for all affected products. In any case, due to the increased impact of a change, they should be treated more carefully. Not only does one change affect many assets, there are typically more changes requested for each shared asset. Since there are more stakeholders involved, more change requests for a shared asset compared to using the same/similar assets in a single product development are likely. This often includes change request asking for “incompatible” changes. If not handled properly in the change management process, the engineers might be kept busy with checking change requests instead of getting things done like coding or testing.

However, the number of defects/issues nevertheless is the same or even decreases over time compared to developing assets for a single product. The reason for this is fairly simple: The number of issues is related to the number of lines of code. Shared code is usually larger than non-shared single product code (due to the variations) but this increase is in most cases moderate. Shared code is often due better tested and more robust (more “running hours” help to expose issues). So in the end, this gives you one of the advantages of product line development: better code with less issues. In one case I saw a reduction of the product line related issues to 10% of the issues after merging and sharing parts of the system in a product line fashion.

It is challenging to control of the speed of change. One project needs new features and fixes available in the shared assets as quickly as possible. Other projects want to have a very stable base not changing too frequently, since every change in the shared assets might cause retesting/rereleasing of these products. This tension can not be completely avoided, so there has to be a compromise. However, the processes and the used technology must be able to relieve these pains as much as possible. And just another dimension of complexity is the increased number of stakeholders in such and scenario. This requires more coordination and communication, which in turn must be more efficient to not burn off the benefits of a SPL approach.

Last but not least comes the communication and information exchange infrastructure. In product line development it is very important that all stakeholders have easy access to all (!) relevant product line information (requirements, test results, change requests, product variant definitions, and feature models) in the same way they can access shared code in a version control system. This requires some kind of centralized and unified management of this information. Most requirements/change management/test tools available on the market and in use in companies are not fully able to support product line development entirely on their own. So usually you will have a requirements tool / change management tool in order to maintain requirements / change requests and and a software product line tool to maintain the variability and variant information. The information in this database(s) keeps changing continuously. And even this information is often not enough. You will have links to supplemental documents for things like architecture diagrams or customer request and similar information. All this information needs to be presented to different stakeholders like project managers or architects in different ways so you will have to provide a wide set of views for those stakeholder. For instance a developer in a specific project just wants to see the requirements and features he or she has to care for, not all features for the whole product line. A product line manager might be interested in comparing different product variants and the states of related change request  for a specific group of shared features. The product line information model must allow to create the relevant views and to perform the related task in high quality without much overhead compared to single system development.

While there are obviously notable differences in the way changes in product line related changes should be handled, product specific issues or change requests  can be handled just like issues or changes in normal single product development.

Getting it right

Of course there is no silver bullet for SPL change management, but the next section gives you some ideas how a product line change process could be organized in terms of roles and organizational structures.

In order to deal with the inherent complexity of SPL change management , you have to set up a proper infrastructure meeting the specific requirements of the organizations, the products’ markets and/or customers’ expectations regarding change management, and many more. However, the basics are very similar in most successful approaches. We use an example to explain the roles and processes in change management. Lets assume there is  a platform providing the core assets, and several projects using these core assets in their product projects. The management of all those individual project and the platform development is controlled by a Product Line Management Office. The essential roles in this office are the product line manager, who coordinates platform and projects, as well as additional tasks of the product line such as inter project reporting and evaluation. Basically he sits on top of everything. Another important role on this level is the product line quality manager who is responsible for controlling and maintaining the quality of all developed artifacts. This is not the role specific to a software product line development but of course even more important here. The domain expert is providing the necessary knowledge about the problem domain and its mapping to the concrete solution  domain.

In this scenario change requests are handled by one Change Control Board. This boards sits on top of all the projects and the platform. Why is this so important? If change requests are by default handled in projects and only the ones which are considered to be critical for the product line are forwarded to the PL CCB, there will be a) at least a delay for critical issues before other projects can learn about the issues and b) many potentially relevant topics will not be forwarded at all, since this means giving control to the PL CCB. This transfer of control usually does not work too well this way. The PL CCB can and should delegate purely single product specific issue quickly to the affected project, but needs to keep in control of all changes which have impact on multiple products or the core assets.

While the change control board acts as a filter for the incoming data, the change requests, the architecture review board is responsible to make sure that the architecture at all times remains stable and usable for all projects. Since evolution of products usually also call for evolution of core assets over time, this task is a continuous task. Other tasks of the architecture review board might also include the description of architectural guidelines, patterns etc. Basically, the board controls the solution space implementation.

Last but not least there is the product line management board consisting of  the product line manager plus the project managers. It is responsible for strategic decision affecting the product line such as setting the scope of the product line, deciding about organizational structures in the product line organization, priorities for product development and similar (high-level) issues.  Also participating in a product line management board are the product sponsors (if separate from the roles mentioned above). Product sponsors actually represent users or a customer of the products (this role is often called product managers) . For some decisions the domain expert(s) will participate. However the product line management board is more about coordinating work in the projects then about conceptual things, that’s why normally only the manager and sponsors are participating.

The domain expert participates in the change control board to ensure consistent handling of changes in the domain and to align changes from different sources in order to maximize reuse and minimize double work.  If possible, the product sponsors should be part of the change control board in order to make sure that changes do not affect the product line strategy. In the architecture review board technical architects and occasionally the domain expert are participating. Only the poor (or lucky, depends on the perspective) developers are not part of any of those boards.

Since those boards only meet in intervals, someone has to do the work in between. As part of the product line management office, the SPL task force deals with the day-to-day business of running a software product line development. And if serious issues arise the task force has to notify the respective boards. The structure of this task force includes the domain expert, who is somehow this task force, technical architects, if necessary business domain experts and also experts for the individual products. This task force identifies and evaluates new features and is proposing them to the change control board. It defines the conceptual domain model in its details but also identifies potentially relevant technology trends. It also looks into the architecture. However, it is most important task is to keep the projects happy by helping projects dealing with a software product line development.


Is Your Change Management Fit for Product line Development?

Even if you have a nice architecture and the very nice technology like pure::variants backing up your product line development, if you are not able to deal with the more or less steady stream of changes properly, you are quickly lost. So what are the important properties of a change management approach when it comes to software product lines?

(1) Change requests should be traceable to the products affected

If your change management just allows you to say there is a change request, but does not allow to record to which out of your existing products this request is related, it’s not suitable for a software product line development. The two simple cases are that just one product is affected or that all products are affected. However,  often a subset of the products is affected. Knowing the affected product subset helps to minimize (especially testing) effort and communication overhead. Without this knowledge all issues have to be communicated to all products unless it is clearly only affecting a single product.

(2) Core assets should be traceable to products using them

If you have to change your core asset due to a change request for one issue fix you have to know which products are affected by this asset change. Otherwise you might test too many products which takes long time and might delay product releases.  Or you have to recall shipped products from customers who are not affected by the problem (since the product not really used the asset). This can be very bad for business.

(3) Change requests should be traceable to related core assets

In many cases it is not sufficient to only be able to trace changes to existing products. Imagine the case that you entered an issue in the change management system and later you create and build a new product. If you are to tell which core assets are affected by issues and your are able to trace core assets to products using them (2), you will able to get traces to the issues which potentially affect this newly created product.

It is obvious that having (2) and (3) in place  (1) becomes a much less challenging task, since the list of the products can be derived (almost) automatically.

None of these properties are mandatory in a sense that a product line approach without those is automatically doomed to be a complete failure.  However, especially in case of initial success and related growing of number of products derived from the product line including higher change request number, this success can backfire quickly without proper change management in place. So you be better prepared for it.

A Sample Change Management Process for Product Line Development

Due to the widely differing domains and variety of approaches for product lines, there is no single change management process that fits all. However,  the sample process described below is applicable for most product lines. In a number of cases the process can be substantially simplified. This will be discussed after the introduction of the core process.

The scenario assumes that a number of products is being developed and/or maintained within the product line in parallel. Products are defined by user visible features/functionalities and build from shared core assets plus some product specific assets. Customers/users usually report defects for a specific product, often naming the affected feature(s). Similarly change requests for adding, changing, or removing of features are often related to product customers/users but others are  coming from other sources such as the product line management board and are not related to specific products. The states and workflow for handling issues is assumed to be a standard workflow available in most issue tracking tools ( with states like “New”, “Assigned”, “Fixed”, “Verified” and a simple dependency relationship between issues). For an example of an (simple) issue tracking tool, have a look at Bugzilla.

The term “issue” is being used as a general term describing defects, action items, and change requests.

The single system approach of just creating a single issue for a product defect obviously does not work in a product line scenario, since a defect in a shared core asset might affect multiple products. Linking the issue to all affected products instead of just a single one is better but has also some limitations: The issue can not be closed before it has been verified for all products the fix is working. If the verification of the fix has to be done for groups of products separately, for some of the products might the fix might already verified to work, for others this information is missing. This is often complicated to record in issue tracking systems, since it would require some kind of intermediate state for the issue like “partially verified” and a list of verified products to be recorded in the issue. Even then, a release of some products for which has been verified successfully is problematic since the issue is not officially  in a “Closed” state unless for all products the fix has been verified to work. Issue Relations


Splitting the single issue in several related issue types solves this problem. A WorkIssue is used to record the progress for solving an issue. ProductIssues are used to record the state of the issue with respect to individual products, i.e. each ProductIssue is linked to one particular product. The purpose of a ProductIssue is mainly to trace the verification of the relevance of the issue for the product and the fitness of the issue’s solution for the particular product. Last but not least a MasterIssue is used to link WorkIssue and all ProductIssues together since the actual issue is only completely closed after the WorkIssue and all related ProductIssues are closed. The MasterIssue is used to document this fact, by depending on the (successful) resolution of all issues it depends on.

spl_cm_create_update_issue CM Activity Diagram for Issue Creation / Issue Updates

The activity diagram shows what happens when an initial work issue is created. After an initial qualification of the issue, depending on the available information, the set of related issues is created. The initial issue becomes the WorkIssue. To keep things simple, a MasterIssue is only created if there are ProductIssues (i.e. there is more than one product affected by the issue). The same process is executed whenever there are changes made to WorkIssue.

This approach will support different goals: proper monitoring of issue fixing in the shared assets and the propagation of the fix into the products, it is simple if only one product is involved (almost identical to the single product development), and it allows different states of an issue for each product which is important to get high speed without compromise on quality.

Of course, the implementation of such a workflow needs proper tooling to support the impact assessment, and also to implement automatic reaction on state changes etc. for issues to remove unnecessary manual work. pure::variants of course can deliver its part here, like supporting the assessment and visualizing the product related issue states in nice views :-).

Final Remarks

Of course this article is not a complete blueprint for your specific way of dealing with change management challenges but it should give you some hints what to watch out for. In some cases simplifications to the workflows are possible: For instance if your organization is doing simultaneous releases of all products, there is no need to deal with individual states for each relevant product.

I find change management issues are not yet well represented in PL research and literature. I would like to read and hear more about these things both from practitioners and researchers! So go and write your ideas and concepts down and share with the rest of the world…

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: