Overview of a Product Management Assessment Framework

In a previous post, I enumerated what I consider to be the most important criteria for evaluating a framework for assessing a product management organization, particularly one focused on software. In this post, I'd like to outline the approach to product management assessments I've defined with these criteria in mind. I'm looking forward to feedback and discussion on the topic.

In an effort to be comprehensive, I've based the assessment approach on the idea of organizational maturity, a blanket concept that I think is reasonably intuitive. To keep things simple, the assessment generates a single value or score for the maturity of the product management organization. The organizational maturity score is a weighted average of 3 "dimensions" described below.

Dimension 1: Process Maturity

Process Maturity measures how well the organization understands, manages and executes its business processes. A business process converts business inputs to business outputs. Business processes don't have to be described with detailed graphical models and reams of documentation to be managed. What's important is that the organization understands which processes are important, have assigned ownership and is continuously improving them. I used ISPMA's framework to identify the "menu" of key process "elements" to be analyzed. Organizations should probably avoid analyzing more than 5-7 processes at the same time.

Each process can be assessed in general terms, e.g., is there a clear owner, as well as process-specific criteria, e.g., are roadmaps generated for different audiences (typically, a sign of maturity). These criteria can be scored, from 1 to 5 for example, so that an average or weighted average score can be calculated.

In terms of key process deliverables, I use a list of types of information organizations should be gathering rather than relying on a predefined set of artifacts. Checking process deliverables such as roadmaps and strategy papers for quality and completeness can help identify process that are suboptimal.

Dimension 2: Professional Competency

The professional competency dimension helps determine if the right people with the right knowledge and skills are executing the product management function. My framework uses a questionnaire that is filled out by product managers and their managers ranking their proficiency with various product management activities. My questionnaire is based on the Dimensions of Competency I've described in this blog (business leadership, product definition and product promotion).

Interviewing product managers, product management and executive leadership is also important, allowing an experienced assessor to determine if those doing product management have the appropriate background and personality to perform at a high level. There is clearly a subjective element to this dimension, but I don't believe any form or tool can replace experience and judgment. I also think it's important to interview members of other disciplines such as development, marketing and sales regarding the effectiveness of the product management organization.

Dimension 3: Organizational Setup Effectiveness

The effectiveness of the organizational setup is probably the most complex of the dimensions. At its heart, it explores if the organization is set up in a way that promotes success. For example, structural analysis looks at to which discipline the product management organization reports. I believe that, ideally, the functional and technical perspectives are separated organizationally. I talk about these different perspectives in my post about the "piles" of product management.

Organizational setup effectiveness also captures aspects such as professional development, engagement between functions, e.g., marketing, sales, and completeness of product development disciplines. For example, a highly effective product management discipline will ultimately fail if marketing or sales are underdeveloped or even absent.

Bringing it All Together: The Maturity Dashboard

Via simple tools, observation and interviews, scores are generated for each dimension. Weights are assigned for each dimension with the weighted average representing the organizational maturity score. Although this approach is not purely scientific, it is highly approachable and supports prioritization of improvement efforts, i.e., the efforts that have the highest impact on the organizational maturity score are those that should be addressed first. This idea is predicated on the idea that the dimensions and the information gathered truly represent critical aspects of the product management discipline.

The figure below shows a fairly simple representation of what can be a large, complex set of data. I've defined the different levels of overall maturity informed by the maturity levels found in the Capability Maturity Model (CMM).

Note the weights assigned to the dimensions to the left of the labels in the table at the bottom. These are examples of values that can be easily adjusted for a particular client. Some organizations might emphasize Professional Competency (or any of the other dimensions) more heavily so would adjust its weight accordingly. 

Assessing the maturity of an organization is an inherently complex problem. The framework I propose attempts to balance objective data with the judgment of an experienced assessor. A great deal of qualitative and quantitative data would, of course, accompany the dashboard.

So what do you think? Is it similar to other frameworks you've seen? What do you see as the biggest gaps? Do you think this approach could be valuable with your organization? 

Previous post: Does your organization really know what it's taking to market?