Skip to content
Browse
BABOK Guide
BABOK Guide
10. Techniques
Introduction 10.1 Acceptance and Evaluation Criteria 10.2 Backlog Management 10.3 Balanced Scorecard 10.4 Benchmarking and Market Analysis 10.5 Brainstorming 10.6 Business Capability Analysis 10.7 Business Cases 10.8 Business Model Canvas 10.9 Business Rules Analysis 10.10 Collaborative Games 10.11 Concept Modelling 10.12 Data Dictionary 10.13 Data Flow Diagrams 10.14 Data Mining 10.15 Data Modelling 10.16 Decision Analysis 10.17 Decision Modelling 10.18 Document Analysis 10.19 Estimation 10.20 Financial Analysis 10.21 Focus Groups 10.22 Functional Decomposition 10.23 Glossary 10.24 Interface Analysis 10.25 Interviews 10.26 Item Tracking 10.27 Lessons Learned 10.28 Metrics and Key Performance Indicators (KPIs) 10.29 Mind Mapping 10.30 Non-Functional Requirements Analysis 10.31 Observation 10.32 Organizational Modelling 10.33 Prioritization 10.34 Process Analysis 10.35 Process Modelling 10.36 Prototyping 10.37 Reviews 10.38 Risk Analysis and Management 10.39 Roles and Permissions Matrix 10.40 Root Cause Analysis 10.41 Scope Modelling 10.42 Sequence Diagrams 10.43 Stakeholder List, Map, or Personas 10.44 State Modelling 10.45 Survey or Questionnaire 10.46 SWOT Analysis 10.47 Use Cases and Scenarios 10.48 User Stories 10.49 Vendor Assessment 10.50 Workshops

6. POA Techniques

6.12 Metrics and Key Performance Indicators

Guide to Product Ownership Analysis

Purpose

Metrics and key performance indicators (KPIs) measure the performance of the product or other attributes that are relevant to developing the product.

See section 10.28 of BABOK® Guide v3 for details.
To create a valuable product that resonates with customers, many characteristics need to be considered. Also, competing goals need to be resolved, including:
  • Business objectives vs. customer needs,
  • Short-term benefits vs. long-term benefits,
  • Strategic considerations vs. tactical considerations,
  • Existing market segments vs. new target segments, and
  • Customer experience vs. additional features.
Practitioners require this information to compare and contrast, and then make the right decisions. Each of these can be the difference between product success or failure.

From a POA perspective, metrics can be categorized based on the level of impact for the product:
  • Strategic Measures: Determine the effectiveness of a product vision or strategy and are directly tied to business objectives.
  • Product measures: Outline customer acceptance and the popularity of the product for the intended customer base.
  • Delivery measures: Indicate team performance and outline the team's ability to develop a cohesive product.
Common measures to consider and track as KPIs include:
Strategic Product Delivery
Return on Investment (ROI): Ratio of total revenue to the cost of investment. Net Promoter Score (NPS): A 10-point scale used to gauge customers' likelihood to recommend the product. Iteration Goal Success: An agreed set of objectives for an iteration.
Monthly or Annual Recurring Revenue per user (MRR/ARR): Ratio of product revenue generated per month/ annum to a number of total users. Customer Effort Score (CES): Evaluates the level of customer effort required to achieve their objectives, usually on a 5-point scale. Escaped Defects: A count or a ratio of defects that are uncovered by the customer per release.
Customer Acquisition Cost (CAC): Total cost of acquiring a new customer for the product. Adoption Rates: Measures customer adoption of the product over a period. Defect Density: Number of defects per line of code (or story point or per sprint).
Net Present Value (NPV): The discounted cash flow generated from the product minus cost incurred over the product lifetime. Feature Usage Rate: A measure of which features within the product are used most often. Scope Change Rate: Ratio of additional effort included in the iteration scope vs. the original effort estimated.
Total Cost of Ownership (TCO): Total cost incurred throughout the product lifetime, including direct and indirect cost. Retention/Churn: Number of product users retained or churned during a specific period. Either metric can be used, but not necessarily both. Burndown Chart: Burndown chart indicates the difference between the actual and planned effort over iterations or releases.
Internal Rate of Return (IRR): Rate of return or the discount rate when applied equates the net
present value to be zero.
  Team Velocity: A measure that evaluates the total number of story
points delivered per iteration.
Time-to-Market (TTM): Time taken for the product, set of features, or single feature to be available and usable by customers.    

To successfully use the insights provided by these metrics, a framework is needed that aligns the signals from these metrics to assess the product goals. Practitioners can use the HEART framework for this purpose.

HEART Framework

HEART is an acronym that stands for:
  • Happiness
  • Engagement
  • Adoption
  • Retention
  • Task Success
HEART assesses customer experience and value. It consists of customer goals categorized into areas that are assessed through signals that outline a specific insight based on metrics.

For example:
Framework Goal Signal Metric
Happiness Improve customer satisfaction Reduction in the effort spent on completing a user
goal
Customer Effort Score (CES).

Frameworks allow POA Practitioners to organize and plan how the product needs to evolve by providing a structure to think about the key metrics.
POA Domain  
Applying Foundational Concepts
  • Metrics and KPI's provide direction for the product.
    • Strategic metrics play a significant role in measuring the alignment of the product to the business goals.
    • Product metrics signify if the product is poorly defined or not.
    • Delivery metrics signify the effectiveness of the product team.
Cultivate Customer Intimacy
  • Since the customer pool can be large, individual
    feedback or surveys might not completely assess how customers perceive a product.
  • Metrics such as NPS, CES, Adoption Rate, and Retention Rates are used to better understand the customer perspective.
  • Although metrics indicate a measurement, the analysis or insights from those measurements require the Practitioner to utilize their knowledge of customer groups and personas, to derive useful insights.
Engage the Whole Team
  • Strategic metrics are commonly adopted by the:
    • Product team,
    • Business leaders, and
    • The product management functions where the POA Practitioners have a significant stake.
  • Product metrics can impact several areas in an organization. For example, they may prompt the team to focus attention on evolving specific features and drive marketing decisions.
  • Delivery metrics must be debated and agreed- upon by the product team. All involved stakeholders need to take part in determining such measures.
Make an Impact
  • Typically, metrics are measurable and create a direct and evidence-based impact.
  • Practitioners should
    • Avoid introducing vanity metrics that only embellish the ongoing work, and
    • Focus on the value delivered.
Deliver Often
  • Metrics that are usually predetermined and Agreed-upon are measured as often as needed to provide direction to the product team.
  • Depending on specific scenarios, some metrics may get prioritized or deprioritized. For example, a short-term metric, such as Churn Rate, may be ignored when the cost of acquisition rises because of a reprioritized strategy.
Learn Fast
  • Practitioners are required to understand metrics and adapt the following as needed:
    • Product goals,
    • PBI priorities, and
    • Effectiveness of the team.
  • A framework like HEART may accelerate the learning process.
Obsess About Value
  • Metrics are one of the ways to evaluate product value and direct customer feedback.
  • Primary and secondary analysis may also provide significant insights to improve product value.
  • Some metrics may provide conflicting insights that require the team to assess how best to proceed.