5.6 Learn Fast

5.6.2 Assess What is Delivered

Guide to Product Ownership Analysis

IIBA.org KnowledgeHub Guide to Product Ownership Analysis 5. The POA Framework 5.6 Learn Fast 5.6.2 Assess What is Delivered

POA Framework

5.6 Learn Fast

Speed and innovation are the two areas in the product landscape that…

After choosing the right set of metrics for the product, Product Owners must utilize the information obtained from these metrics. Any metrics used as KPIs must lead to the discovery of new insights. The challenge most Product Owners face is to turn the data obtained from metrics to action. This is due to the abundance of data and an ad-hoc approach to analyzing the metrics.

Each metric contains attributes that help in the product assessment:

A trigger,
A frequency of measurement, and
A level at which the insights are expected.

A trigger situates a metric in the product development lifecycle, which means a metric is only relevant when a situation or scenario presents itself.

For example:

A mobile product with a freemium model during the MVP/MMP launch phase:
- May be concerned with Daily Active Users (DAU) to assess the engagement of the product.
- May not be relevant during a stable phase when Revenue Per User (RPU) is more suitable.

The frequency of measure indicates how often a metric will be measured.

For example:

A delivery metric like escaped defects can be evaluated each time the product is released to the customers.

A strategic measure highlights a fact about the strategic posture of the product.

For example:

ROI may provide an idea about the expected returns from the product which provides insights on the financial perspective. Other metrics may support why the ROI may be high.

For example:

An increasing trend in the Adoption Rate, which is a product metric, may support the ROI numbers.

Therefore, analyzing at a strategic level does not always equate to just analysis of strategic measures.

Based on the outcome expected, Product Owners can assess various metrics, to perform:

Strategic Assessment,
Market Assessment, and
Delivery Assessment.

The scientific treatment of data from various metrics should be considered, to generate insights when conducting an assessment, where decisions about the products are evidence-based.

When a strategic assessment is conducted, the insights are at a level where the product's business or customer value, is evaluated against the:

Product vision,
Roadmap, and
The strategy.

Multiple metrics are involved while deriving the actions that will help increase value. It may involve a combination of:

Strategic measures,
Product measures, or
Delivery measures.

How POA Helps Strategic Assessment

When a strategic assessment is performed, the insights may provide indicators for the business goals of an organization.

For example:

The assessment may reveal that revenue generated from the product has decreased, which is observed from a stagnating trend in Monthly Average Returns (MAR). The team may choose to:
- Reorganize the product roadmap to focus on monetization of certain features within the product, or
- Change the product strategy to focus on improving market share rather than revenue.

POA Techniques for Strategic Assessment

Agile Extension Techniques

Kano Analysis: Understand which product characteristics or qualities will prove to be a significant differentiator in the marketplace and help to drive customer satisfaction.
Real Options: Determine when to make decisions. It is useful in determining the flow of the product backlog and the priority of PBIs, against strategic measures.
Value Stream Mapping: Provide a complete, fact-based, time-series representation of the stream of activities required to deliver a product. It can be modified to assess whether the intended value is provided, and the overall strategy is sound.
Visioning: Assess strategic measures against the overriding vision.

BABOK® Guide Techniques

Balanced Scorecard: Manage performance in any business model, organizational structure, or business process. It helps with aligning backlog items to objectives.
Benchmarking and Market Analysis: Evaluate the efficacy of strategic capabilities against competitors.
Decision Modelling and Analysis: Enable considered decisions and their outcomes to align the product to the strategic goals.
Metrics and Key Performance Indicators (KPIs): Assess decisions or actions needed to course-correct using strategic and other measures.
SWOT Analysis: Assess the product and organizational strengths, weaknesses, opportunities, and threats to generate strategic insights.

Case Study: Strategic Assessment - Retailer

Background
Super C has two lines of business that involve picking up items at the store that were ordered online, but the items were sourced differently:

The grocery orders were sourced at the individual stores, where store employees would physically pick the items ordered off the shelves to deliver to the customers at their vehicle.
General merchandise orders were sourced from distribution centres and shipped to the store for customers to pick up, but still delivered to their vehicle by hand by store employees.

Challenge
When reviewing KPIs, Super C noticed:

Adoption of picking up orders at the store that had been ordered online had stagnated,
Online ordering of grocery items had skyrocketed.

Management expected in-store pickup of all online orders to grow, and they wanted to understand why one type was flourishing while the other was stuck.

Gary, the Product Owner for the in-store pick up of general merchandise, reviewed product assessment to see if any strategic changes were needed to move the dial.

Action
A month earlier, Gary had the product development team add a free-text option to the Net Promoter Score (NPS). Customers could complete it every time they picked up items ordered online. The team was surprised that many customers filled in the reason for their NPS rating, which anecdotally indicated to the team that customers wanted to like the service, but that something could be improved.

In studying the comments received, customers complained that they:

Had to wait for "normal" shipping time for online orders to reach the store for pick up but could get their online grocery orders the same day they ordered.
Did not like having to visit two different websites to order groceries vs. general merchandise.

Gary shared this information with the upper management of Super C. What was normal to the executive team - that Super C had multiple lines of business operating separately, specifically ordering groceries or general merchandise online - was not at all obvious to customers. Customers thought Super C was "just" Super C, whether shopping in a store, online for groceries, or online for general merchandise. Using two different interfaces to interact online with Super C was not resonating with the customers.

Outcome
The separate development teams for online grocery and online general merchandise came together as a larger team focused on closing this gap. Working with executive management, they performed some benchmarking against three other major retailers operating multiple channels of sales to customers to see what lessons they had learned and how their customer experiences varied from Super C's.

With this information, Super C was able to:

Adjust their strategy with the goal to unify all online channels,
Solve the problems general merchandise experienced,
Leverage what was going well with online grocery, and
Apply the best practices in the industry.

Lessons Learned
While metrics tell a story, they do not necessarily tell the whole story. They are very good at showing where opportunities for improvement exist, such as this example where similar lines of business were performing differently.

Gary realized that the product he owned (in-store pick up of online orders) was relied upon by upper management for the strategy of the company. As a Product Owner, he was able to assist with the strategic assessment of the product, which ultimately led to a strategic change across multiple business units.

A market assessment of the product involves understanding how customers react to the product, and what customer actions drive a better experience. This assessment may involve insights from multiple metrics that prompt the Product Owner to change the product scope or features.

How POA Helps Market Assessment

POA is used to continuously assess how the delivered product is providing customer value, which requires generating insights by using multiple metrics. The insights obtained while conducting customer assessment involve multiple customer perceptions about the product, including:

Usability
Experience
Desirability

Although these intangible parameters are quantified through metrics such as NPS, the resulting product changes must be derived through this assessment by the Product Owner. The PO must use multiple techniques, such as:

Observation,
Orchestration, and
Experimentations

To better design, such as:

Features,
Transactional experiences, and
Value of the product.

POA Techniques for Market Assessment

Agile Extension Techniques

Kano Analysis: Understand which product characteristics or qualities will prove to be a significant differentiator in the marketplace and help to drive customer satisfaction.
Value Stream Mapping: Provide a complete, fact-based, time-series representation of the stream of activities required to deliver a product. It can be modified to assess whether the intended value is provided, and the product-market fit is sound.

BABOK® Guide Techniques

Decision Modelling and Analysis: Enable considered decisions and their outcomes, so that the market responds positively to specific product decisions.
Focus Groups: Generate insights by eliciting feedback from a specific customer group.
Metrics and Key Performance Indicators (KPIs): Inference decisions or actions needed to course-correct using the product and other measures.
Root-Cause Analysis: Evaluate the causes of irregularities in the market indicators.
SWOT Analysis: Assess the product and organizational strengths, weaknesses, opportunities, and threats to generate insights to improve customer receptiveness to the product.

Business Data Analytics Guide Techniques

Exploratory Data Analysis: Generate market insights that are backed by strong evidence.
Hypothesis Testing: Scientifically verify an intuition, or market phenomena, that has a bearing on the product.

Case Study: Market Assessment - Food Manufacturer

Background
As a manufacturer of poultry products and pet food, Poultry Plus operates under regulation by government agencies to ensure the safety of food that goes to consumers. Tests are performed at various stages to validate compliance with regulations, as well as quality standards provided by customers whose products are being produced. When a foreign material, such as small metal particles, appear in food, either upon testing or visual detection, a series of processes kick off for investigation to correct the situation and mitigate a similar risk from occurring in the future.

Challenge
Poultry Plus has recently implemented a home-grown application for recording and tracking incidents where foreign materials are detected within the food products produced at their plants. The market for this tracking application is the Quality Assurance organization. There are various access roles for users of the application.

QA Operators can enter:
- Details about the foreign material detected, and
- The circumstances under which it was found.
QA Managers can:
- Add notes,
- Make updates,
- Enter actions taken to rectify the situation, and
- Mark an incident as "resolved."

Prior to the application's existence, the process of tracking was done with a series of entries in spreadsheets and emails, which was a very inefficient process.

The new application streamlined the initial capturing of incidents. However, the QA Managers did not enjoy the same efficiencies as the QA Operators and soon their investigation and follow-up activities moved back to manual tracking in spreadsheets. Soon they were wondering why they had agreed to use this new application.

Action
Product Owner, Jenn, used root cause analysis to determine the underlying reasons for the QA Managers' ultimate rejection of the new application. She worked with the QA Managers to fill in a matrix with details about why they were not using the application.

The reasons fell into a few common categories:

Data missing from the application,
Inability to correct data entry errors,
Inability to determine who worked on a particular resolution, and
Inability to add more information after the incident was marked "resolved."

Next, Jenn collected ideas from the QA Managers on how these concerns could be addressed and worked with the development team to discuss what options may be available.

Outcome
This assessment resulted in a new set of user stories prioritized ahead of those already in the product backlog.

Jenn and the Analysts on the team held a workshop with the QA Managers to

Design their ideal process and prototype including where additional fields could be placed.
Revisit the access roles, agreeing to add a higher level of permissions so that resolved incidents could be updated after the fact.
Add version history of the incident report, to include:
- Who made the entries and/or updates,
- When they were made, and
- What updates were made.

Once the enhancements were developed and implemented, the QA Managers adopted the use of the application since all their needs had been met. There were still some desired refinements, but those remained in the backlog for addressing as the processes and the supporting application evolved.

Lessons Learned
When the QA Managers initially rejected the use of the new application, Jenn realized that she had not paid close enough attention to her "market" as a whole. While she did a good job of capturing the needs of the users who entered the incident reports initially, and the in-the-moment investigation that followed, she did not assess what might be needed after an incident was resolved.

An important step in the development process is to consider market needs all the way to the end of the journey.

The delivery metrics are a good starting point for delivery assessments since they are tied to a team's delivery objectives. They must be unique for the team responsible for product delivery. For a delivery assessment, the product team collaboratively sets up the delivery goals, and metrics are defined around the goals. Periodic assessment must be conducted to learn whether delivery goals are being met:

Schedule,
Quality,
Cost, and
Value from each delivery.

How POA Helps Delivery Assessment

The Product Owner is the custodian for tracking the changes to delivery metrics. When the delivery metrics are defined, the assessment is conducted for a metric, or a group of metrics, to determine whether delivery goals are being met.

For example,

If an upward pattern is shown in a Value per Effort metric, the Product Owner may deduce that each delivery cycle is producing value for the customer. The "value" in value per effort is defined and agreed upon when the delivery measures are defined.
A metric where the average age of a story is tracked may prove to be a good indicator of whether the product backlog is sufficiently groomed. An older story may indicate that the Product Owner needs to improve the stories and acceptance criteria so that the delivery team can include the stories for development.

BABOK® Guide Techniques

Metrics and Key Performance Indicators (KPIs): Metrics and key performance indicators measure the performance of product delivery. Various delivery and other metrics are defined, and they are used to understand the impact on delivery. Course correction required for streamlined product delivery.

Case Study: Delivery Assessment - Food Manufacturer

Background
Poultry Plus was on the verge of rolling out the new lifecycle management system to their pet food division, called OneMethod, a commercial, off-the- shelf product. Although it met 90% of their needs, there were certain functions of OneMethod that required customization or configuration, mostly related to how the data collected in OneMethod would be integrated with the company's ERP system, Higher Vision. The requirements for those customizations were approved and the necessary configurations to the system had been made. The Product Owner, Kavika, was working with the business stakeholders to schedule user acceptance and rollout activities that would hopefully conclude at the end of the month (on schedule).

Challenge
It became apparent mid-month that the business was not ready to move forward with OneMethod. The pandemic of 2020 led to business disruptions and the decision was made to err on the side of stability. The implementation was pushed out.

Higher Vision was implementing a new version, which would require significant additional development on the Poultry Plus side to make sure the data from OneMethod would integrate properly to Higher Vision. An implementation SME for Higher Vision accepted a new role. Management wanted OneMethod rolled out by the end of the year.

Action
Given all these obstacles, it was not a simple matter of just taking the existing implementation schedule and moving it out a few months to account for the pandemic disruption.

Product Owner Jenn had to take a few steps back and reassess the schedule.

She worked with the business stakeholders to assess how the operators were adjusting to the new ways of working, which had been brought on by the pandemic, and gained consensus that they should be able to resume, considering a training and implementation plan, in 60 days.
She worked with the Development Team Manager to determine when a new resource would become available to take on the effort of rewriting the Higher Vision integration based on the new version. A new resource had been identified, but he would require a period of onboarding and knowledge transfer. Most likely, he would not be ready to complete the integration work for 90 days.
Other SMEs recommended an additional two weeks of user acceptance testing after the integration development was completed. This time lag was to make sure everyone was confident about the system, given the general level of uncertainty that came with the pandemic.

Outcome
With all these new considerations, Jenn was able to line up a new schedule that, put rollout in January. Upper management was satisfied with this delay.

Lessons Learned
It can be disappointing when eagerly anticipated plans for a major implementation cannot be met, but it happens. Contingency plans are good to have, but a contingency plan that allows for a pandemic may be impossible.

When a major disruption to a schedule occurs, it is important to ensure that the schedule does not simply shift, but that all related new obstacles to the shift are assessed so that reasonable expectations can be set.