Coming up with a great set of metrics is an art in itself — they can lead to better Product Decision, makes it easier to align all stakeholder, and discovery around how your product is being used.

Each Product will have 3–5 Key Metrics, that can provide a holistic view of the product’s value to the end user. (Read more here). The key metrics should act as proxies to the business value the product is generating, and also be responsive to product changes. A good product company prioritizes features that optimize its key metrics. A great product company combines metrics data and intuition (developed over years after being in the market) to come up with their strategy.

Thanks to more understanding on how to lead Data Science Projects, most data science teams have now moved from research divisions to delivery orgs — that is, they are expected to add value to the company swiftly and not be stuck in research mode forever. Data Scientists now form a valuable part of most product teams and need to be able to convey their impact through well-defined metrics for their impact.

Can product value always be measured and analyzed?

Mental Model for thinking about Experiment Metrics

Now, you need to design your A/B experiment. What metrics are you going to track to understand the impact of the model and the behavior of different sets of users?

We will be able to define this if we understand the User Journey in the current product, and where he derives values from ML.

The Interaction Point

How to quantify users in the test group are finding the content to be more relevant at this stage? A simple metric here would be the Click-through rate, i.e, of the top 3 recommended, users in the test group are clicking more on it vs the control group.

The Value Moment

Moving the needle

Segmenting Users

I have definitely made a lot of assumptions for this post, but here I just wanted to introduce how we can think of metrics from a more product standpoint.

Subscribe to my weekly newsletter here-

Connecting the Dots.