In a world of Metrics

Coming up with a great set of metrics is an art in itself — they can lead to better Product Decision, makes it easier to align all stakeholder, and discovery around how your product is being used.

Each Product will have 3–5 Key Metrics, that can provide a holistic view of the product’s value to the end user. (Read more here). The key metrics should act as proxies to the business value the product is generating, and also be responsive to product changes. A good product company prioritizes features that optimize its key metrics. A great product company combines metrics data and intuition (developed over years after being in the market) to come up with their strategy.

Thanks to more understanding on how to lead Data Science Projects, most data science teams have now moved from research divisions to delivery orgs — that is, they are expected to add value to the company swiftly and not be stuck in research mode forever. Data Scientists now form a valuable part of most product teams and need to be able to convey their impact through well-defined metrics for their impact.

Can product value always be measured and analyzed?

Yes. No matter how “fuzzy” or intangible the value may appear, you can apply quantitative measurements to any product, aka, use a proxy. For example, it is impossible to know how much value Medium adds to its readers, but a good proxy is how much time a user is spending weekly on the website reading.

Mental Model for thinking about Experiment Metrics

Suppose you are working with the Personalisation team at YouTube, and you have to come up with a baseline model for recommending the following section -

Now, you need to design your A/B experiment. What metrics are you going to track to understand the impact of the model and the behavior of different sets of users?

We will be able to define this if we understand the User Journey in the current product, and where he derives values from ML.

The Interaction Point

Where does the user interact with your model in the UI? Is it a new section? Or does the section already exist, and is going to now get powered using ML? Let’s assume the section already exists, and in the first A/B experiment, 5% of users get recommendations through the ML model.

How to quantify users in the test group are finding the content to be more relevant at this stage? A simple metric here would be the Click-through rate, i.e, of the top 3 recommended, users in the test group are clicking more on it vs the control group.

The Value Moment

Your product’s value moment is an event, an action, or a series of events and actions that represent the moment that a user found value in your product. Are the users finding actual value in the content they clicked on — this can be defined in terms of minutes spent watching, shares, likes, download, etc. A relevant metric here could be the average time spent by users watching recommended videos.

Moving the needle

All good on tracking online metrics for the model, but are these translating to the key product metrics that the stakeholders care about? Are the users in the test group on average more engaged — Are they spending more time on the platform? How is the ads revenue getting impacted?

Segmenting Users

Depending on the population chosen for the experiment, cohorts need to be created based on percentiles of activity (to identify power users), region-based cohorts/breakdowns by countries (there’s a lot of cultural difference in how people use the app as well as their data usage). Try to break it down as much as you can and try to understand the impact of the ML model on these different cohorts. Even in an experiment with flat results, a cohort with substantial users may have had a positive reaction. If we can correctly identify these users, we may be able to drive value for the business by rolling out the new variant selectively to the subpopulations that had a positive reaction. This can substantially increase the value of flat experimental results.

I have definitely made a lot of assumptions for this post, but here I just wanted to introduce how we can think of metrics from a more product standpoint.

Subscribe to my weekly newsletter here- aashaysachdeva.substack.com