What’s your vision? Which are your business hypotheses? How would you validate them? – These are the questions that I’ve been addressing a lot lately to the teams I’m working with as an Agile Coach.
Coaching an organization that has adopted Agile for more than 5 years, I’m glad to see that the agile way of working has become the norm for most of the people there. Even if they were resistant to change in the beginning, the agile practices are now integrated in their day to day work in a very easy and natural way.
Yet, many product owners fail in creating products that truly address the customers’ needs. While many factors can conspire to cause this, there is an important one that we will explore in this post – data.
This is not meant to be only a theoretical article, but a guide for all the agile minds who seek a source of inspiration for their products.
In 2011, Eric Ries has revealed – with his Lean Startup movement – a whole new approach for creating and launching great products. He helped us understand how the Build – Measure – Learn loop can prevent us from investing energy and money in building something that nobody wants. Easy to understand, inspiring and intuitive in the beginning, the philosophy behind the Lean Startup way requires focus and a data driven mindset when applied.
Is it a big deal about being data driven?
We all agree that measuring stuff in order to get more and more data is easy. On the other hand, to be data driven means to get the right data, which provides you with the right information, that trigger the right course of action.
Lean talks a lot about eliminating waste from our processes. It’s the same with the data. Measuring something that doesn’t help us at all is waste and we have to stop doing it. While it’s nice and easy to track many metrics, it’s also a sure way to lose focus.
Lean processes need lean analytics
Lean Analytics takes the Measure part from the loop to a whole new level. Use it to measure your real progress. Use it to ask yourself the right questions, which will reveal the right answers very fast.
There are 5 things that we need to consider when we want to define a good set of metrics.
Qualitative versus quantitative metrics
We all have an appetite for quantitative data; it is easy to aggregate, it involves statistics, models, and easy to understand graphics.
Yet, quantitative data is hard to use when the level of uncertainty is high. When you’re craving for a direction, you need to talk with people, to understand their potential interactions with your product, to get insights about the problems they are facing.
Quantitative data don’t tell what the reason behind a 1, a 3 or an 8 is. You need to walk up to people and find out why.
Though, there is a tricky part when it comes to qualitative data – it is always subjective, imprecise, unclear and emotional.
Get yourself ready when you discuss with your potential users, define a set of specific and objective questions and don’t let your opinions and enthusiasm ruin their answers. If so, the only person you’re lying is yourself.
Ask experience based questions to identify customers’ needs, not only questions that relate to the future. The second category may be misleading because the answers can reflect responders’ aspirations, not their real actions.
Imagine you want to open a restaurant with healthy food and you go ask people about it. Honestly, who would declare that they don’t want to consume healthy food? In reality, though, what do they really consume day by day?
Vanity versus actionable metrics
If you want to confront the inconvenient reality and follow the right direction, define a set of metrics that trigger some actions, some changes in your business model.
If you want an ego boost, define some vanity metrics – for example, number of total users, number of page views, number of total purchases/downloads or number of followers. If you are in the situation of measuring these metrics, have you ever asked yourself what would you do differently based on these information? Are those metrics a real reflection of your success? I don’t think so. Vanity metrics are the ones that can only improve over time. They make you feel great, but they don’t help you inspect and adapt in a proper way.
Are those people useful for your business? Are they your target customers? Or maybe they just used your product once and vanished forever.
So, instead of measuring page views, think about measuring the time that your users spend on page per month; Instead of measuring total number of buyers, analyze how much money do those buyers spend per order, how many items, what is their upgrade rate? Instead of counting your followers, consider counting their engagement per post.
The actionable metrics are the ones that help you understand if something goes wrong. For example, total number of users acquired over a specific period of time, when you performed some specific changes. If the number of users acquired is less than when you performed another changes, is pretty clear that you are not going in a positive direction. Actionable metrics enable you to iterate, to experiment, to understand customers’ behavior and needs; they help you understand where to spend your money.
Be careful with what you establish as metric. What you want is what you get.
Exploratory Versus Reporting Metrics
Reporting metrics refer to the information that we know we need and we calculate them all the time. For example, number of total users acquired over a specific period of time, percentage of active users, revenue, etc. These are the known unknowns – we know we need them, so we calculate them.
However, these metrics are not enough in a word that is changing all the time, when the competition is high and all we have to do is to continually find new approaches, to come up with innovative and disruptive ideas. “It’s what we think we know that keeps us from learning.” Innovation is about unknown unknowns. It is about exploring, simulating, finding something that we didn’t know before – here we talk about exploratory metrics.
Exploratory metrics are about digging into data, finding patterns, understanding what the most engaged users have in common, what else do they need? How can we adapt our business based of those people needs?
Exploratory metrics are a great source of pivoting our business for better results.
Leading Versus Lagging Metrics
Leading metrics are used to predict the future, while lagging metrics are used to analyze the past. Both of them can be actionable and can help us to take decisions, but leading metrics contribute on minimizing the waste, reducing the risk and help us to adapt faster.
Let’s consider a very easy to understand example. A very familiar lagging indicator is the weight loss. If you want to measure it after a period of time, you step on a scale and you find out. If the results are unsatisfying, they will trigger an action. But, by the time you find out, it may be too late. In this case, the leading indicators are very simple – number of calories taken and number of calories burnt. Leading indicators are easier to influence and help you adapt in time.
Now let’s translate this into business.
You may be in the situation of counting the number of people that stopped using your application in a specific period of time – this is a lagging indicator. If this number is higher than usual, is clear that you are doing something wrong and you need to improve in the future. However, the users are already gone and, most probably, they will not return. A leading indicator, in this case, can be the number of customers complains. This is an indicator that you can influence in time, before is too late.
Correlated Versus Causal Metrics
Data can be powerful, but also misunderstood and misleading. Not understanding the difference between correlated and causal metrics can be one of the most dangerous things when it comes to decisions, yet one of the most common mistakes.
According to the Bureau of Statistics correlation is “A statistical measure that describes the size and direction of a relationship between two or more variables”. On the same time, causation “indicates that one event is the result of the occurrence of the other event”. Basically, correlated ate two metrics that happen to change together; on the other hand, if one metric causes another metric to change, they are causal.
Mixing up these two concepts – correlation and causality – may lead to hasty conclusions like: staying in school is the secret to a long life, a diet of fish leads to less violence, bad odors cause disease.
The common problem with all these conclusions is that that people consider two correlated trends and present them as causal. A deeper analysis presents this in a more realistic way. For example, the level of ice-cream consumption and the sunglasses sales change together. They are correlated. It doesn’t mean that they cause each other. Both are caused by the summer weather.
You can very easy get caught up in the excitement of assuming that you found out a pattern and only because it comes from data you think it must be objectively true. But you need to have the ability to step back, analyze the overall picture and avoid making this crucial mistake.
Another important aspect to keep in mind is that, most probably, you won’t find a 100% causal relation between two factors. There are many other factors that can cause something to change. You’ll find few independent metrics, explaining a percentage of the behavior of the dependent metric. But even a degree of causality is important.
If you have a big enough, relevant number of observations, you can run a reliable regression test in order to understand the impact of each indicator on the indicator that is important for you.
In conclusion, I would say that the above picture summarizes very well my thoughts.
‘They used to say knowledge is power, but now there’s Google – information is everywhere, and cheap.’
Is not the information that generates a great success, but how we adapt our business based on it. And this, in my opinion means wisdom.