In the last decade, the combination of greater internet penetration, higher number of connected devices, and cheaper and more accessible storage has caused a huge spike in the volume of user interaction data. This fueled internal demand within companies to unlock actionable insights from the explosion of data. As a result, product analytics as a function began to emerge producing the virtuous cycle of more A/B testing and experimentation leading to faster product iteration leading to accelerated development releases leading to compounding product growth. Increasingly, a company’s ability to compete and innovate on a product is driven by how successfully it can apply analytics to mine for insights across vast, unstructured data sets from disparate sources. In other words, the future belongs to data-informed companies.
In a series of documents over the next few months, we will provide deeper guidance on how to build great data-informed companies. There are several characteristics of great data-informed companies but it largely boils down to two — focus on impact and culture. In this document, we will provide guidance on the characteristics of data-informed product companies.
Achieving Full Potential of Product Analytics
Before the recent data explosion, products were primarily built using intuition. Intuition can be grounded in data but not necessarily in a structured form. Intuition comes from processing all of the available quantitative and qualitative data subconsciously. Because the data is limited to what you see and what you have learned and the fully scientific method of data science has not been followed, it generally has biases, and may result in flawed decision-making with significant risks.
This doesn’t mean that intuition has no place in product development. It remains extremely valuable during the early stages of product development when data doesn’t exist. At this stage, much of the approach to product development depends heavily on design and engineering and less so on data science. This was also true a decade ago when there was far less data captured and analyzed for even more advanced products. As a result, most early products were design and intuition driven.
With the explosion of data, companies began to find ways to effectively utilize data to drive impact and value. As Analytics rose as a function and started to show greater value, Data Science, Data Engineering and Data Infrastructure, became key functions within product organizations.
Early data work was all about counting numbers. (How many users do we have? What is our revenue?) There was really no science behind the data. The next phase of data analytics was automating the counting by building dashboards and visualizations. Most of this phase was about logging the right data, ensuring that it was of high quality and correctly defined, and creating the right pipelines and tools to automate all of these tasks in a scalable way. This was the essence of data infrastructure and data engineering functions.
The next phase in the evolution of data analytics was to ensure that the right products and features were being shipped. This is the advent of data science — applying the scientific method to product development. Experimentation was a big part of this evolution for two reasons: power of incrementality and failure of intuition. First, although we tend to seek big wins, success is more often about getting many small things right. Over time, incrementality and the power of compounding are what ultimately drive most product successes. Second, our intuition is very fallible, and in many cases, it is no better than random selection. In order for data-informed companies to see continuous improvement, a “test and learn” culture is necessary.
Finally, data analytics is invaluable not only for counting numbers, building dashboards and shipping products, but for helping to define goals, roadmaps, and strategies. Arguably this is the highest leverage provided by an analytics team. Facebook and Instagram provide good examples of this: Facebook Live, Instagram News Feed Ranking, and the “stories” feature would never have existed without the strategic insights driven by the analytics teams.
In a series of documents, we will shed light on how to build a world class data-informed company that can drive the greatest impact through data and thereby achieving the full potential of product analytics.
Characteristics of a Data-Informed Company
All companies have missions. Data-informed companies define metrics and goals to support their pursuit of the mission. They align the incentive systems (promotions, compensations, etc.) with their pursuit of these goals. In our experience, we have seen the strongest data-informed cultures exhibit the following characteristics — maniacal focus on impact and building a culture of measurement and truth-seeking.
Define Impact
It is hard to define impact without understanding and defining product success. Setting the right goals and metrics helps quantify success of a product. With this in place, one can objectively begin to quantify impact too.
Great data-informed companies are organized and propelled by a unifying metric. A unifying metric is a single, actionable, top-line metric that encapsulates the vision for a product. It should be easy to measure and clearly connected to the business’s core drivers. Without a unifying metric and a related goal, it is very hard for a company to become truly data-informed and achieve its highest potential.
However, simply having a unifying metric is not enough. How it gets used matters just as much. It must be deployed by senior leadership to determine and inform what goals and metrics matter. Empowered with this clarity, each individual in the company is able to prioritize their work thus improving productivity and reducing needless debate.
After established what impact means with the help of a goal, the next step is to ensure that we are able to measure impact. Without the instrumentation for measuring impact, one would be flying blind and would not truly know if they are making progress to the goal.
Measure Impact
Impact can generally be broken down into three categories:
- Moving a metric: Say your team has a monthly active user (MAU) goal. When the team identifies creative ways of increasing MAU and executes against those ideas, then they are creating impact by moving a metric.
- Influencing a product or business change: Say your team has discovered, through deep exploratory analysis, that focusing on SMS notifications on Android in India would help increase MAU. Influencing that product change is a way of creating impact.
- Influencing a process change: Say your team has identified a new way to automate or scale, doing more with less. This is creating impact by influencing a process change.
While these are not mutually exclusive or exhaustive, they do serve as good guidelines for measuring impact.
Measuring the movement of a metric requires an agile infrastructure and a culture of experimentation.
Create Agile Infrastructure
A great analytics team comprises of quantitatively minded people (analysts, data scientists), but also requires data engineers (DE) and data infrastructure engineers (DI). Engineering talent build scalable infrastructures ensuring speed, reliability, and useability. A good infrastructure is agile and adaptable for future demands. Companies whose infrastructures are only optimized for short-term needs generally fail to scale.
Most data infrastructure involves some amount of data acquisition (logged product events, sales, external data sources, etc.), transformation of the data into a usable format and loading the data into systems that power reporting, dashboards and ad hoc analysis. The visualizations in the reporting and dashboards are commonly the ‘report cards’ of the business and should be watched carefully. The purpose of ad hoc analyses undertaken by the analytics teams will frequently be to understand any deviation in the metrics against expected values.
DI groups own the underlying computing machinery and the DE group is responsible for moving data between these systems and working with the analysts to productionize repeated analyses.
Without well-thought and well-constructed infrastructure, teams will be, at best, delayed in their ability to understand phenomena impacting their business, or at worst, not be able to understand them at all.
Have a Culture of Experimentation
While intuition is incredibly valuable for building products, it does not scale. Data-informed companies need to establish a strong test and learn culture, codifying intuition into hypotheses and experimental design. The underpinning of this test and learn methodology is that small improvements every week compound in way that results in much greater impact than large gains infrequently. Not only is it harder to identify ideas that result in huge gain, it does not scale either. As a result, experimentation to drive impact has become common in all highly data-informed companies.
Amazon and Facebook are good examples of how a culture of experimentation allows for hard data to inform product decisions. The two statement below by Jeff Bezos and Mark Zuckerberg established the importance of experimentation to build products.
“Our success at Amazon is a function of how many experiments we do per year, per month, per week, per day.” — Jeff Bezos
“One of the things I’m most proud of that is really key to our success is this testing framework.… At any given point in time, there isn’t just one version of Facebook running. There are probably 10,000.” — Mark Zuckerberg
Have an Expansive View of the Analytics Team
In order to build a truly great data-informed company, analytics must be involved at every stage of product development and needs to be embedded within the product team. At the outset, the analytics team should help craft the relevant metrics to your product’s success, measure progress continuously and help identify risks and growth areas for the business.
Moreover, their data-driven insights must be a constant input to product development. Generally the core product team consists of Engineering, Design, Product Management, Analytics and User Experience Research. Here is how a product team generally functions in many companies.
- Engineering writes code;
- Design makes the product easy to use and look good;
- Product management is responsible for vision, strategy, and execution;
- Analytics sets goals, creates a roadmap, and determines strategy; and
- User experience research fosters understanding of user behavior.
It is important to use the analytics team for the highest leverage problems and have them produce the greatest impact — ensuring that they can help drive goals roadmap and strategy for the product.
Have a Rigorous Roadmap Process
Fast product development, testing and iteration require an efficient roadmap process. For example, the growth team at Facebook has a product building process that lasts 10 weeks. This entire cycle has three phases: understand, identify, and execute.
In the understand phase, the analytics team conducts deep exploratory analysis on a range of phenomena (e.g., advertiser churn) with the primary goal of creating a roadmap. The focus of this analysis is to identify the issues and opportunities that will lead to the greatest impact. Once the team understands the biggest areas for impact, the entire product team works together to identify a roadmap.
From the roadmap, a list of initiatives, tasks for specific functions, Finally, the team moves into the execution phase. This is the last,and longest, phase of the 10-week cycle.. This process is invaluable for fast iteration and optimization, but it’s worth noting that the emphasis of the team is to move fast and not necessarily produce innovation.
Hire Well and Empower
Product analytics is a nascent function and is still evolving. Both data science and data engineering are continuing to find their feet and evolving their vision in even in the most advanced companies. As a result, building a world-class team is harder in analytics than in other functions as there are very few great leaders in this function.
Having said that, building world-class organizations is the only way companies can sustain themselves over decades. Most companies do not go under just because of a lack of ambition or vision; usually poor execution is a root cause. A huge part of sustainable execution is building amazing world class organizations.
Three key dimensions are chiefly valuable to consider when building a world class organization — people, culture and process. Driving impact by hiring A+ players, empowering them to do great work, and mentoring them as future leaders by showing them purpose, setting a bottom-up culture that empowers people to high levels of excellence, building a strong transparent organization that has accountability, ownership and trust as its core values and that focuses on company first, business unit second, team third and individual last.
Final Thoughts
Even with a great infrastructure, a culture of experimentation and having the right processes in place, if the leadership does not fully embrace building a data-informed culture, then it is very hard to fully utilize the value of analytics.
As an example, at Facebook, as the company evolved to be data-informed, “code wins arguments” became “data wins arguments”. Product reviews evolved from whiteboarding user flows to debating the problem statement and agreeing on the metric that will define success. The “identify with data and treat with design” is how products improved and evolved. Analytics itself evolved from counting numbers to building dashboards to experimentation to setting goals, roadmap and strategy for the product. All this was only possible because the leadership fully embraced analytics as central to the product.
During this evolution, a scientific method of hypothesis testing was devised, and a scientific method for setting goals and creating roadmaps and strategies is continuously evolving. This continuous improvement is at the heart of building great products, and a data culture that supports this is imperative. Members of the product team need to know how to use data effectively, and to that end it may be worth training every individual on best practices for building data-informed products.
Common Pitfalls that Data-Informed Companies Face
- Senior leadership does not appropriately embrace a data-informed culture. If a company’s senior leadership does not believe or understand the power of data, it is nearly impossible to build an amazing data-informed company.
- Analytics is misused as a service function. Analytics needs to be deeply integrated within the product team and should not be a consulting wing. Analytics teams’ goals should be tightly aligned with the product goals and the success and failure of the team should align with that of the products. The most common mistakes include spreading analysts too thin by staffing them across multiple products, treating analytics as a service function to engineering or product management, and not embedding data scientists within product teams.
- The role of an analytics team is too narrowly defined. In the worst cases, analytics is treated as a data-pulling team. In contrast, analytics teams should provide substantial leverage in helping to define goals, build roadmaps and develop strategy for products.
- The data engineering team is weak. The best way to scale a data science team is to have an excellent data engineering team. When the ratio of data scientists to data engineers gets too high, data scientists often spend too much time pulling data rather than performing analyses that give valuable insights.
- There is a lack of a strong tools team. In order to scale the analytics organization, strong tools are needed. This includes data cleaning tools, ETL tools, scientific tools, analysis tools, and visualization tools. Without access to and integration of these tools, it is hard for an analytics team to make significant progress, let alone to scale.
Takeaways
For a company to make tangible and repeatable progress towards its mission, the entire organization must perform at their full potential. Being impact focused and having a data-informed culture are important for leveraging the most from an analytics team and building a truly amazing product.
This work is a product of Sequoia Capital’s Data Science team and was originally published on Medium. Chandra Narayanan, Hem Wadhar and Ahry Jeon wrote this post. See the full data science series here. Please email data-science@sequoiacap.com with questions, comments and other feedback.
