Original article: A method for measuring analytical work by Benn Stancil

Motivation: Data teams can measure virtually anything but not themselves. We fall back on flawed proxies such as experts’ opinions or the good and bad outcomes of an analysis. Often we can’t measure the result at all because there’s no way to assess the counterfactual, which in turn, makes it difficult for aspiring analysts to grow. As a first data hire, there’s no one to provide feedback or compare your ROI against.

What’s the solution?

“Analysts should judge their work by how quickly people make decisions with it.“

Using time to decisions as a measure of ROI

Here is how Benn defines the framework to measure the data team’s work:

“The moment an analyst is asked a question, a timer starts. When a decision gets made based on that question, the timer stops. Analysts’ singular goal should be to minimize the hours and days on that timer. […] The lower the number is across all the decisions we’re involved in, the better we’re performing.“

Benefits of that metric

• it’s directly related to the goal of making data-driven decisions

• the decision-makers can directly evaluate the impact

• the faster we deliver those insights, the happier our stakeholders

• if those stakeholders find those insights ineffective (conclusions based on inaccurate data and flawed reasoning), our work is likely not good enough

• it encourages the right incentives

  • it forces us to understand the problem we were asked to solve — knowing which decisions need to be made improves focus on important and planned tasks, rather than ad-hoc work

  • it makes us understand the operational context in which decisions are made

  • it mitigates the back and forth on when something is done

  • it emphasizes the value of communication and presenting the analysis results in a clean and digestible way

  • it helps with planning—we can build more concrete estimates or at least sense when things are taking too long when no decision gets made; it also detaches the value of analytics work from its outcome.

Potential drawbacks of that metric

• it might push analysts to advocate for their opinions rather than the truth

  • this would imply that the only evidence that matters in making a decision is one that we can articulate with a chart (which Benn argues is not true)

  • we can never be truly neutral — analysis will always be colored by small biases; we’re better off acknowledging our opinion than pretending it doesn’t exist

• it might encourage analysts to omit confounding arguments and present misleading results to make decisions quickly

  • even if someone would do that, such laziness and bias towards a specific outcome would most likely come to light, eventually harming the reputation of that data person

  • to whatever extent it creates bad incentives in the short term, it counteracts them by creating the right incentives over the longer term

  • “If you know the right path forward, do whatever it takes—cheat, lie, it doesn’t matter—to convince me to take it.“ — once you build a reputation that you make the right decision based on what you know about the business, not only what you see in data, that’s how you go from being advisors to executives: people simply hand the decision off to you.

Core message & CTA

The time to get from a question to a decision is a metric that creates the right incentives for data teams to provide value to stakeholders over the longer term.

“Ignore all the ambiguity around measuring analytical quality and ROI, and do whatever it takes to make others more decisive.”