Embracing speed and utility in data analysis
Directional analysis needn’t be confined to the rigor of traditional statistics. Data misuse is a problem, to be sure, but so is over-analysis!
If you torture the data enough, nature will always confess.
— Ronald Coase, Economist, Nobel Prize Winner
So true. (If you doubt me, read this— great book. or this or this). Data “torture” is manipulating data to yield the results you want, even if the data tells a different story. Statisticians use incredible rigor to try to prevent manipulation.
But data “torture” can be an effective tool if the analysis is meant to be directional. Directional analysis yields quick results — telling the story of the data, but focuses less on exactness and more on speed. Said another way:
Will you make the same business decision whether your analysis says $5.1 million or $5.3 million? Probably so.
In these cases, you can alter your approach to the most useful calculation — adding utility at a small cost to precision.
Here at Granify, we track the relative opportunity our sales team influences through an internal metric called “CSR.” Avoiding the boring specifics, the equation uses a metric called “split” which varies from client to client. Historically, we applied a weighting (by client revenue) to split. See walkthrough below:
- Average split would be 62.5%. Lame, considering 1 client is bringing down the whole pack.
- For weighting, sum total revenue ($300 in this example) and each weighting value is the specific revenue of that merchant, divided by the total revenue. (column “Revenue / Total Revenue” above).
- To generate the Incremental value, multiply the metric you are weighting (“split” in this example) and the revenue weight.
- Sum the incrementals to produce the weighted value. Ta da!
- To get CSR, we multiply this weighted split against the rest of the CSR equation.
With the example above, notice that Client B no longer brings down the average, or at least only a little.
Revenue weighting is great:
- Our equations reflect “true-er” values by emphasizing the relevant data
- Results are more aligned with expectations
- Result are intuitive
But there’s a dark-side as well…
- The equation already kind-of accounts for revenue through related figures… so now we’re weighting it twice, kinda?
- With it weighted, you cannot “dig in” at a more granular level.
- The weighting creates a weird “incremental” that doesn’t make sense on it’s own. These incrementals must be summed for the final result.
To summarize, the weighted approach aggregates the values FIRST, then completes the rest of the equation.
So I replaced the weighted approach with a granular approach- meaning, I calculated CSR at the merchant level, THEN aggregated all the merchant values together: flipped the order of operations. What this does is add actual client-level metrics that make sense for each client website individually, though now we can’t weight the split.
No big deal though — the end aggregation matches within ~5–15% of the weighted calculation. While certainly notable, the use case will produce the same business decisions.
A granular approach is more useful, with an equivalent end result aggregation. This brings me joy.
Analytics is regularly compared to storytelling, and for good reason.
The approach is an important, though sometimes overlooked or oversimplified, part of data storytelling. The adoption of any analysis is dependent on how the message is received. The message itself is entirely dependent on the approach, the data, the use case and the audience.
The best approach for new and existing analysis is also not always obvious. Build stories from the foundation up. Challenge existing methods. Clearly identify what precision is necessary, and how the data will be used. If it’s for billing, be accurate to the cent. If not, maybe you have more flexibility in your approach.
Look for opportunities to improve data ingestion at your company, or come work at Granify and we can work on data together!
And Comment- Please share how you have sped up your analysis, or if there are other relevant tips and tricks you have picked up. Thanks!