Every so often, I’m asked by a junior analyst or aspirant data scientist for advice on how they could be better in field of data. Should they study neural nets? Should they go start on Kaggle or contribute to open source? Maybe make a portfolio?
I usually tell them one thing: “know your data”, because all our work as data scientists rests on that foundation. That’s the hill I’ll die on and one major reason why I even write this newsletter.
But another thing I really want to tell new people, but have a very hard time articulating, is the importance of doing “good enough” work. That speed and approximate solutions to pressing matters are just as important as giving rigorous results. I heard repeatedly from people that fresh graduates entering the workforce, and people who come out of academia have trouble un-learning the habit of always searching for “The Right Answer”. It takes them a couple of months (sometimes over half a year) before they adjust.
On the surface, explaining the concept sounds very good and reasonable, but I get stuck trying to give any sort of explanation on how to determine when something is “good enough”. It’s some form of arcane art that I don’t pretend to fully understand. All I know is that I’ve gotten better at it than when I first started over a decade ago, but am very likely to say the exact same thing a decade later.
What constitutes enough?
The true answer, to no one’s surprise, is “it depends”.
The level of rigor required when discussing an early idea is different compared to what needs to go into the pitch deck that will be presented to potential investors. The most important part of the skill is building up a giant dictionary of situations and learning what constitutes “enough” for a given situation and audience. As experience builds, you can make reasonable guesses at what would be acceptable in unfamiliar situations.
That sounds daunting, but it doesn’t mean that we’re eternally trapped in reinventing the wheel, retracing the footsteps of everyone who has come before. There’s ways to bootstrap the learning process at least somewhat. It's that learning process that I want to write about today.
Learning to predict what stakeholders need
At its core, the end user of data is usually a human trying to make a decision. If they’re satisfied that they have enough information to make a choice with confidence, our job is largely complete (within ethical constraints).
Within the context of business’s internal operations, there’s usually not even a need for things to be particularly polished or refined. More sales from split variant C is more sales, whether it’s been reported with a fancy automated dashboard, or some math done on a post-it.
So what are some of the things that I’ve learned about the learning process?
Treat it sorta like a lean startup/MVP based process
Bounds can be almost as good as actual estimates
Ask if something easy but inaccurate might be good enough before starting work
Ask what’s worked before
Treat it sorta like a lean startup
A few drafts prior to this article here, I went researching how other fields handled the “good enough” problem. Someone in the past 50+ years must have put some serious thought into this and wrote it down somewhere, right?
That was a bit of a mistake.
According to the Google ngram viewer, various forms of the Pareto Principle, “Good enough principle”, the 80-20 rule, etc. all started appearing in books in the 1960s. Those phrases don’t explicitly say “there’s a good enough point of diminishing returns”, but they all pretty easily imply it. That said, I couldn’t find anything useful on the internet to link and reference from that body of work.
A lot of what was available is vague, unfalsifiable, “it depends, you’ll know it when you see it” type stuff. Do the good enough thing and it’ll work, if it doesn’t work it wasn’t good enough. Totally useless stuff. Somehow, Wikipedia has a really sad stub for “principle of good enough” as a software engineering article and indicates ties to agile software development, along with similar nonsense articles. I’m sure there’s a weird story behind those phrases.
Searching for variations of “good enough” on Google Scholar is even more exciting. Papers from all sorts of fields pop up, from computing and machine learning, to psychology, econometrics, you name it. All full of people working on tough real-world applications of research, and needing to invoke the magic of a pragmatic “good enough” to simplify problems and analysis down to something manageable. Nice to know, but not helpful.
Finally, I started looking at the concepts of MVPs, minimum viable products, a term that grew in popularity around 2009 with the Lean Startup movement. MVPs pops up in startup talk and also in tech industry topics like agile software development. Surprisingly, this was where I found the most useful material for learning about learning what “good enough” represents.
A foundation of the Lean Startup methodology is aiming to minimize time and energy wasted on long development cycles that don’t involve testing and getting feedback from the end customer. Instead of spending a year building a full featured application before showing anyone, build an MVP that captures the core experience, show it to people, get feedback, and make adjustments.
There are some interesting guides for planning what goes into your MVP. There’s many stages involved in hypothesizing, testing, validating, and iterating on a product until there’s an MVP. There’s various opinions on what exactly constitutes an MVP, so that alone isn’t going to solve the issue.
The same set of techniques can apply quite well, in my experience, for figuring out how to maximize our impact-for-time-spent in doing data work. It’s too easy to start a large “correct” analysis when people might be perfectly happy with something much simpler. Without prior experience, it’s hard to know how comfortable an audience is with more uncertain methods.
Instead, it’s much more effective to chip away at the problem with easier, low-effort methods first, see if those results are sufficient. Get feedback as early as possible and adjust course.
Bounds can be almost as good as actual estimates
All else being equal, fast is better. An answer that’s mostly correct today is often worth more than a very correct answer next week. Decision-makers are making a risk trade-off calculation in their minds when deciding how to process data that’s of lower quality.
While more formal statistical estimates come with error bars, less rigorous analyses often don’t come with easily calculated ones. Luckily, people have the ability to do risk evaluations about bad data, so long as they’re informed about where and how things can go wrong. So if you can provide even a rough sense of where the bounds are, people will make use of that information.
For example, we know that our estimates of number of unique people who buy Wonder Widget #5 are under-counted, because we have a few wholesaler accounts that represent multiple customers. Just knowing that any number we get is an under-count (and never an over-count) can help.
Just the ability to say things like: at most we could have lost 100 customers last week. At the minimum we have gained 15 customers vs the other variation. That’s good enough for some applications.
Ask if something easy but inaccurate might be good enough before starting work
Usually there are multiple ways to measure something, some are decidedly more effort than others, in exchange for being more accurate.
I could fingerprint every single visitor to my homepage and figure out how many people have ever visited, or I could just count unique IP addresses and get reasonably close, to within maybe an order of magnitude or better. I could do a giant engineering project and wait to collect data, or just run a few Unix commands on the server logs. I obviously prefer one method over the other because I’m lazy, but it’s up to the stakeholder to decide if they’re OK with it. So the best thing to do is ask whether the inaccurate way is “good enough” up front.
My example is a bit exaggerated, so the choice is obvious, but I’ve personally been surprised when people are happy to accept a really quick and dirty estimate. It really doesn’t hurt to ask, especially if you can articulate the cost/benefit of doing it the harder way.
Ask what’s worked before
For more important meetings with higher level execs that you don’t have regular contact with and can’t get regular feedback, it’s always a good idea to get an idea of how that audience prefers to see information.
Some people prefer to walk through the base analysis and come to their own conclusions, others prefer to just hear the conclusions. Some people prefer a certain level of formatting and others don’t care nearly as much. Some will ask really pointed questions about specific measurement details, and others aren’t as concerned.
Instead of making a guess and making unavoidable mistakes, leverage people around you to get a sense of what is appropriate.