Views on dashboards being answers, or not
It's messy because we're messy
I don’t normally write multiple posts about the same topic, but last week someone emailed in a question about last week’s post — “If dashboards aren’t the answer, what’s an alternative?”
And it of course got me thinking. A lot. Because there’s a huge amount to unpack in a seemingly innocent question.
The thread above had LOTS of people chiming with various opinions about what’s right/wrong with dashboards, as well as various opinions about potential alternatives and use cases. I’d recommend you check it out because the variance is too wide for me to cover today.
In terms of failure, various themes popped up:
#1 complaint is that no one looks at them, it’s often seen as wasted resources
Actions are rarely taken based on a dashboard
Dashboards often break and maintenance takes time away from other work
People ask why multiple dashboards disagree, which is usually a big time sink project
Instead today I’m going to put some thought poking at “what is it we’re asking our dashboards to do?” in hopes of coming to some conclusions about why we so often see them as failures.
Job #1 — dashboards help us know when to make a decision (situational awareness)
Since dashboards are like fixed panes of glass that provide a particular view out into the world, they’re generally consistent over time in what they show. This property of providing a single consistent view of the world in theory allows observers to notice that “hey, things aren’t heading in the right direction, we need to make a course correction.”
This is very much like the windshield of your car, it consistently shows what is in front of you and if you notice that you’re starting to veer into a tree, you make adjustments.
So far, nothing about the above job description, or analogy, should be particularly controversial. You might say that this is one of the textbook uses of a dashboard. The term “dashboard” itself comes from automobile instrument panels that provided information to aid in driving. (Okay, okay, prior to even that ‘dashboard’ was the name of a board that kept you from getting mucked up by the horse pulling your carriage. But I digress…).
But at the same time, I think most people would admit that the majority of dashboards fail at this job, because people don’t look at them. They’re too busy, distracted, or plain uninterested. Unlike our car dashboard and windshield, ignoring a data dashboard typically does not end in a horrific, life-threatening crash. The consequences are so low that people frequently get away with not looking at them.
This is why I suggested that every dashboard of this type should have a formal meeting built around it where people get together, look at the thing, and decide if they want to do anything about what the dashboard is showing. Because it’s not people’s fault that looking at a dashboard is uninteresting and low priority for them, we need to work around this fact and force the issue.
This job also overlooks the fact that most times, we only want situational awareness of a problem until it stops being a problem. Once we want to move on to the next business problem, the dashboard is at the end of is useful life.
Job #2 — We want “self serve” access to information for various people, so they can do stuff (access and exploration)
This overlaps with Job #1 somewhat, but I’m calling it out distinctly because the intent here is to give people room to explore and find stuff on their own, not just “observe a trend and decide there is an issue that needs raising”.
The topic of what “self-serve” means in a data/information context is a bit polarizing. We can all think of horrible examples from both ends of the spectrum, with people who could’ve used data access to avert disaster not having it available, as well as people who abused access to data to come to very bad conclusions.
So while some people might object to these sorts of dashboards, some will exist. There’s a number of problems with self-serve dashboards:
They’re never at the right ‘altitude’. They’re either too high level, or too detailed. Adding controls for users to tweak things to address this tends to make things exponentially more complex
Building and maintaining such monster dashboards can be a huge undertaking
People need to be trained on how to use the dashboard effectively. They need to be taught what’s normal, what’s abnormal, what the dashboard can'/can’t say. Who has time to do that?
I think if we have to be honest, the majority of dashboards wind up being created under this somewhat confused banner. The fact that we have to make this complex data product that is consumed by multiple people with different levels of interpretation skills, interest, and agendas already make it a big setup for failure.
On top of that, you can’t easily set up a “get together and look at the dashboard” meeting when it’s supposed to be this self-run exploration tool. So there’s not even a good hack to make sure these things are valuable.
The only time I’ve managed to make these self-serve dashboards successful was when I made them bespoke for a specific team, or more often, a specific data-savvy member of a team. That person was the interested stakeholder who would use the dashboard regularly to get stuff done and figure stuff out.
Job #3 — the dashboard is a specific tool, used for a specific purpose, and isn’t used for anything else (specialized tool)
Springing off of Job #2, there are a very specific set of dashboards that only rarely get created and are designed for a very narrow purpose. An example of this would be a dashboard that is used to analyze a very specific kind of fraud pattern. Such a dashboard would be ignored most days, right up until an incident is reported and we need to drag out the past analysis in the form of a dashboard to check.
My favorite real life example is perhaps a fire alarm control panel. They provide access to various firefighting related functions (reports of what fire systems have been triggered, where the fire could potentially be, control over various systems, etc.), and are only used by a small group of highly trained people, only a few times a year (mostly regular inspection, testing, and actual emergency). All other times, these panels are ignored by everyone.
In both of the above scenarios, we’d never judge them to be a failures despite their niches. They’re very actionable in the few times that exist. Sometimes they’re highly refined and well maintained (like the fire alarm panels) or they’re a bit sketchy (like the ‘can-of-beans dashboards’ I referred to last week)
The problem is that this type of dashboard is almost never built on purpose. They’re usually of the can-of-beans variety. Not great.
So, let’s consider some alternatives
Alerts for everyone!
One frequent attempt to replace some of the work that dashboards do is to switch from a “pull” framework where people have to initiate the data flow themselves (which they’re not motivated to do), into a “push” framework where people are notified when they need to pay attention to a metric.
Most of these solutions fall under various forms of “alerting” and “anomaly detection” work. As far as I can tell, it’s currently an area of active research, especially because of how we have all these dashboards we need to get rid of.
The biggest problem with such push systems is that it’s an extremely difficult problem. At the minimum, you need some system to look at historical values, then make some determination that the next incoming value is anomalous enough be worth bothering a human about. There are lots of methods for attempting to do it, but generalized algorithms can’t capture the contextualized nuance that humans use. My car has a lane departure warning that beeps if I come close to wandering out of lane, it generally works fine. But it also beeps constantly on certain very narrow roads, and definitely goes off if I change lanes/exit without using a turn signal. It just can’t know my intent and thus over-alerts me. In response, I’ve been trained to know when to just ignore it.
As time goes on, and AI/ML advances come, I’m sure we’ll get better at this technique. I haven’t seen what the state of the art can do for this problem. But unless I can get that state of the art tech working for my random dashboard metric in time for my next meeting, I don’t have the energy for learning it.
Make everyone data savvy
The whole “self serve” dashboard thing tends to fall apart for very human reasons — the users who are give the data aren’t really equipped to serve themselves. While this might seem like a hopeless problem, it’s actually possible to train large parts of an organization to have the skills (and environment) needed to self serve and participate in analysis. It just takes a ton of work, organization, and willingness both to train and learn for everyone involved. But it can be done.
If you have access to data savvy team, then you can actually open the self-serve floodgates even more and give people access to APIs and analytics datasets for them to play with. In theory this is what certain BI tools like Tableau and Looker aim to provide, assuming you have access to humans capable of using them.
Make fewer accidental cans of beans and more specialized tools
I’m very big on intent, because things usually turn out better when we create artifacts deliberately instead of by blind accident. This means that we (and by that, I mean myself) need to be much better at being clear that I’m making a dashboard tool where the only real user is going to be my future self when someone returns with a similar question.
Just admitting that up front will have an effect on how I name and save where the dashboard lives, as well as the types of features and explanations I’ll be including. And it’ll be ugly as sin because I won’t have to spend hours designing for a fictitious user.
There’s probably some more jobs, and solutions, out there
I doubt that I’ve managed to do a full enumeration of all the possible jobs that dashboards have been asked to do over the years. But it seems to be that if we break down what we actually want to accomplish, we can get away with not having lots of dashboards lying around, leaving only narrow slices of work to be done.
If only we were good at articulating these jobs up front.
About this newsletter
I’m Randy Au, currently a Quantitative UX researcher, former data analyst, and general-purpose data and tech nerd. The Counting Stuff newsletter is a weekly data/tech blog about the less-than-sexy aspects about data science, UX research and tech. With occasional excursions into other fun topics.
All photos/drawings used are taken/created by Randy unless otherwise noted.
Curated archive of evergreen posts can be found at randyau.com
Supporting this newsletter:
This newsletter is free, share it with your friends without guilt! But if you like the content and want to send some love, here’s some options:
Tweet me - Comments and questions are always welcome, they often inspire new posts
A small one-time donation at Ko-fi - Thanks to everyone who’s supported me!!! <3
If shirts and swag are more your style there’s some here - There’s a plane w/ dots shirt availble!