If you’re reading this newsletter, chances are you live in a highly developed economy, the greatest hallmark of which is the specialization of labor. Just about everything you touch, consume, and use has been created by a huge army of people and machines along a ridiculously huge, global supply chain. The homes we live in, cups we drink from, keyboards we type on, come fully-formed out of a box for out use. Unless we take it up as a personal hobby, there is very little reason for any one of us to build stuff on our own any more.
In general, this is considered a positive for society in that people who don’t have to spend time and energy understanding how their car, air conditioner, or surgeon works can better devote their energy into being good at some other valuable thing in society. Specialization is a bedrock principle of our civilization.
But (and of course, if I’m writing this, there’s a but), there’s some interesting perspective you can gain when you poke your nose into “other specializations”, however briefly.
For example, take this tech joke/meme:
Most of us would find the joke funny because we work in tech and data. We know just how messy and cobbled-together everything is behind most of these systems. Giant nests of if-then statements for edge cases, secret human reviewers doing the “AI” work, all the “magic” is just the thinnest coating of marketing hype and slick UI.
But think for a moment about how you treat all the other things in your life. There’s “complete stuff I use and don’t tinker with” versus “things I can change”.
If you break a car down, there’s all the stuff that provides the functionality — frame, engine, wheels, brakes, etc. — all of which is carefully arranged and safely packed away in body panels and trim that are largely cosmetic. To a normal consumer like me, the “car” that I interface with is entirely the cosmetic outside stuff. I don’t know how to mess with most of the internals and would rather not. It’s a “complete unit” unless I spend hours watching videos to replace a single headlight bulb. Meanwhile, more skilled people see through the illusion of completeness and know how to change and modify parts.
The only difference between the viewpoints is really just the knowledge of how cars are put together that allows one to see it as a collection of pieces versus a completed whole.
In another example, take the simple locks that secure the doors to your home. We take them for granted, but the basic principle they operate on, a key raising little pins to just the right height that allows the lock to turn, has been around for millennia. At some point in my late 20s, I thought learning to pick locks would be fun (locksport). For less than $100 investment in some tools and practice locks, I eventually learned that I’m a horrible lock pick. But it only took a couple of hours of watching videos to turn the ubiquitous locks in my life from these indecipherable black boxes to things that I can now regularly change the key combination for whenever I need to.
Data infrastructure is the same as everything else, moving bits under a UI facade
Turning back to talking about data, there are many out there who are coming from an a less coding, more pure analytics background who view all the tools that we use as these “complete” things. This makes the whole field seem significantly more polished and intimidating than it actually is.
Take DAG-based tools that are commonly used to handle data pipeline jobs, like Airflow or maybe Luigi. At the highest of levels, these tools are replacements for a set of janky shell script logic mixed with cron jobs that anyone can hack together on their own. The data community tends to use these DAG tools because the tools provide significantly better UI/UX, maintainability, and portability over home-grown solutions, but if you really really wanted to be bull-headed and go your own way, you could.
Most other data tools are similar in that they exist to solve a problem, and the techniques they use to solve the problem are surprisingly “pedestrian” at a high level. They’d do the things you’d probably do. Meaning you are more than capable of understanding them if you take some time to engage with the topic.
I should add that throughout all of this, I’m not being dismissive of “UI/UX”. I think a tool with a bad interface is inflicting pain at scale. To end users, the interface is everything. Bad interfaces make for useless, or even dangerous, tools. Imagine if you had a blender that had 6 buttons on it, except 5 of the buttons will cause the blender to burst into flame. Users rightfully have a strong preference for products that look good and are easy to use if functionality is equal — that’s why we tend to prefer certain data tools over others.
So if you’ve always been daunted by the fact that there exists all this “magical” data infrastructure out there that you might be relying on for work, go ahead and ask some people what it is and how it works. Break it down into it’s pieces just a little bit to see how it’s put together.
You’ll be surprised how obvious and (largely) reasonable all that infrastructure is in hindsight.
Standing offer: If you created something and would like me to review or share it w/ the data community — my mailbox and Twitter DMs are open.
About this newsletter
I’m Randy Au, Quantitative UX researcher, former data analyst, and general-purpose data and tech nerd. Counting Stuff is a weekly newsletter about the less-than-sexy aspects of data science, UX research and tech. With excursions into other fun topics.
Curated archive of evergreen posts can be found at randyau.com.
Join the Approaching Significance Discord, where data folk hang out and can talk a bit about data, and a bit about everything else.
All photos/drawings used are taken/created by Randy unless otherwise noted.
Supporting this newsletter:
This newsletter is free, share it with your friends without guilt! But if you like the content and want to send some love, here’s some options:
Tweet me - Comments and questions are always welcome, they often inspire new posts
A small one-time donation at Ko-fi - Thanks to everyone who’s sent a small donation! I read every single note!
If shirts and swag are more your style there’s some here - There’s a plane w/ dots shirt available!
I really like your perspective that substrata the mystery from the technology and tools. It makes me think about what other thing are out there, that seem to overwhelming to understand but really aren’t. It’s all about the mindset!