Attention: As of January 2024, We have moved to counting-stuff.com. Subscribe there, not here on Substack, if you want to receive weekly posts.
Some time in what I can only assume is the ‘80s or ‘90s, someone had the brilliant idea of putting the gas line coming into my house right in the perfect spot under some brick stairs where a small crack in the brickwork would allow water to eventually drip onto the pipe when it rains a lot. Since it is a very big and expensive project to so much as touch a gas line with any tools, and attempts at stopping the water intrusion kept failing, there isn’t much that could be done about it other than mitigation.
Now, some 40, 50 years later, that “deferred maintenance” has come due and let’s just say that it’s been a stress-filled, expensive, experience. Winter is coming, and I really really want to have heat before it comes. So yes, for a large number of weeks now, the term “deferred maintenance” and it’s close cousin, “tech debt” has been on my mind in various ways.
In my day-to-day work, there’s a relatively low amount of tech debt that’s directly attributable to me. That’s not because I’m some genius at avoiding making tech debt, but because a vast amount of my work takes the form of disposable one-offs. Can’t cause tech debt if your code doesn’t even become part of the tech! My enduring work are primarily processes and frameworks for how people on my team do work. It’s not the sort of thing that fits into git and can be deployed into production.
But it seems to me that there’s always more process stuff to do. I’ve seen so many announcements of “we have a new way of doing XYZ” that I’ve started to notice that this new way is a rehash of an old way from 5 years ago that had been replaced by something else. It’s like watching a pendulum swing between centralized planning and decentralized planning, of tall vertical silos and versus cross-functional teams. Why is it that I’m effectively contributing to this churn?
Out of curiosity, I searched for the term “process debt”. That vaguely sounded like what I was dealing with. It felt something like the intangible workflow/process variant of technical debt. To my surprise, there seems to be a very tiny handful of papers using the specific term, most dating from 2020 and afterwards. So at least someone out there is discussing the existence of yet another class of problems that organizations are facing. I suspect that the organizational psychology people and management people have their own terms for this but I don’t can’t think of what it is.
Either way, since the language around the notion of process debt doesn’t seem particularly settled yet, I’m just going to write down what I think it means here.
Process debts are inefficiencies in how an organization does something. Like technical debt, they’re trading execution speed and toil over the long term for execution speed in the short term. Over time these inefficiencies become intolerable, leading to breakdowns or other issues unless dealt with.
Here’s a concrete example. Imagine you need to run a study and due to staffing constraints you need to hire an external research agency to handle all the logistics of finding and recruiting participants for you. You put out an RFP detailing the work needed, you consider the proposals from a couple of vendors, have legal review all the contracts about the scope of work, NDA, payments, etc.. You get executives to agree to sign the checks. It’s a big pain in the butt, and it takes over a month of wrangling before everything is in place to get things ready and recruitment starts. It’s painful, but works well enough for this one-time study you don’t anticipate ever doing again.
But then, people noticed that you ran this cool study and they want to do something similar for their own work. The first couple of times, people just copy the same process you followed, since it worked. They’ll borrow your existing contracts as examples, and they’ll even make incremental improvements to the process. But regardless, it’s still painful and slow because a lot of the work is still a one-off custom job.
Over time, this methodology becomes common. The process largely hasnt’t changed, but the company is doing ten of these a month. The amount of hours spent doing this work is skyrocketing. The obvious solution reduce the toil is to standardize the process. Have a standardized contract with standardized rules for every vendor. Have a vendor vetting process that sets a short list of pre-approved vendors. Have a set of pre-approved recruiting techniques and language ready. Maybe even negotiate a package deal at a discount rate with specific vendors.
This gap between what’s actually done and what should be done, is the sort of process debt that I’m thinking about. If we were all hyper-rational about our processes, we would’ve implemented the standardized process the moment other teams started asking about running their own.
The problem is that it takes extra time to put all this work together into the new process, and no one is really rewarded for standardizing everything. Everyone’s got a study they need to run right NOW, not in the 3-6 months it takes to standardize the process.
I’m sure all this sound familiar to everyone for multiple reasons. First this sounds almost exactly like how tech debt happens, just instead of code we’re talking about largely intangible “processes”. The other reason is because this sort of difficulty struggle to juggle short and long term ways of doing things is just so damn familiar in any work environment. I can’t count the number of initiatives, team shuffles, reorgs, and pivots I’ve seen over the years in order to try to overcome issues around how things are being done.
Process doesn’t “fix” easily
Stop for a moment and think about the patterns of work you engage in. How do work requests come in? How do you decide what to work on? Where are the places that you'd automate if you only had a week without distraction? I'm confident that there are places where you're experiencing a pain point but haven't solved it yet for whatever reason. That's unaddressed process debt.
It’s easy to just say “Well, someday the debt becomes due! Go work on all of it before it becomes a problem!” but I don’t think it’s that simple. As far as I know, there doesn't seem to be an objective “solution” to process problems. I’m pretty convinced that perfect processes can’t exist for more than a fleeting moment.
All processes, even the most perfect and simple ones, are in a perpetual state of becoming utter failures in the face of entropy and life. Your perfect, super quick process for recruiting participants breaks because laws or budgets change. The streamlined ML pipeline you built inevitably needs a refactor due to scaling or business changes. The work intake process that stopped all the repetitive dashboard requests breaks down because we hired too many people at once and someone broke the norm.
I think this constant state of falling apart is one of the reasons why we see those large pendulum swings between extremes. There’s positives and negatives to either of the extremes in how we do things, and we inevitably pick one side to focus on until its negatives become unbearable and then switch back and forth like a meandering river. We’re not doing this because we have no memories and forgot that we tried the other way before. We’re switching because we’ve gone as far as we can tolerate the shortcomings and now have to go and address those gaps that had been left behind.
This makes managing process debt an interesting problem.
If process can’t be “fixed” in any long/mid-term sense, I can sorta understand why a lot of organizations adopt the “it’s working now, just keep doing things the same way until it breaks” strategy. Since things are going to break anyways, let’s maximize the time between failures. It’s practical in the sense that the next reimagined process will be built to address the immediate issues that caused things to break. The cost is, there’s no predicting what the breakdown will look like. It’s conceivable that an organization gets so stuck in its ways for so long that it gets disrupted by the marketplace and somehow dies.
Other places try to be more proactive about handling process changes. They’re constantly looking at existing processes and asking whether they can be improved, or more importantly, need to be discarded in favor of something else. Being proactive like this has the obvious benefit of avoiding letting issues fester around for too long. The tradeoff here is that a lot of time and energy is going to be spent examining and working on process improvements that may not actually address everything that needs addressing.
At least with tech debt, there’s usually a clear reason why the debt exists and some expectation of benefits if the debt is paid off. Processes don’t even have those basic properties. It’s just trying to peer into the future and decide whether things will become unbearable sooner or later. Plus since the majority of process work is social, not technical, there’s all sorts of weirdness about implementation.
Just think of all the times you’ve had a manager or executive come and impose a new work process upon a team. Everyone groans about having to learn some new tool, or how teams were broken up and shuffled around. Change aversion needs to be dealt with. Change fatigue is also another problem if people attempt to shuffle things too much. All these factors mean that everyone winds up doing this slow dance between different idealized work structures. The hope is that on average this keeps us moving forward for getting our actual work done.
So anyways, I’m not completely sure where I’m going with this. It’s just that I found it interesting that so much time and energy can be put into this class of work and it doesn’t really get seen as anything besides a nuisance or “manager work”. Because all of us participate, build, and influence processes all the time. We’re inevitably going to be paying the cost of our processes. We’re also the ones who are going to see the explosions firsthand when things go wrong, so it feels important that we pay more attention to this.
Standing offer: If you created something and would like me to review or share it w/ the data community — just email me by replying to the newsletter emails.
Guest posts: If you’re interested in writing something a data-related post to either show off work, share an experience, or need help coming up with a topic, please contact me. You don’t need any special credentials or credibility to do so.
About this newsletter
I’m Randy Au, Quantitative UX researcher, former data analyst, and general-purpose data and tech nerd. Counting Stuff is a weekly newsletter about the less-than-sexy aspects of data science, UX research and tech. With some excursions into other fun topics.
All photos/drawings used are taken/created by Randy unless otherwise credited.
randyau.com — Curated archive of evergreen posts.
Approaching Significance Discord —where data folk hang out and can talk a bit about data, and a bit about everything else. Randy moderates the discord. We keep a chill vibe.
Support the newsletter:
This newsletter is free and will continue to stay that way every Tuesday, share it with your friends without guilt! But if you like the content and want to send some love, here’s some options:
Share posts with other people
Consider a paid Substack subscription or a small one-time Ko-fi donation
Get merch! If shirts and stickers are more your style — There’s a survivorship bias shirt!
Entropy, man. Or the driver of Kuhnian paradigm shifts--the old ways persist until the inherent contradictions can no longer be rationalized and have come into open conflict. Au fond (pun intended) is the curious mental accounting that overweights $1 of revenue against $1 of cost savings (except when time comes for headcount trimming). I saw this in banking. Loan production was a profit center where volume was king. Loan servicing was a cost center where unit cost minimization was exalted. You can anticipate where this goes, so details are omitted.
This reminds me of the nightmare back in the 1980's that was process mapping. Months of my life were frittered away listening to consultants and drawing huge flowcharts, and covering them with sticky notes, so we could redesign our processes, find the gaps where there was no real process, etc. It actually made sense, but when push came to shove, the needed changes rarely got sufficient executive backing to make it happen. One guy did have the cojones to fix the annual budget process, which changed it from a 13 month process (yes, 13 months for an annual process) down to 6 months. He was made CFO. What I saw was that the problem with creating an efficient process is that it usually crossed departmental boundaries, and so any change in the status quo stepped on too many toes. Only a deep commitment from very high up could force a change.