Attention: As of January 2024, We have moved to counting-stuff.com. Subscribe there, not here on Substack, if you want to receive weekly posts.
I’ve worked with various engineering teams for many years now, and there’s a fairly large chunk of teams that have an implicit belief of a form of technological meritocracy — as in, if they build a good product (often from a technical standpoint, but also just in general), then success will come as a given. Users will beat a path to their doorstep with little effort.
Having sat on data teams that have worked with UX, marketing, and sales functions, it is often the case that I have to tell the eng team that this is definitely not how things work. Without some minimum effort spent making sure that people know that the product even exists to begin with, there will be no purchases and thus no money to pay anyone’s salaries. Teams usually struggle with this truth, but when their adoption numbers are just absolutely dismal and the funding runway gets ever shorter, they come to the realization that marketing isn’t just a buzzword.
So imagine my surprised Pikachu face when I’m told that I need to do better at making sure other teams (and more importantly, senior management) what my achievements have been in the past month/quarter/year.
The problem of down-biased self calibration
I've noticed that many researchers and analysts share a common problem when they evaluate their work. Most of us are hyper aware of caveats and flaws and can see all sorts of ways things could be better if only we had more time or resources. This constant awareness of what our research cannot conclude and the drive to do better research is great for producing strong research findings, but it also biases us towards underselling our work and it's associated impact when it comes time to tell people what we've done.
My wife has often said that I only notice the negative sides of everything and ignore the positive, so I likely take this whole notion to extremes. Very often I'm surprised and confused by feedback from people who see my work and start saying very nice things. Sure, I guess that analysis is “cutting edge” because it's such a niche question that few people would be bothering to even spend effort writing or publishing about it?
Yes, I get that the stakeholders are seeing how the analysis I’ve put out will enable all sorts of interesting stuff on their end. They’re usually biased towards seeing the positive (read: useful) parts of the work as applied to their own work. So on average, stakeholders are going to generally see us in a more favorable light than we view ourselves.
But things get messier when we have talk about our work achievements to people that are a couple of steps removed from being a direct stakeholder. For example, how should we talk about our work when showing it to some executive 3 management layers away?
When explaining our work to someone so organizationally distant, we’re required to not just describe what the research is, but also the context that called for the research to begin with. So now, we have the opportunity to amplify our biased self evaluations by underselling the context of the problem instead of just the research itself. I’ve literally caught myself noting down “We did XYZ research because some team needed the information to make a decision.” I’d often forget to even ask what the specific decision to be made was beyond knowing it was for a broadly important project.
Suffice it to say, “I spent two weeks analyzing data to make this research report for a team so they can do something with it” makes for a pretty pathetic account of what I’ve done in the past quarter. Borderline “why are we paying this guy any salary at all?” quality there. But yup, in my younger (and not so younger) days I wrote stuff that essentially was that. I luckily had the help of managers who would slap some sense into me and get me to add important details. Things could have gone significantly worse for me without that help.
These days, I’ve learn to spot the most obvious bits of these and stop them before it gets too bad. I still often forget to note down the broader context that leads to a given piece of work and have to ask people about it after the fact, but I at least know to ask for it now.
Leaning into “objectivity” to help mitigate some bias
While there’s lots of ways to learn how to write about our own work better, one that seems to work well for analyst/researcher types is to just lean hard on concrete facts where available. While I definitely have trouble figuring out where the line between “accurate” and “boasting” is, but my work contributed to an ultimate $5 million increase in revenue, I don’t have to put too much opinion into it.
Sure, this thin veneer of objectivity is also full of subjective decisions. We as data folk know this probably better than most people. From deciding what we’re going to report on and the specific details of how to count and attribute work, we can make these metrics sing however we want. But most people I know are much more comfortable working within those confines than simply freeform writing things from scratch. It just doesn’t feel so weird if you point to a giant quantifiable stack of cash and say “I helped get that”.
Even if you can’t measure things in currency, there’s often some kind of concrete thing to point to. My analysis led to the decision to stop a launch to fix issues. The experiment showed that our prototypes were on track and we could move forward. The dashboard I made was visited by a thousand employees a week. Multiple PMs attended the research presentation I gave and many asked detailed questions. While there’s room to debate to what extent your contributions to each are valuable, it’s indisputable that a contribution was made.
But there’s no substitute for clear wording
Even if you gather all these facts and context and achievements together, there is one piece of art needed to make things “work” that requires practice — the actual writing part. While you’d think that someone who has literally written 1500-word posts every week for three years would have no problem wordsmithing a 100-word description of my quarterly achievements… you’d be very wrong.
I continue, to this very day, to be pretty bad at this. The only comfort I have is that I’ve very slowly upgraded from “utterly terrible” to “pretty bad”. Again, this is because of all the critical bias I have when viewing my own work. I’ve been told by multiple people over the years that I can make a significant achievement sound as exciting as a monotone presentation on bread turning stale.
The only way I know of to improve is how most writers improve their writing — write what you can, share it with someone you trust, get feedback, revise. While in theory you can revise your own work in isolation, it’s extremely difficult and requires taking breaks where you don’t look at the text for a while — not conducive to quick project turnover.
So here I am, over 15 years into working, and still trying to figure out how to describe my work achievements to others. Maybe in another 10 years I’ll finally reach a decent level of comfort with it. Until then, I’ll just rely on getting feedback from my friends and colleagues.
Hopefully, everyone out there who are probably feeling just as uncomfortable about this whole process will feel better than it’s not just them.
Standing offer: If you created something and would like me to review or share it w/ the data community — my mailbox and Twitter DMs are open.
Guest posts: If you’re interested in writing something a data-related post to either show off work, share an experience, or need help coming up with a topic, please contact me. You don’t need any special credentials or credibility to do so.
About this newsletter
I’m Randy Au, Quantitative UX researcher, former data analyst, and general-purpose data and tech nerd. Counting Stuff is a weekly newsletter about the less-than-sexy aspects of data science, UX research and tech. With some excursions into other fun topics.
All photos/drawings used are taken/created by Randy unless otherwise credited.
randyau.com — Curated archive of evergreen posts.
Approaching Significance Discord —where data folk hang out and can talk a bit about data, and a bit about everything else. Randy moderates the discord.
Support the newsletter:
This newsletter is free and will continue to stay that way every Tuesday, share it with your friends without guilt! But if you like the content and want to send some love, here’s some options:
Share posts with other people
Consider a paid Substack subscription or a small one-time Ko-fi donation
Tweet me with comments and questions
Get merch! If shirts and stickers are more your style — There’s a survivorship bias shirt!