Today’s post stems from conversations and questions from people over time. Just here and there, enough to stick to my brain.
In what way does my work differ from any other data scientist when the skills and requirements look largely the same?
As I’ve noted previously, the job title that I have, “Quantitative UX Researcher”, is still a very rare in industry, with only a handful of very large tech companies really incorporating it into their official job titles for hiring purposes. As is expected from a title that isn’t used in many places, the definition of what exactly constitutes a qUXR is a bit hard to pin down. From some angles we sorta look like data scientists, from other angles we look like engineers, analysts, and sometimes even academic or industrial researchers. I have colleagues that hail from all those fields, they are all awesome. It’s about as equally a confusing mess as it is for data science proper — lots of people have the skill set and rightfully qualify, with little rhyme or reason beyond that shared skill set.
Zoomed out a bit to average over the various opinions people have about what basic skills qUXRs need to have to be effective at their jobs, and it usually falls into five major buckets (that are still pretty vague and cover a lot)
Statistical things — experiment design, hypothesis testing, regression, etc.
Programming things — analyzing data at scale takes code, usually at scale
Data analysis skills — data handling and doing good, careful, analysis
Researcher skills — asking good questions, balancing competing concerns, rigor, communication, etc.
“User focused” stuff — an interest in understanding users and how impacting users, approaching problems from perspective of users
At a very high level, with an exception for the last, they look similar enough to what you’d expect for a general purpose data scientist. Except perhaps there’s clearly no mention of AI/ML in there (not that I believe all DS need those skills either).
What’s with that “User focused stuff” bullet?
As I’ve previously argued, many companies likely have an equivalent of a quant UXR role, they just have some other title like data scientist, data analyst, business analyst, product analyst, etc.. The job of “person who helps us understand our customers so we can build better stuff for them”, is not a new one.
It seems obvious that a “user experience” researcher, even a quantitative one, would need to have a focus on users. But when we refocus things on a quantitative/data science context instead of the more traditional qualitative UX researcher role, things get a trickier.
Data science often has the luxury of hiding behind a thin veneer of neutral objectivity — “my A/B test is merely showing what’s more effective vs my objective function”. While I don’t particularly believe that veneer is true and DS injects tons of bias and viewpoints into their work, some try to take comfort that they just do math-y/computer-y things and what the machines do with it is somehow a separate debate.
But in UX research, I find that attempting to hide behind “neutral” algorithms (I’m not getting into algorithmic bias this week so we’ll just leave it in quotes) tends to make for poor UX research. The ultimate job of a UX researcher is to discover results that highlight places where user and business interests align, and advocate for changes to be made there. The hard work is guiding the research work to find such insights with finite resources. It’s not a game of just shotgunning random neutral tests until we get lucky. There’s a lot of creativity, exploration, and iteration involved.
So the user focused part means having an opinion where appropriate. It’s the difference between knowing that getting a 0.5% increase in checkout success with a copy change is good for the business, and realizing that a fixing the 10-step checkout process might be causing more user pain and is worth taking a look at.
It’s knowing when there’s not enough information to have an opinion on something and doing research to figure it out. For example, realizing we’re not sure how long users are willing to wait in queue for customer support and developing a method for measuring whether there’s an effect.
These questions and responses come out thinking about users in a way that a straight data analysis wouldn’t require someone to. The data won’t say many things to an analyst because the data flat out isn’t being collected to begin with.
It’s hard to exactly characterize all this because it’s not a specific tool or methodology. In my mind, it is a way of thinking that’s developed by direct experience and watching other people do it. It’s a mix of understanding what users are likely to want tempered by domain knowledge, user empathy, as well as wielding the methodological tools needed to get at the answers.
What I find fascinating about all this is that every one of us has to live through life as a user of something. We’ve all experienced frustrations at bureaucracy, crappy interfaces, long wait times, etc.. While most of us aren’t going to be qualified to have opinions about what being a user experience of a 787 pilot is, with some imagination, we can still make some educated guesses about some basic things need to be (e.g. controls should be easy to reach and not strewn about in random order).
— Before I get murdered by angry UXers for oversimplifying, let’s be clear that a basic naïve sense of user empathy isn’t nearly enough for UX work, but it’s an important shared starting point we can build on.
Despite having this shared baseline, there are plenty of candidates who haven’t don’t seem to exercise these skills — often because they haven’t been asked to before. And since it’s a way of looking and thinking about the world, it feels really uncomfortable early on. It requires stepping out of the comforting “neutrality” of analysis work into having opinions and hunches about how things should be. Then we take those hypotheses and validate them, while also gathering more hypotheses from others.
Maybe we aren’t sure whether people in the office are happy with thermostat setting, but from the number of people wearing jackets indoors in July (which we have an opinion as being abnormal), it might be worth investigating. The question sounds pretty obvious when stated out loud, but it takes practice to take that perspective and notice there could be an issue in the first place. It’s just like how it takes a lot of knowledge, practice, and training to “get” how to write good code that takes a CSV from one end and spits out a chart the other. I have no good idea on how to teach that either besides hands-on practice with some pointers.
What always has me puzzled is that there’s also plenty of data scientists that “get it” when it comes to users. Sometimes these skills show up on a resume because that person has done work in product development and their user-centered thinking skills have a chance to show through, but I’ve seen people where you’d never know that they’ve ever worked with user data before from their resume but just having a conversation with them it’s obvious they know it to their bones. Meanwhile I’ve seen people who seemed to have worked on very use-focused projects on paper, but will utterly fail at even considering the user despite clearly working on the code and algorithms of the project.
Finding out all of this has to wait until the live interview portion to come out because it comes up in countless small ways throughout the work. And by then, a lot of people’s time has been needlessly wasted.
Sounds fun, so how do I get in on this UX stuff?
While processes vary from organization to organization, the most general framework for how qUXR works falls under the broad framework of “User Centered Design”. Early in the process there’s a research phase, and lots of testing/validation steps throughout, that is where we most naturally plug our work into the work of others.
There’s endless amounts of articles written about “design thinking” that’s worth perusing a bit. (Do note that some try to make it sound like magic pixie dust executives can sprinkle on product teams to make good product — sorta like how AI is treated).
In more academic fields, there’s a lot of overlap between fields like industrial design, human-computer interaction, cognitive science, theories of learning, and others. At this point I’m pretty positive that you’ve brushed across some bits of this vast web of knowledge before without realizing it. It borrows concepts from all over.
And while I definitely don’t advocate for everyone to learn design, to make things even more complicated, some people in the field of UX go so far as to say that everyone’s a UX designer. They do caveat it by noting that we might not be very good at it, but things we make and do will influence a user’s experience and is thus a design decision.
As researchers, the goal is to make sure that everyone making design decisions has good information to work from. It’s our job to realize that “hey, maybe some users have too many notifications”, do the work and find out “if we ping a user over 30 times in an hour they quit FOREVERRRRR”, and then make sure everyone who has the power to send notifications know they have the power to greatly annoy our customers.
Playing this complicated game of figuring out if problems exist, how to measure them, and then making sure problems are avoided before they’re even problems is why I enjoy doing the work I do.
And I’d like to have more friends doing it with me.
No one sent in anything for me to share/review this week. So if you have anything for next week, send them on over~
About this newsletter
I’m Randy Au, currently a Quantitative UX researcher, former data analyst, and general-purpose data and tech nerd. Counting Stuff is a weekly data/tech newsletter about the less-than-sexy aspects about data science, UX research and tech. With excursions into other fun topics.
All photos/drawings used are taken/created by Randy unless otherwise noted.
Curated archive of evergreen posts can be found at randyau.com
Standing offer: If you created something and would like me to review or share it w/ the data community — my mailbox and Twitter DMs are open.
Supporting this newsletter:
This newsletter is free, share it with your friends without guilt! But if you like the content and want to send some love, here’s some options:
Tweet me - Comments and questions are always welcome, they often inspire new posts
A small one-time donation at Ko-fi - Thanks to everyone who’s supported me!!! <3
If shirts and swag are more your style there’s some here - There’s a plane w/ dots shirt available!