We should watch people dissect and build products

Because we learn a lot that way

I went outside over the weekend (!!) and found myself at the Orchid show at the botanical gardens. An excuse to use the camera!

Last week, a reader linked me to an article they wrote, A Eulogy for Dark Sky, a Data Visualization Masterpiece by Srini Kadamati. That was a very good in-depth analysis of the very careful design that went into how the Dark Sky weather app would show you weather information. So good that it sorta reminded me that I should occasionally write about my UX researcher job too.

For context, Dark Sky had been a popular weather app, especially in the hyper-local forecast space where weather reports and predictions are tuned to your specific location instead of the general city/area that most forecasts are given in. That means the app will tell you if/when a specific thunderstorm cloud on radar is likely to fly over your head. They developed a bunch of tech to power those predictions and were acquired by Apple in 2020. The app was finally disabled in favor of Apple’s own Weather app this year in 2023. But as an Android user, we had lost our version of the app ages ago.

The post is worth a read by itself just from a “marvel at a bunch of fairly subtle design decisions” point of view. The app does a LOT with showing relevant weather information contextually. As Srini points out, the app design is constantly thinking what bits of weather information are most likely relevant to a user at the exact moment they’re using the app, and it will display those elements most prominently. When viewing the current weather, if there is some kind of inclement weather warning, or just generally bad weather in the near future, it’ll show those more prominently along with storm maps and such. But if you switch to viewing weather a few days in the future, the view it presents is much more tuned towards helping you understand what things will be like in the future because you’re likely planning something.

As product researchers/analysts, we need to appreciate the design process too

The best reason to read that post I linked isn’t just to see someone list out a bunch of subtle design features that most users honestly would not even realize was happening. Instead, it’s a good way to get familiar with some of the fundamentals of working with products in general — the fact that people use products for various reasons, and understanding those reasons is a path towards improving your product.

There are many different names and frameworks that have been proposed to help think about the whole concept, “Critical User Journeys” and “Jobs To Be Done” are common examples. The common gist that connects these frameworks together is that people pick a problem to fulfil one or more needs — for example, a need for information or a need for eating food in the morning. Having a good understanding of those needs and all that context surrounding people using your product will highlight places where you can improve or expand your product.

So the Dark Sky analysis choses to go about this by breaking things down into user “Goals”, such as making social plans at a location and time in the future. That goal is then matched with a set of (hypothesized, unvalidated) potential information needs. For example, the user with the goal might be asking “Can I go skiing this Sunday?”. Srini then lists out a couple of other possible user goals, as well as their corresponding informational needs.

Since this is a retrospective of the design decisions that the Dark Sky team eventually made and launched in their product, everything seems kind of obvious. We work backwards from how they decided to do some things, and guess at what goals and needs they were trying to solve for. It’s all post-hoc explanation, and we can theorize anything that we want. We obviously can’t see all the test designs, the discarded concepts, the experiments that confused users.

But imagine if you were on a product team that was asked to design a weather app from scratch and you don’t have the benefit of hindsight (or a competitor to imitate). The designers and product owners are going to want to use this sort of goal breakdown to come up with designs that try to address as many of these needs as elegantly as possible. The big challenge for them is that many of these goals and needs might overlap or conflict and they have to juggle the needs in the design. They also have to balance the overall navigational complexity of the product. For example, they could theoretically make a separate page for every single goal and have dozens of pages, but would that be the best way to do things?

Meanwhile, the researchers likely have already done a lot of work building the table of goals and information needs to begin with. We’ve been working since the original project idea was being considered. Qualitative researchers would have been interviewing users, proposing goals and needs, and then trying to validate those ideas with further research. Quantitative researchers like myself might be helping run surveys, or analyzing any existing data to try to put a relative size on various behaviors and user segments.

The reason why us researchers need to understand design is because our work is a critical input into the design process. It’s critical that the designers are presented with a reasonably well-rounded list of the most important goals of users for a potential product. No one person, or even a group, can hope to anticipate what a world of needs that millions of users will have. A fully study won’t either but it gets us closer. It’s up to the research team to figure out how to best sample and collect that data and then use our abilities to synthesize things into cohesive insights that the design team can use as their guides.

But the work doesn’t stop at the pre-design phase

Pre-design research work is a LOT of work up front, and it also needs to be done on fairly tight deadlines because the design teams might be sitting practically idle (on this specific project, they can keep busy in other ways) until enough research comes in. But just because that phase of work is done doesn’t mean there’s no more work to do. Since design is an iterative process, there are going to be multiple cycles of doing research and hypothesis validation along the way.

A lot of that early stage validation work is going to fall on more qualitative research folk who will be talking to people in small numbers because an unreleased product by definition won’t have users at scale. But eventually, as alpha, beta, and final versions of the product are released and data gets collected, quantitative methods will become more increasingly more interesting.

When the time comes, one of the bigger challenges for us quants is going to be finding a way to measure if the goals of users are being fulfilled — are we successful at building the thing we promised to build?

Obviously this requires knowing what user goals exist, but we luckily have that from the pre-design work. It also requires figuring out ways to measure how users are progressing along their journeys. That means coming up with measurements that give a decent approximation for what is often only going on inside a user’s head as they’re doing various things. This is where we can spend a couple of days trying define what “satisfaction” or “spent meaningful time” even means from a measurement standpoint.

Then on top of the epistemological concerns, there are technical ones too. Measuring things used to be relatively simple when everything was just links, page loads, and POST requests from forms. Nice clean discrete events that happen largely in sequence.

Nowadays, with modern web designs and telemetry frameworks, it’s a giant complicated mess of spaghetti. Users can do multiple things (or everything) on a single page and you’ll have to find ways to pull things apart. Pages might infinitely scroll and page elements might look almost identical coming out of a factory pattern. The state of pages can constantly change with context and it’s a challenge to make sure all the details are tracked. Navigation paths swim between multiple places. Users do things in multiple tabs.

Given how complex modern page designs can become, and the ridiculously long tail of action sequences users can take, it becomes increasingly important to understand what users are trying to do. You have to approach things with a base level viewpoint of what certain tasks will likely look like. It’s extremely difficult to approach the analysis from a pure data-only perspective because so many things are intertwined.

For example, take any basic weather app. It’s going to display a default first page, often of current weather data. Think about what you’d do if you were trying to measure how many people are successfully completing the “I need to know the weather now to make a decision” job.

You’d probably start with just counting people who view the first page w/ current weather info. Most of the information a user would want is likely going to show up by default so their task might be completed right then. But since it’s the default page, you might want to exclude anyone who goes to some other page that is dedicated to another task.

Then you realize that people might be accidentally opening the app and so you have to come up with a rule to exclude those people. Maybe later you year from other researchers that there’s other important pages for this task that you need to somehow include. By the end of a year, the definition of “people who are looking at the weather” has become a 50-line CASE statement with a bunch of edge cases tuned to finally captures what you think you want.

It’s highly unlikely you can come close to replicating that simply by “looking at the data”. An agnostic approach will often only be able to catch high level trends, but you can’t separate out the small patterns from the noise.

So please, get familiar with products and user work flows. Not everything has to be shown from first principles and raw data patterns. We’re in industry and can often cut some epistemological corners on what we “know” to be true based off hunches. We can always come back and validate our assumptions as we go along. Everything we measure is a vague proxy of what we really want to know, so a handful of extra simplifying assumptions can’t hurt if you’re careful about it.

---

If you’re looking to (re)connect with Data Twitter

Please reference these crowdsourced spreadsheets and feel free to contribute to them.

• A list of data hangouts - Mostly Slack and Discord servers where data folk hang out

• A crowdsourced list of Mastodon accounts of Data Twitter folk - it’s a big list of accounts that people have contributed to of data folk who are now on Mastodon that you can import and auto-follow to reboot your timeline

---

Standing offer: If you created something and would like me to review or share it w/ the data community — my mailbox and Twitter DMs are open.

Guest posts: If you’re interested in writing something a data-related post to either show off work, share an experience, or need help coming up with a topic, please contact me. You don’t need any special credentials or credibility to do so.

---

About this newsletter

I’m Randy Au, Quantitative UX researcher, former data analyst, and general-purpose data and tech nerd. Counting Stuff is a weekly newsletter about the less-than-sexy aspects of data science, UX research and tech. With some excursions into other fun topics.

All photos/drawings used are taken/created by Randy unless otherwise credited.

randyau.com — Curated archive of evergreen posts.
Approaching Significance Discord —where data folk hang out and can talk a bit about data, and a bit about everything else. Randy moderates the discord.

Support the newsletter:

This newsletter is free and will continue to stay that way every Tuesday, share it with your friends without guilt! But if you like the content and want to send some love, here’s some options:

• Share posts with other people

• Consider a paid Substack subscription or a small one-time Ko-fi donation

• Tweet me with comments and questions

• Get merch! If shirts and stickers are more your style — There’s a survivorship bias shirt!