Announcement: I’ve officially lost my marbles and have decided to ramp up my writing output to create a second paid subscription tier! To encourage more of y’all to support me, I’ve set up a 60-day free trial offer for those interested but not too sure what things will be like. But fear not! The free weekly Tuesday posts will continue. I’ll be creating more content on top of the usual. To jumpstart things the first subscriber-only post was also published this morning poking at Substack’s payment flow. Going on, subscriber-only posts will be posted around Thursday/Friday every 1-3 weeks.
Imagine that you have some kind of very simple webpage, like a “subscribe to my newsletter” page. It operates very simply — load the page, enter your email, optionally enter some payment info to pay for the subscription, then press a button to submit the form.
Imagine if you are the data person and have been tasked with understanding what users are doing on the page, with an eye towards potentially making an improvements later.
First you calculate most high level conversion rate (after filtering out users who already signed up) and find out that 1% of all the unique browser page views actually sign up.
Management freaks out because 1% sounds really low. They spin up a ton of hypotheses for what could be wrong. Is the sign up page too complex? Did the payment section scare people despite there being a clear FREE option? Or maybe people visiting just aren’t interested in signing up? How do we go about eliminating some hypotheses, calming management down, and help figure out a plan?
Bring on the funnel analysis
What I've described above is the textbook use case for funnel analysis. You pour users into a page at the top, and see how many get to the end. The general strategy is that you try to identify every possible point at which users can give up and stop progressing through your linear(ish) page flow and hope to identify the places where the most users drop off.
For example, in the example above, we have 100% of users visiting the page, then only 10% of the original group even start typing their email address. Next, 9% actually finish typing their email and select a payment method (which includes a free option). By the time everyone finishes the payment related stuff, you're left with that dismal 1% that everyone is concerned bout.
Given this picture of the funnel dropoffs, an analyst can come to a number of conclusions.
One analyst cries out — that first dropoff is huge! We lost 90% of our potential customers there!!! Imagine how much more we can make of we found a way to convince an extra few percent through the funnel.
Another stands up and asks — why did we lose 1% between typing an email and going to the next step?! It's one text box and a “Next” button — surely we must be doing something ridiculously silly to lose people that way.
Finally someone else hops up to point out that we lost 8% of our potential customers when they're supposed to either pay or pick the free option. Why doesn't everyone just pick the free one if they don't want to spend money and they’ve already went through the trouble of all the typing? Who doesn't like free?! Either way this group is closest to the actual money and we can most directly affect their decisions.
All of these are plausible interpretations of the data, and interseting enough to investigate. Which one to pick to investigate first largely depends, as it always does, on specific details. How confident are we at being able to make improvements in a given section. How much effort is it to make changes? What factors are beyond our control?
Where’s the Intent?
Perhaps the most central question to every step within a funnel-like process is whether the user actually had any intention of proceeding to the end, or even just the next step.
If that particular user had the full intention to get to the end and give us money, but then ultimately failed to so so, its a very strong criticism about how we designed things. It’s a “this is OUR fault” situation. There must have been enough friction in the process to have overcome their desire to finish, and it was our job to minimize that barrier and we failed at it.
If, on the other hand, the user had no intention of finishing, then it changes the whole problem and solution space. Maybe we have to market things better, provide a better sales pitch, or maybe that person just wasn't our target customer to begin with and there was nothing to be done and we should exclude them from analysis consideration. No amount of flashing banners and fancy design can (ethically) convince someone who doesn’t want to buy our product to give us money.
Intent becomes this sword with which we can divide the world in two. On one side, all the high-intention people that failed to complete are likely to have issues we directly caused such as bad usability or broken infrastructure. Product teams love to analyze the high-intent people because they tend to respond well to product fixes.
All the low-intention people have a myriad of potential issues, but the biggest one being their lack of interest. “Fixing” that (to the extent such issues are even “broken” or “fixable”) lies more in the realm of marketing and sales. Maybe it’s poor sales copy on the site. Maybe there’s just no awareness of the benefits of signing up. Maybe we bought a ton of ads that went to the wrong target audience and now we paid for a lot of visitors who don’t care about us. All this stuff is often handled by completely separate teams than the typical product/engineering team.
The problem, obviously, is that intent is impossible to directly observe.
If we have telemetry about user actions (clicks and page views and such), we can only make guesses based on what they’re doing. In the example, 90% of users (really, unique IPs/cookied browsers, but close enough) who saw the sign-up page didn’t even bother to type their email address in. There can be all sorts of reason why that is. Maybe there’s a bug that made the box to input your email not even appear. Maybe the page is too busy and users couldn’t see it. Or, maybe the people really just were honestly not interested and wound up on the page.
An analyst can probably come up with all sorts of heuristics to create an artificial gauge of intent. Anyone who views the page for less than 2 seconds and leaves was “obviously” browsing, so we can exclude those people from our analysis because today we care about getting interested users to the finish line. Users from Reddit came in due to a viral post and they’re clearly less likely to stick around. There’s endless weeks and clever little analyses you could run to slice and dice the data in an effort to understand just how many visitors actually had intent to sign up.
Despite the many plausible-sounding ideas, at the end of the day, they’re all guesses and correlations. You can attempt to run with the idea and build out something that might “fix” the problem for these people, but you might be completely wrong and waste everyone’s time.
OK, maybe we can ASK some of these people — you know, do some qualitative research? That sadly doesn’t usually go very well because how do you find these anonymous people who refused to even give you their email to talk to? If you try to do something cute like get a widget that pops up a survey on the page when people visit, it wouldn’t be surprising that the self-selection bias on the results is astronomical since the effort to respond to the survey is likely higher than the initial sign-up step. Recruiting random people off the street to look at your site and comment on the funnel is also fraught with tons of sampling bias issues.
I honestly can’t think of a fast solution to the task of scooping out intent in such situations. It’s a lot of painstakingly scraping information from where you can get it, testing out hypotheses, and generally having lots of experiments fail and learning from them.
If you keep working in product, you’ll be fussing about intent forever
Just earlier this year, I was working on a product that had a relatively simple purchase flow page where only about 15% of people seeing the flow even completed it. There were all sorts of concerns about that low conversion number, and we had been debating whether a specific section of the flow that had been recently added was making things worse. But once it became clear that the team was going to keep working on this, and the feature we were launching didn’t appear to be enough of a problem to cause such a large drop, we took a step back.
We looked carefully at the funnel drop-offs and learned out that if people started interacting with the purchase process (typing into the form, clicking on a dropdown, etc), then 85% of those people would wind up buying something. The main explanation for the original 15% rate was because the entire product site itself had very few things to do, so users most likely had just explored and stumbled upon the purchase flow to see what how it worked. We couldn’t explicitly prove that most visitors weren’t interested, but enough circumstantial evidence existed that led us to that belief.
Once the team found out that 85% of users who seemed to express the barest minimum of interest, they quickly decided to go work on other things. If we hadn’t first tested out the hunch that team members had that “most of the users visiting the page probably never intended to buy”, we might still be doing A/B tests or ad-hoc analyses to try and figure out why people weren’t buying. Now, the team can focus more on adding features and getting some marketing budget to make sure people are aware of the product and want to buy it.
In other teams, other companies, the search for intent merely takes on different identities. Sometimes it’s straight up called some variation of “user intent”, “purchase intent”. Other times it shows up in methodology frameworks like “Jobs to be Done”, or “Critical User Journeys”. Whatever it is, we’re asked to somehow measure and then use this fuzzy construct to make sense of why people do things.
In academia, there would be full blown theories built around the concept of intent, complete with well-tested ways to measure intent with respect to certain contexts and fields. There’s plenty of papers that mention such things on google scholar (the phrase appears quite popular in information retrieval/search engine contexts in the 2010s). Sadly we don’t have that luxury in industry where the problem domains are too niche. All we get are niche little rules of thumb that sorta work for our individual domains. The intent signals that “work” (as in, correlate with our desired outcome and pass the sniff test”) for e-commerce likely doesn’t work for purchasing a home and are again different for customer complaints.
But hey, at least when you come across situations where people start asking why users aren’t flocking to buy/use/adopt the new thing that was launched you’ll have a head start on the answer — they probably don’t care. Yet.
About this newsletter
I’m Randy Au, Quantitative UX researcher, former data analyst, and general-purpose data and tech nerd. Counting Stuff is a weekly newsletter about the less-than-sexy aspects of data science, UX research and tech. With some excursions into other fun topics.
All photos/drawings used are taken/created by Randy unless otherwise credited.
randyau.com — Curated archive of evergreen posts.
Approaching Significance Discord —where data folk hang out and can talk a bit about data, and a bit about everything else. Randy moderates the discord.
Support the newsletter:
This newsletter’s Tuesday posts are free and will continue to stay that way, share it with your friends without guilt! But if you like the content and want to send some love, here’s some options:
Share posts with other people
Consider a paid Substack subscription or a one-time Ko-fi donation
Tweet me with comments and questions
Get merch! If shirts and stickers are more your style — There’s a survivorship bias shirt!