This bi-weekly paid subscriber post of thoughts on Randyās mind brought to you by a lot of eye rolling.
So a new startup has been making the rounds on my Mastodon feed amongst my UX friends. Iām not about to give them SEO but if you search for the term synthetic user youāll likely come across itā¦ unless more startups pop up in the next couple of weeks. Considering the climate of AI-hyped startups these days, I would not be surprised if competitors popped up like weeds if it gets any sign of traction at all.
The pitch of said startup isā¦ āUser research, without the usersā. They use generative AI to create fake synthetic personasā¦ and then have you run your product ideas against said personas and they tell you what they like or donāt like about it.
Why go through the trouble and expense of recruiting and interviewing samples of live humans for a product or usability study when you can ask an infinitely cheaper GPU that wonāt tire or need compensation! Generate product ideas endlessly and get feedback. Find product-market-fit in minutes! Get the feedback you need to make product decisions instantly! Do it cheap and fast!
For the hell of it, I asked Googleās Bard what it thought about the idea and it weaseled out with an āIt Depends(tm)ā. But it did mention the risk of the results being not accurate. Much of its other opinions are fairly nonsensical and premised on me developing an LLM.
The reaction of the UX community was about what youād expect ā a lot of eye rolling and skepticism because the concept is missing the point of user testing. When I showed it to my coworkers, I quipped that it was like having a business without the customers. Another researcher said that while they thought itās a terrible idea, they might know a couple of product managers who would probably love the idea.
Even if you give way more benefit of the doubt that is probably warranted, the whole concept loses sight of why we need to validate our ideas against real people in the real marketplace to begin with. No amount of imagination and personas can approximate the lived experiences and views of people. In fact, within the UX community, the use of personas, the summarized fictious āaverage people of a certain typeā continues to be questioned as to whether theyāre an effective tool to use or not. Itās not clear cut and there are ways to use and abuse researcher-generated personas based on real interviews and studies. You donāt even have to inject LLM-powered fictions on top of it all to be on methodologically uncertain ground.
The whole point of user research is that weāre always surprised by the combined cluelessness AND cleverness of users. People will have needs weāve never heard of, have work processes weāve never seen, priorities we canāt imagine, and they will attempt to use our products in ways we never intended. Itās up to us as good product people to take that and decide whether we want to encourage or discourage that behavior in our product.
Hopefully, making the correct choices leads to increasingly happy users willing to pay us money. An LLM thatās effectively the statistical average of all the text on the internet will capture some of this variation and sentiment, but no one knows to what extent and completeness. I have no idea how capable such systems are in expressing unexpected, novel statements.
PMs already try this anyway without LLMsā¦
Even prior to this whole LLM thing, there have always been PMs who wanted to run their own usability studies. Iāve seen qualitative researchers immediately jump up and volunteer to give training to these folk because, without training, the PMs would often ask questions that are biased towards the end result they wanted. The training was meant to help then avoid falling into a giant confirmation bas trap.
For example, a PM might directly ask whether a user would want a specific feature. Most users would say yes since the PM would likely ask about features that had been requested by others before. But just because a user finds a feature desirable verbally doesnāt mean that itās exactly the thing that needs to be built immediately ā there could be more pressing issues that need addressing. Knowing that a user wants a specific feature doesnāt tell you how to design and implement said feature. Understanding users is more about understanding the motivations behind their actions than any specific action. If we understand peopleās motivations, we can much better anticipate what they are likely to do.
And therein lies the danger. When you can summon and tune your virtual users to fit whatever population distribution you want, and theyāre all going to mimic some arbitrary āpersonaā via āthe magic of AIāā¦ well, how are you going to do anything except encounter confirmation bias? How can you surprise yourself when youāre largely presenting to a set of people that youāve specified the fabrication of?
If you ask any of the chat bots whether users would accept some ridiculous product, like āwould users want to use a product that gives them electrical shocksā, theyāll probably give the expected answer of āprobably not except in certain situationsā. But most people typically donāt need an AI to tell you that. Tone things down to ideas like āwould a user pay $5 to speed up a wait time by 5 minutes?ā and the answers effectively read as āit dependsā, which is also the expected answer. If anything, itās hard to thread the needle to ask these things a question where it doesnāt give an obvious answer, let alone a surprising one.
To the extent that you want to pay for a rubber duck to sound ideas off of until they donāt sound ridiculous, I can totally see it working. Itās just not clear exactly what value it brings beyond that.
Machine Translation, but for UX
So, another market that was disrupted by ML tech a decade or so ago was translation. Machine translation had gotten significantly more convincing over the past decade as new techniques were developed. Things improved enough that people who werenāt familiar with both languages themselves found it very believable. Enough so that students started trying to it on their homework and tests (but teachers can still spot those instances most times).
Translation agencies started leveraging that technology to increase their throughput, effectively taking on more work for little increase in staffing. Some of the less scrupulous outfits would offer cut rates because theyād hire significantly less qualified (read, cheaper) editors to āfix upā machine translated text without correcting obvious language issues. Such texts still come out fairly crappy, but they get something out fast enough to take on tons of work. These agencies can often get away with their shoddy work because clients often canāt read the target languages, so they canāt judge the quality of the work.
End of the day, this behavior put a lot of pressure on the lower end of the marketplace. Itās a lot of work explaining the value proposition of paying a relatively expensive translator compared someone who claims to do it for half the price or less, and faster. The higher end of the market, the legal documents and companies that have experience with translation is less affected because they can still see the gap in value.
Itās very easy to see this play out in the near to mid term if such robo-UX takes root. The same people who are going to think itās a great and wonderful idea are also the people who arenāt equipped to know where the pitfalls are.
For simple things that seem obvious to experienced UX folk, like not putting huge friction points between users and their desired activity or saying that āusers like cheap and secureā, the advice will probably be usable. But for deeper, more complex things, human experts are still going to be needed to do the job and capture the nuance that the computers canāt.
But just as the translation industry canāt stop the technology creating a perverse situation for a certain segment of the market (that arguably might not have been part of the original market for translation services), thereās no force on Earth that will stop people from trying to use what appears to be a cheap and easy solution to their problems. It remains to be seen whether people will find it useful enough to adopt, but these things can go pretty far just on promises and illusion alone.
Itāll be up to us to keep emphasizing the value that we bring to the table as researchers.
No, Iām not looking forward having to that either.