I really like this "surprise us with something new" vs "surprise us by telling us what's old is wrong" framing. Before last week, I honestly would've thought that LLMs would've been pretty good at the former thing - and even more so after reading this tweet from an OpenAI researcher who talked about how LLMs "dream": https://twitter.com/karpathy/status/1733299213503787018
But I read a book last week by a guy who did a bunch of (quantitative) analysis with OpenAI, and he said that was exactly what is was bad at, which was surprising to me. But it does make me wonder - maybe the dream analogy is actually closer to the truth than I realized, because dreams aren't really new experiences; they're repackaging of our experiences in a new configuration. It's old material, jumbled up. Even if LLMs have other material to draw from (they are trained on everything anyone knows, after all), the way they work basically keeps those dreams localized around the ideas they're being asked about. So it's not quite like they're limited by their experience, but they're limited by a kind of distance. And though they can bridge really big distances easier than we can - eg, write a Shakespearean sonnet about McDonalds' french fries, in the style of a pirate - they have to be told to do it. There aren't I don't think the spontaneous connections that happen so easily for people.
(And trying to provide that stuff is very hard. On a few occasions, I've tried to use ChatGPT for help on something, and realized that the most useful part of the exercise is typing out all the context to stuff into the prompt. It's a form of rubber ducking, where going through the process of explaining was much more useful than what it gave back.)
Great article!
I'll have to read again tomorrow, to digest everything. There is valuable insight here.
I really like this "surprise us with something new" vs "surprise us by telling us what's old is wrong" framing. Before last week, I honestly would've thought that LLMs would've been pretty good at the former thing - and even more so after reading this tweet from an OpenAI researcher who talked about how LLMs "dream": https://twitter.com/karpathy/status/1733299213503787018
But I read a book last week by a guy who did a bunch of (quantitative) analysis with OpenAI, and he said that was exactly what is was bad at, which was surprising to me. But it does make me wonder - maybe the dream analogy is actually closer to the truth than I realized, because dreams aren't really new experiences; they're repackaging of our experiences in a new configuration. It's old material, jumbled up. Even if LLMs have other material to draw from (they are trained on everything anyone knows, after all), the way they work basically keeps those dreams localized around the ideas they're being asked about. So it's not quite like they're limited by their experience, but they're limited by a kind of distance. And though they can bridge really big distances easier than we can - eg, write a Shakespearean sonnet about McDonalds' french fries, in the style of a pirate - they have to be told to do it. There aren't I don't think the spontaneous connections that happen so easily for people.
(And trying to provide that stuff is very hard. On a few occasions, I've tried to use ChatGPT for help on something, and realized that the most useful part of the exercise is typing out all the context to stuff into the prompt. It's a form of rubber ducking, where going through the process of explaining was much more useful than what it gave back.)