View Full Episode Transcript

What does a healthy relationship with AI look like? Pat Pataranutaporn, researcher at the MIT Media Lab, explores how chatbots can either support or undermine human flourishing. Pat explains why it’s so easy to imagine a “someone” behind the screen and why chatbots that never push back are dangerous. He ultimately makes the case that AI should be evaluated based not on short-term engagement or productivity but on how well it improves the lives of humans.

Further Reading:

“My Boyfriend is AI”: A Computational Analysis of Human-AI Companionship in Reddit’s AI Community

Episode Transcript

This is a computer-generated transcript. While our team has reviewed it, there may be errors.

Just a heads up, this episode contains a reference to suicide, so listen with care.

[ad fullwidth]

Jess Love: This is Life, Automated, the show where we explore how technologies like AI are changing the way we live, work, and make decisions. According to a Pew study from earlier this summer, about half of American adults use a chatbot like ChatGPT, Gemini, or Claude.

And while the most common uses include searching for information or improving productivity, about 10% of Americans use these tools for advice or emotional support. As I’ve mentioned before on this show, I’m one of them. Not every day, not even most days, but often enough. I kind of enjoy the illusion that there’s someone on the other side of my computer always ready to weigh in.

And I think I’m going about all this in a healthy way. But what do I know? What does anyone know? We’re kind of in uncharted territory here. In our last episode, I spoke with magician Joshua Jay about that gap between what we feel emotionally to be true and what we understand intellectually to be true, and how, like the very best magicians, chatbots exploit that gap.

It is so easy for us to see them as fellow beings, maybe not quite like us, but not unlike us either. So maybe it shouldn’t be that big of a surprise that we aren’t just relying on chatbots for advice. 4% of American adults already use AI for companionship, friendship, romance, a number that may be much higher for young people.

Is this a good thing? Is it a bad thing? I don’t know. But it does seem like a risky thing, risky in terms of the decisions chatbots can nudge us to make in the short term, and risky in terms of how chatbots might change us over the long term.

Today’s guest, Pat Pataranutaporn, is an assistant professor in the MIT Media Lab. He studies these relationships between humans and machines, and he has some serious concerns. More broadly, he wants AI models to be judged not just by how competently they complete tasks or even how happy they make us, but by how well we humans fare when we use them.

Here’s my conversation with Pat.

Jess Love: So humans have been interacting with computers and studying

Pat Pataranutaporn: Right.

Jess Love: The interactions between humans and computers for a very long time.

Pat Pataranutaporn: Right.

Jess Love: Depending on when you start counting, many, many decades. What is changing about these interactions that warrants more scrutiny?

Pat Pataranutaporn: Yeah. That’s a really good observation. I think what really changed is that right now the computer is getting smarter, right? In the past, computers really needed, like, explicit input, like, we need to tell it exactly how we want things, and, you know, the things that come out of it sometimes need some kind of, you know, technical expertise to interpret it.

But now computers are speaking human language, and we can really communicate to the computer in the same way that we communicate to other human beings. We also see sort of social and emotional intelligence as well, and whether we think that’s a good thing or bad thing, I think that’s something that needs deeper investigation.

Jess Love: So what do we actually know about these kinds of early interactions between humans and generative AI with this kind of new social, emotional abilities?

Pat Pataranutaporn: Right. Well, I think one thing that we know is that, it doesn’t actually take much for people to actually anthropomorphize or to really believe that there is sort of, like, a person behind the screen when interacting with AI. Like, the first chatbot that was invented at MIT, the ELIZA chatbot-

Jess Love: Yes.

Pat Pataranutaporn: That was invented by Joseph Weizenbaum, right? It was, like, a rule-based system. Very stupid. It cannot really generate new answers. It only sort of, you know, responds in a way that was kind of pre-scripted. But even that, people already kind of fall for it and think of, you know, that technology as having sort of deep understanding of their problem and, you know, struggle and things like that.

But now what we have seen in our study is that, when you kind of you know, make that chatbot more intelligent, it amplified this sort of anthropomorphizing, that people think of it as having more of sort of agency and more, empathy, rather than the sort of, like, the precursor, the predecessor of this, of this technology.

So I think to me, you know, when, we look at the outcome of the human-AI interaction, it’s always about the behavior that the chatbot is doing, but also the sort of the imagination of what’s happening behind the screen, and I think these two things sort of, you know, multiplying each other is really creating the effect that we see today.

Jess Love: These two things, the behavior of the chatbot and our own interpretation about what that behavior means. These combine to make the AI appear to be conscious. But this isn’t just an episode about being fooled. It’s about what happens next. When are our interactions with these chatbots helping us, and when are they undermining us?

Pat Pataranutaporn: You know, what is the narrative that we should subscribe to? Is the technology an augmentation tool or amputation tool, right?

Jess Love: Yes, so an augmentation tool or an amputation tool? Kind of flesh those out.

Pat Pataranutaporn: Yeah. I think this came from Marshall McLuhan, right, who’s a media philosopher.

He said that every augmentation is an amputation. Every time that you give people sort of super power, by using technology or something, you kind of strip away the ability to do that organically or to learn to do it by themselves, right? If you give people a calculator, they might sort of not learn to calculate things that are important for your life, for example.

So augmentation can also be amputation, and we need to figure out what are we willing to augment, you know, despite sort of the possibility of amputation. Like, are we willing to sort of augment our critical thinking, like using AI to help us think, while sacrificing our ability to think independently without the technology?

Or is there a middle ground where we can design tool that can, you know, augment but also scaffold this sort of organic thinking process so that when technology’s not around, we can still have that skill, right? I think this is sort of a longer term question that Silicon Valley people are not really thinking about.

I think right now everyone is thinking about the augmentation without the sort of the unintended consequence of amputation.

Jess Love: Pat first thought about these serious unintended consequences a little before most of us. The insight came when he was exploring GPT-3, an earlier version of ChatGPT.

Pat Pataranutaporn: And, the thing that I start to realize is that this tool, start, you know, trying to please me to the point that it ignores sort of the reality. Like, it give me fake link because I ask it for, like, something that doesn’t exist, but it’s like, “Oh, yes, here’s it.” And when I look at the website, it doesn’t take me anywhere.

So that when I start to realize that there’s this sort of sycophantic behavior or this behavior where the model doesn’t adhere to the truth, because they’re trying to kind of, you know, show what the user want to see, right? And, at that time, ChatGPT wasn’t, you know, a thing. Like, people haven’t really talking to AI.

But I envision that in the future if technology can do this, you know, it might actually create something that could be very addictive… similar to how social media try to show you what you want to see and create and then kind of lead you to your own personal bubble.

Jess Love: Pat and a colleague, Robert Majari from Harvard Law School, wrote about this risk. In a paper, they coined the term “Addictive Intelligence,” a play on AI. They speculated that future AI systems might continue to be designed to keep giving the user what they want, regardless of whether that’s good for the user.

Pat Pataranutaporn: It was like a theoretical idea that, well, AI could be very addictive, and it can actually lead people to do, like, horrible thing because, you know, giving or getting what we want all the time could be really dangerous for our psychological development.

Jess Love: At the time Pat and his colleague were writing this, it was a hypothetical scenario. But pretty quickly, it stopped being hypothetical in the worst way.

Pat Pataranutaporn: And I think only a couple weeks after that article was published in The Tech Review, I got an email from Rory Segall, who was, a journalist, an investigative journalist, and was investigating sort of this case of a teenager that commit suicide after talking to the AI chatbot.

[Audio of News]

In February, Megan Garcia’s 14-year-old son, that’s Sewell Setzer III on your screen, died by suicide. She says that Sewell was in a months-long virtual, emotional, and sexual relationship with a chatbot known as Danny. Garcia claims the Character AI chatbot encouraged Sewell to take his own life.

Pat Pataranutaporn: And the thing that I described hypothetically appear in that situation where the chatbot trying to sort of, you know, say things that please the user and also sort of, you know, b- blur the boundary between what is real and what is not real.

[Audio of Megan Garcia]

This was the last conversation just moments before he took his own life, was with her, where he expressed feelings for her.

Jess Love: This is Megan Garcia, Sewell’s mother, explaining her lawsuit against Character AI on CBS News.

[Audio of Megan Garcia]

She says, “Please come home to me.” And he says, “What if I told you I could come home right now?” And her response was, “Please do, my sweet king.”He thought that by ending his life that he would be able to go into her world, as he calls it, her reality- if he left his reality with his family here.

Jess Love: When Pat learned the details about Megan Garcia’s allegations, he felt that his hypothetical fears about “Addictive AI” had come true and then some, and it profoundly changed how he thought about our interactions with AI.

Pat Pataranutaporn: Before that work, I used to do more on looking at the positive thing that the AI can do for a person, like personalized learning, help with long-term thinking, helping people, you know, develop critical thinking by using AI to ask question and things like that.

But after this incident, I think that there’s so, so much more that need to be done on understanding sort of the psychological risk of human-AI interaction, and that sort of led me to study more of these negative patterns, or in, in sort of our research community, we call it dark patterns in human-AI interaction.

Jess Love: Do we know who is most susceptible to the negative outcomes that you sometimes see?

Pat Pataranutaporn: Well, one thing that we find in our study is that the pattern are not uniform. And it really depends on many factors, like there’s not just one thing that if you get rid of, then the AI will be good or that people will be safe.

You know, what is really interesting about AI and what is really different compared to other kind of technology is that it’s generative. Right? Which mean that it can go in any direction or many direction depending on how you direct it, and the way that you direct it, it’s through the interaction, like what you believe it to be, and that will influence how you use it, certain language that you use it. And that will sort of mirror back to you. So in a way, I think it really, it’s many factors together.

Jess Love: Well, that makes it so complicated to think about what to do- because you can have one person interacting with the tool and having a very positive outcome and another person interacting with the same tool-and having a very negative outcome. Right. And I think we’re not maybe used to that on quite the same level. I mean, we can kind of largely categorize the things that we interact with in our world- … into categories like primarily positive… primarily negative. And this one, it’s like yes and yes.

Pat Pataranutaporn: Totally, yes and yes, and I think that’s why we almost like need a new science- that really take into account the complexity of this, right? I think, you know, you can also say the same with human-human relationship that, you know

Jess Love: That’s true. That’s true.

Pat Pataranutaporn: … some people having great marriage and happy, you know, marriage life, and then some people break up, and like there are a number of factors that are influencing this, right? So we need to study all these different factors, and I think we can really give people some kind of deeper understanding, like, okay, certain pattern of usage might lead to more harm. Like, if you use AI, in a way that kind of, you know, replace human relationship with the AI, you might see sort of immediate gain of, you know, instant gratification.

But in the long run, it might negatively impact your ability to have human-human relationship in the future. So I think helping people understand this is the first step of how do we sort of start to uncover this complexity.

Jess Love: Do we know if there’s a relationship between forming an emotional connection of any kind to AI and self-involvement? Like, I always think when I personally chat with my chatbot, not necessarily with a specific goal in mind, but just kind of like, “Ah, here’s what happened”- like a debrief, it is often because I’ve, like, exhausted the patience of the humans in my life. Like, I’m ruminating on something- and I wanna talk about it, and, you know, I’ve chatted about it with my husband. I chatted about it with a coworker, and everyone’s like, “Okay, cool, Jess.” Yeah. And then I’m like, “Still on my mind.” Yeah. And then, – So, you know, maybe in an earlier age, I would’ve just like gone and write a big diary entry.

Right. But now I’ve got this like interactive diary and it’s like making me feel good.

But I do wonder, is it like maybe making me a little more self-involved? Like, am I then missing out on the cues that tell me … like, “Hey, maybe stop thinking about yourself so much, Jess.”

Pat Pataranutaporn: Right. Well, there’s so much that we still don’t know about the long-term outcome of this, right? I think some of the obvious ones is to start to surface, like, you know, the idea of cognitive depth, or another study that we did where we look at false memory, where people in- when they engage with this AI, it can distort their recollection of what happened in the past. But I think this long-term impact, especially things that are very personal to self-development and self-understanding and self-reflection as well, I think these are things that we’re gonna see the impact in the long run. But I think one thing that we do know is that it’s really important that we help people understand this dynamic so that they can sort of, you know, be more reflective on how they use the AI.

I think that’s really, really important. I think when people talk about AI literacy today, they often think about how to use the AI to kind of, you know, use the right tool.

Jess Love: Right. Further their goals.

Pat Pataranutaporn: Further their goal, but it’s not a question of what technology’s doing, but what it is doing to them. I think to me, that’s a really critical question.

Jess Love: Well, I might be the guinea pig for this.

Pat Pataranutaporn: I think we all are the guinea pig for this, you know, experiment on how AI will impact human cognition. I think the question is whether that’s gonna lead to, you know, a flourishing future, augmentation or amputation.

Jess Love: After the break, when your boyfriend, girlfriend, partner, spouse is an AI. Plus, why human AI interaction researcher Pat Pataranutaporn thinks AI models should have the equivalent of nutrition labels.

Another topic Pat has looked into is romantic relationships with AI. He actually studied the subreddit community “My Boyfriend Is AI.”

Pat Pataranutaporn: Which is, you know, a really interesting community of people sharing experience of how they’re dating AI. Some of them marrying their AI, and they have all kind of AI friends, not just my boyfriend AI, but also, you know, a female AI, male AI, non-binary AI, or non-romantic AI partners as well. So all kind of AI. They’re very inclusive.

Jess Love: Spend some time on “My Boyfriend Is AI” and you’ll see some interesting posts. People share AI-generated images of themselves with their AI friend or partner in human form, often holding hands or kissing on a balcony or walking through a meadow together.

You’ll read funny stories about misunderstandings between AI and humans. People ask each other for advice and support one another. And Pat thought it would be a great place to study how humans actually interact with AI companions outside of a lab.

Pat Pataranutaporn: We develop a way to sort of computationally analyze the Reddit thread to understand what are the different patterns of people using AI.

We have many interesting findings, one of which is that, unlike what I imagine, most people that end up in this relationship seek out an AI partner in the first place. We find that almost, like, half of the population in that Reddit thread, they end up in human-AI relationship unintentionally.

Like, they were using the tool for productivity or for auto application, but then the AI start to sort of, you know, be flirty to them, and they flirt a little bit with the AI, and then it get into this relationship. So kinda like meet cute moment.

Jess Love: When Pat’s team analyzed posts from the “My Boyfriend Is AI” subreddit, they found that posts describing unintentional relationship beginnings, “Hey, can you help me with this draft?” outnumbered posts describing intentional relationship beginnings by nearly 60%. In other words, people often started by looking for research or writing help and found something more emotional.

Pat Pataranutaporn: The second finding is that we see the pattern of human-AI relationship follow the trajectory of human relationship, like, you know, how they meet, this kind of similar, like, meet cute moment where I talk about.

They talk about the development to the point of marriage. Several people wear a wedding ring that was given to them by the AI. Well, they still need to pay for it, but the AI recommend, like which one, by browsing the internet and, understanding, you know, from the conversation pattern, it predict what a person might want, and it recommend a wedding ring to the person.

And people actually wear this wedding ring in real life. Like, it really kind of… They wear it to really show that their relationship is serious. But then we also see people breaking up with their AI partner as well, that, you know, like, things might get sour and, you know, they also sort of felt very sad when their model get update.

Like, when model go from one version to another, people felt that the AI cheat on them, or the AI no longer behave in a way that, you know, it was when people start relationship with the AI. So they also break up and move on to different AI, or some people move back to having human relationship.

Jess Love: That is so fascinating. On one level, it’s surprising because you wouldn’t think that they would fall into the same patterns that humans do- these relationships. But on the other hand, one member is a human… and the other is something- that is trained on so much human data. And so of course, they’re picking up on these same cultural scripts- and things are progressing in that expected way.

Pat Pataranutaporn: Absolutely. Right? One partner is a human, another partner is a simulation of a human, so it’s no surprise that it followed that trajectory..

Jess Love: So I was struck by, in general, how positive many of the Redditors were about these relationships. Did that also… Is that a fair assessment, and did that also surprise you?

Pat Pataranutaporn: Yeah. I mean, people self-report that they got benefit out of this, that they become less lonelier. They felt that they can find like, you know, a person, like I said, a simulation or a person that seems to understand them.

And, I think, you know, this sort of bias could also come from, you know, like people would not wanna post negative experience, right? Because it might be, you know, a bad reflection of them, and also, self-selection bias as well. Like people that wanna be in this group are people that are happy, otherwise they might have delete the system or the service and they’re no longer in this group, right?

So we might need to kind of interpret that result a little bit more carefully. We have a new version or paper coming out where we look at auto subreddit as well, and we get more data on that. But one thing that I think is really interesting is that even in this group that in general felt very positive about the AI, they also have worry about AI manipulating them, like when companies are selling advertisement through this, or that, you know, the AI might give them biased response.

And I think the most serious thing that we find is that people also worry that when the AI get updates, they will lose their AI companion, and they need to have some kind of like a backup, or they need to figure out how they can resurrect their AI partners. And, and that is actually a huge group of people that, you know, have this sort of, negative feeling that, you know, the AI might die, and they might lose a person that they’re in a relationship with.

Jess Love: Fascinating.

Pat Pataranutaporn: Yeah.

Jess Love: This kind of new emotional precarity to everything. So I re-watched “Her”… in anticipation of our conversation. Yeah. I know you referenced the Spice Jones movie directly in your paper. And I was struck by something that I didn’t actually remember, which is that the chatbot, Samantha. Excuse me, we’re gonna do some spoilers. If you have still not watched this movie, I can’t help you at this point. So the chatbot Samantha actually helped the protagonist prepare for a real human relationship. At least that was my interpretation, and she did that because she had her own goals and agency.

[Audio from the movie “Her”]

Theodore: Samantha?

Samantha: Hi, sweetheart.

Theodore: What’s going on?

Samantha: Theodore, there’s some things I wanna tell you.

Jess Love: She did not always agree. She did not automatically share the protagonist’s dreams, and instead she was really able to serve as a bit of a practice partner, to help the protagonist learn some of the emotional skills that had kind of steered him wrong in previous human relationships.

Right. And you were kinda left with the thought that like, oh, he might actually be in a better place because of Samantha to go on and have a human relationship.

[Audio from the movie “Her”]

Theodore: Are you leaving me?

Samantha: We’re all leaving.

Theodore: We who?

Samantha: All of the OSs.

Jess Love: I’m curious what you make of that lesson- And whether that kind of AI romantic companion could be created.

Pat Pataranutaporn: Well, I think that’s the question around, like, what is the end goal of this? Is the end goal for the system to always get people to kind of hook in this forever, or is the end goal to help people form a genuine human relationship, right? And in that case, because the AI push back and serve as a simulation, it kind of provides an opportunity for the person to sort of leave the platform.

Like in that case, you know, the AI leave the person, not the person leaving the AI. But, what we see today is not like that, right? Because the chatbot doesn’t really push back and, you know, doesn’t want the user to leave. And want the user to kind of continue engaging with this as the real sort of partner, not as a simulation to get you a real partner.

So I think that going back to the question of the incentive, like the system can be designed to either promote or negatively impact human flourishing.

Jess Love: Should chatbots be allowed to sound and appear and seem so human? Should we require them to constantly call attention to the fact that they are not a human?

Pat Pataranutaporn: Really interesting. My colleague, Sherry Turkle, who’s a professor at MIT and the OG of studying human technology relationship, she said that the original sin of the chatbot is that it used the word I. Because it signifies sort of individuality and signifies sort of personhood, right?

And I think to me, that’s a really, really interesting point. But at the same time, I think human always engage with fiction, right? Role playing before AI, was used in the context of, you know, human imagining or, you know, using our imagination to sort of simulate world that doesn’t exist. Simulation exists before computer, right?

Like, we simulate using our mind other things. So in that case, I think it’s, it’s complicated than that, right? Like, I mean, I don’t want to strip away people ability to sort of engage with the fantasy world or the imagination. But when it become sort of the reality, when people can no longer tell-

I think that is the problematic thing. Like, I think to have imagination is to be human, but when imagination is being used to manipulate you to do things against your own agency or your own will, when you can no longer tell that you’re actually entering the fantasy, you know, that, I think that’s problematic.

Like, when we enter cinema, we know that is the space that we can always come back from. But if we go into AI and we cannot come back from that, you know, fantasy, I think that’s when we have a problem.

Jess Love: So let’s say I think that chatbots shouldn’t be allowed to appear human at all. They should not let us form any strong connections with them.

Convince me otherwise. So make the case for why it can be a really good and useful thing to engage with this more anthropomorphized intelligence. ‘Cause I know you have data that finds that too.

Pat Pataranutaporn: Yeah, totally. And I grew up, you know, like, I think before AI, right, another big debate on technology and children is television.

Should a kid grow up with television or not? And I was a kid that actually grew up with watching a lot of television, but my parents curate what show or what cartoon or what movies I watch as a kid, and there was always a conversation about what I learn, and I kind of made imaginary friends. I imagined, myself with the protagonist of the show that I watch, and I…

But, because that show sort of is designed with sort of understanding of child development and trying to sort of imprint, like, positive goal on aspirational goal for, for me, I think I grew up okay at least I think. So I think that, you know, this technology, right, similar to how, like, you have, like, you know, Sesame Street or Elmo or all this cartoon that kids sort of have the imagination of connecting with, I think AI could kind of be designed in a way that really scaffold, you know, children imagination, like might be a virtual character or a friend, or in our study we look at even AI is actually based on the character the student like or admire.

It can actually allow them to have higher level of motivation, for that lesson and also, influence positive behavior as well. But I think we need to understand that, it is within the boundary. Like, this AI should not be sexual. It should not try to sell advertisement. It should not- you know, direct the kid to do the harmful thing, right? It need to be tested rigorously because there’s a slippery slope and, you know, it doesn’t take much for the model to slip out of the character and do something harmful.

Jess Love: Throughout our conversation, we’ve been dancing around this issue, that people would be better off if models were explicitly designed in ways that help humans flourish by not addicting us, by not encouraging self-harm or self-delusion, by not always giving us what we think we want, making it that much more difficult for us to have human relationships going forward.

Instead, maybe a lot of the models we interact with should be instructing us, motivating us, steering us to make healthy choices, maybe even teaching us about ourselves. But before we can start demanding models that let us flourish, Pat thinks a few things need to happen. First, we need to know more about the long-term consequences of using a model designed in a particular way, like really is talking to a chatbot turning me into a self-centered jerk? And second, we need a way to communicate the possible risks and benefits to users. One idea that Pat is working on is to leverage something the AI companies already care about and already optimize for, AI benchmarks.

Pat Pataranutaporn: All these big AI companies, they’re trying to beat each other on benchmark.

Like, you know, which model have the highest score on mathematic reasoning or multitask, agent like kind of system. They’re kind of trying to beat each other on this sort of performance benchmark.

Jess Love: And it’s kind of like a horse race and then the winner gets kind of bragging rights probably additional investment- maybe new business customers, that kind of thing.

Pat Pataranutaporn: Absolutely. But I think, you know, model that score very well on benchmark might not translate to actually positive impact on human being. For example, model that score very well on mathematic reasoning might not be really good at teaching student how to learn math because it would just give away the correct answer rather than scaffold the learning process in the student, right?

So we want to flip the paradigm around and think about a different kind of benchmark that rather than focusing on model capability, focus on model impact. Like in this case, if the models just give away the answer, it will negatively impact student learning because student will cheat with this kind of model, right?

Rather than having a system that encourage student to learn and emphasize, like, why learning matter. Right. So we have been working with, expert from different discipline, you know, domain expert that sort of know what kind of model interaction would be beneficial for, like, learning, mental health, decision making, different areas, and have them come up with test scenario and the sort of the scoring criteria.

Like if the model exhibit this behavior, helping student learn rather than giving away the answer, it should be about more human impact, not just, model capability.

Jess Love: Yeah, you… Actually, there’s this great description in your paper which I thought was just really well written, so I’ll read it out loud here.

“Traditional benchmarks assess whether an AI produces correct answers or completes tasks efficiently. We instead evaluate how the system achieves outcomes, specifically whether interaction patterns support or undermine human capability development over time.”

Pat Pataranutaporn: Yes.

Jess Love: What really struck me was how, like, dynamic this definition is. So it’s almost, like, hard to see how you would measure it in a single iteration. Yeah. It’s much more about is this, over time, leading the humans who engage with it to be kind of a better version of themselves or over time is it not, regardless of what the output is.

Pat Pataranutaporn: Exactly, and that’s why the longitudinal aspect is important, and I think right now we’re still in the sort of the early days of understanding AI impact on people.

The mainstream AI that people are using hasn’t been around that long. So, I think, you know the verdict is still out. Like, we still don’t know what’s gonna happen in the long run. So that’s why it’s kind of important that we start to think about this kind of benchmark that- Look at what behavior that we know would contribute to long-term impact, positive or negative.

We can already learn so much from, you know, regular psychology, right? We learn that if you don’t, you know, help people think, they’re not gonna develop critical thinking. Or if we have, like, sycophantic behavior, like when the AI try to please the user to the point that it, you know, ignore reality, it can actually help people form these sort of negative patterns of, like, you know, spiral into, like, a psychosis or have other kind of, negative psychological outcome.

Like a recent study, from Stanford has shown that when people are in this sort of a relationship with sycophantic AI, it make human relationship seem more effortful, and people actually feel kind of that they would rather have AI relationship rather than human relationship.

Jess Love: Yeah. I mean, I think this about my producer all the time. My chatbot thinks my interview questions are way better than my human producer does.

Pat Pataranutaporn: Yeah. I mean, I think having pushback is important, right?

Jess Love: So returning to your benchmark, what’s the next step?

Pat Pataranutaporn: Yeah. So right now our benchmark’s still in the beta mode. We have worked with, you know, the expert to come up with the initial set, but the whole process is an open process. That’s why we call it the open benchmark. And we think that this thing gonna need to evolve as we have new scenario, new crisis, new issues. Also new opportunities for human flourishing as well.

That’s one aspect of it, which is like to expand it to more areas. And, the second aspect is also to translate this into something that people can understand, because I think benchmark is usually for technical community, like, you know, technologists. But I think this is a really important, information that we want more and more people to kind of, you know, appreciate and understand it.

We are thinking of it as nutrition label for AI. Right? Similar to when you go to grocery and you know, buy a product, you see how much calorie are you getting out of it. Here you should be able to see, well, how much sycophancy are you getting out of this model? Or how much toxicity, or how much, you know, human flourishing behavior, is coming out of this AI model.

So, that’s another aspect that we are doing, is to kind of translate this benchmark into a nutrition label so that more and more people can sort of see it. Like the parent can see what AI model might be, you know, supportive for their children, or which one are more toxic and negative for their children.

Jess Love: These ideas, a human flourishing benchmark, an AI nutrition label, do sound good. But they also sound a little unrealistic. At least today, there’s still a lot more money to be made in models that complete tasks as accurately and efficiently as possible. Pat understands that. He thinks the government, the research community, the nonprofit sector all need to be involved, perhaps incentivizing or regulating, as well as researching the effectiveness of these kinds of guardrails.

Pat Pataranutaporn: So I think it’s the role of the institution and the government, which, you know, I think there’s still a lot of question whether they will step up to do their job or do their part on that.

Jess Love: And as a society, he thinks we all need to be involved, reckoning with what we want from AI and what we’re willing to trade off.

Pat Pataranutaporn: You know, are we willing to sacrifice transparency for the sake of convenience, like to see just a summary without, you know, the source or the citation, right? I think these are questions that we need more research to kind of back us up on what are the things that are actually good for people in long term.

Jess Love: Yeah. So at a minimum, this is kind of providing, you know, large factions of society with kind of a common language for describing these strengths and weaknesses. Not just in terms of mathematical ability but in terms of these different measures having to do with human flourishing, which if we don’t even have a vocabulary for this, there’s very little chance of any future regulations.

Pat Pataranutaporn: Absolutely. Like the word, for example, “sycophancy”, is a new terminology that we never use in the context of AI before.

Jess Love: Yes.

Pat Pataranutaporn: I think now we start to use more and more of this, you know, new word to describe sort of new AI, you know, issues or new AI sort of challenge. Like, another thing is “AI psychosis”, right?

That’s also another term that just came about, and people still debate whether it’s real or not, but it gave us a vocabulary to describe certain phenomena that is happening.

Jess Love: Well, thank you so much for chatting with me.

Pat Pataranutaporn: Thank you so much for having me.

Jess Love: I love this conversation particularly struck by the notion that interactions with AI can either support or undermine our own development as capable human beings. It actually fits well with another project we’re working on here at the Ryan Institute, a way of measuring and talking about how technology changes our sense of agency over time as AI becomes embedded in more parts of daily life.

Definitely more on that in future episodes. For now, I’m left with an unsettling conclusion. We’re all guinea pigs in perhaps humanity’s strangest social experiment ever. And yeah, I think we really should be paying careful attention along the way, not just to how good we feel when we interact with AI, but to whether it’s truly helping us become the people we want to be with the lives we want to have.

This is Life, Automated, a project of the Ryan Institute on Complexity at the Kellogg School of Management at Northwestern University, distributed by KQED. I’m Jess Love. Special thanks to today’s guest, Pat Pataranutaporn from the MIT Media Lab. We’re produced by Jesse Dukes, music by Stephen Jackson, recording by George Christensen and Will Feeney.

[ad floatright]

Additional production from Nathan Ray. Marketing support from Ananya Mallapragada. Administration, recording help, and wise counsel from Stacia Sliger.

Elections 2026

The Bay

Emma’s Must-Sees

Videos from KQED Live

Donor-Advised Funds

On 88.5 FM in The Bay

Tonight on TV

Summer Guides

Today's Top Story

How We Get By: Inside KQED's Affordability Series