Similar related issue. My family has been dealing with some medical issues lately and my mom has been using Claude a lot to explain various complex issues, analyze prognoses, etc. and I’ve noticed that it’s often excessively positive. Like everything that happens is super positive news and a huge step towards recovery.
Now I’m not a medical professional so I don’t know for sure it’s wrong but it sets off my BS alarm for sure.
Oh God, I never really thought about that in terms of AI for medical use but that actually makes AI sycophancy so much more terrifying.
Especially thinking in terms of how it could be used to dissuade people from seeking preventative care. (Which lets be honest, is probably exactly what is intended by at least some of the billionaires promoting AI in medicine as a convenient way to avoid having to go to the doctor for “every little thing.”)
On the contrary I’ve used it a bit for benign ouchies and it is very conventional and will always always refer you to an actual doctor.
But it might be another sycophancy aspect, and I wouldn’t be surprised if it went off the rail when you prime it with stuff like “give me a natural holistic ancestral remedy for XYZ”
Well I wouldn’t say it has discouraged care. Frankly it has been more accurate than I would have expected. It’s more that whenever you give it an update it goes wow that’s fantastic news! But then if you read the actual details it’s not so fantastic lol. So it will maintain this sort of fake positivity even as it accurately explains some of the bad things that could happen.
“I want to go on a murderous rampage”
Great idea! Here’s some tips for staying safe and getting the most out of your adventure:
Satire is dead. I’m pretty sure I saw something about someone being able to get one of ‘em to help them plan a mass shooting so…
We are going to see so many new and interesting psychological and subsequent societal impacts from the global adoption of conversational AI…
Don’t people get the same validation from humans? I’m not convinced that this shows a problem with AI specifically.
The study compares rate of sycophancy of AI compared to human responses, the point is not to say only AI does it, it is to say that it focuses on pleasing more than actual humans do and that has adverse effects
It does not say humans are good at it, but that AI is worse
Oh, I completely missed that there was a linked study, I was just reacting to the image in isolation. That is interesting to know though
Less often
I guess you can get some of that effect in specific echo chambers.
That’s blaming technology for a human problem.
———
If you remove AI from the scenario, that human is still a problem.
AI in the hands of a human without those issues, is a non-issue.
Therefore, the problem is very clearly in front of the screen, not behind it.
This sounds like guns don’t kill people, people do!
This is a whataboutism. In the same way that I view nicotine, gambling, alcohol companies as wrong to try to trigger existing neuroses in their customers with their ads, ai companies are unethical for their role in triggering antisocial behavior, and even promoting it, in their users. “What about psychotic people” isn’t a valid defense of llms.
They are “guns don’t kill people”-ing it. Ignore them.
This is bullshitism, it is the same as saying that “alcohol companies are unethical for their role in triggering drunk driving”. I don’t see how AI companies despite of their many issues, are somehow intentionally triggering any antisocial behavior. Most of the “ethical problems” that we have with LLMs are a result of the user being retarded.
Hey asshole, this isn’t an unanswered question. At least click the link before jumping to your gf’s defense:
Here, we show that sycophancy is widespread in leading AI systems and has harmful effects on users’ social judgments.
Hey asshole
Do you scream in people’s faces in real life? Because this comment makes it seem like you scream in people’s faces in real life
That’s the word you found offensive? Not their ablist slur, or my intimating about their love life? Or even bullshit? How is asshole worse than bullshit? Where do you think bullshit comes from?
Hey asshole, that doesn’t mean it was intended to trigger antisocial behavior. Don’t assign malice to ignorance. Notice that this doesn’t mean that corporations are unethical by design, just saying that this doesn’t prove it was a deliberate choice.
The problem with this platform in general and with AI “Ethics” is that hate obfuscates the discussion.
I don’t give a fuck if it’s deliberate. I think it is, and the article gives good reason to believe this is a choice by tech corpos to boost engagement, but I also don’t think it matters really why they are being harmful.
If you noticed, I was replying to the comment affirming it was a deliberate choice and also making an arrogant appeal to whataboutism. I had a quick read on the paper, and it also wasn’t clear to me how they even measure sycophancy. They talk about this “action endorsement rate”, but it wasn’t very clear whether the data annotation (from a really large sample apparently) was done by hand or done automatically. In either case, it would remain to the researchers involved to subjectively define whether an answer is sycophantic or not.
One thing you should know about most of this ethic research, is that it is bullshit. The people are a bunch of very opinionated lunatics that can’t see beyond the vomit they spew to each other in a pseudo academic tone.
I hate capitalism and corporations as much as any other lemmy user. But when it comes to AI, you have to keep in mind that is is a technology, and whether technology is “good” or “bad” depends entirely on the people using it.
Fuck you and have a good day
Good day sir! Well slopped!
Yet despite distorting judgment, sycophantic models were trusted and preferred. All of these effects persisted when controlling for individual traits such as demographics and prior familiarity with AI; perceived response source; and response style. This creates perverse incentives for sycophancy to persist: The very feature that causes harm also drives engagement.
AI doesn’t have to be sycophantic, but it is. If people really believe it can be used to change the world for the better, they’re going to have to start by acknowledging what human traits have made the world so fucked up in the first place.
It reminds of one of the best parenting tips I was ever given by somebody who was raised by a parent with a maternal instinct in the negative range. She basically said something like when I’m not sure what to do as a parent, I start by thinking about what my mom would have done in the situation. “Step one: Ok, don’t do that…”
Even if there can be no consensus on how AI should be used to improve the world, it does seem like we should at least be able to agree on the maladaptive traits we know we want to avoid passing on since they definitely aren’t doing us any favors as a species.
