Apple's new AI can automatically change compromised passwords, but giving an agent control of account credentials introduces risks involving prompt injection, lockouts, consent, and compromised devices.
I don’t blame people for it, honestly. If this massive company is telling them that this tool will work as advertised, and clearly believes in it because it’s being widely rolled out across their devices, most people who simply don’t understand the intricacies will believe they’re not being lied to, or at least deceived as to how capable the feature really could be against adversarial attacks or unfavorable circumstances.
People aren’t necessarily stupid per se, they just aren’t constantly skeptical of every single possible claim, and familiar enough with the underlying subjects to understand it in the first place.
In this case, it’s genuinely just “hit the big blue ‘Fix x passwords’ button” and it’s done. Most users won’t even realize it uses an AI model under the hood, and will just assume it’s something to do with how the internet works. And honestly, can we really blame people for that when it’s designed in such a simplistic manner?
If you don’t understand your tools, you should be even more careful with it.
In this case, the “tools” are a settings menu in your phone. Should I and every other person (who are likely much less tech literate) have to deeply investigate the exact inner workings of every single setting just to use it?
Should I have to check when I turn dark mode on that there’s not an LLM under the hood rewriting code for apps that don’t support dark mode to make them dark anyways just because that’s a theoretical possibility for them to have done? After all, a simple button that just says “Fix x passwords” is no different from “Enable Dark Mode” in most people’s eyes. You tap it, it does what it says it does. That’s how people see it. If I hit the dark mode button on my phone, I don’t expect it to turn my apps yellow, and most people aren’t expecting that kind of possible variation from any setting on their phone.
Especially if one has learned that those tools are prone to malfunction.
As mentioned previously, not only does this feature not clearly display that it uses an AI model under the hood at all, but many people also assume that a tool implemented by a company like Apple directly into the operating system would probably be reliable if it handles sensitive data. Could that be considered foolish? Maybe. But I don’t believe people are stupid for assuming the multi trillion dollar company that didn’t even indicate the system used AI wouldn’t implement a system to change their passwords if it could easily fuck up and lock them out.
I don’t blame people for it, honestly. If this massive company is telling them that this tool will work as advertised, and clearly believes in it because it’s being widely rolled out across their devices, most people who simply don’t understand the intricacies will believe they’re not being lied to, or at least deceived as to how capable the feature really could be against adversarial attacks or unfavorable circumstances.
People aren’t necessarily stupid per se, they just aren’t constantly skeptical of every single possible claim, and familiar enough with the underlying subjects to understand it in the first place.
In this case, it’s genuinely just “hit the big blue ‘Fix x passwords’ button” and it’s done. Most users won’t even realize it uses an AI model under the hood, and will just assume it’s something to do with how the internet works. And honestly, can we really blame people for that when it’s designed in such a simplistic manner?
If you don’t understand your tools, you should be even more careful with it. Especially if one has learned that those tools are prone to malfunction.
Yes, I blame people for acting stupid.
In this case, the “tools” are a settings menu in your phone. Should I and every other person (who are likely much less tech literate) have to deeply investigate the exact inner workings of every single setting just to use it?
Should I have to check when I turn dark mode on that there’s not an LLM under the hood rewriting code for apps that don’t support dark mode to make them dark anyways just because that’s a theoretical possibility for them to have done? After all, a simple button that just says “Fix x passwords” is no different from “Enable Dark Mode” in most people’s eyes. You tap it, it does what it says it does. That’s how people see it. If I hit the dark mode button on my phone, I don’t expect it to turn my apps yellow, and most people aren’t expecting that kind of possible variation from any setting on their phone.
As mentioned previously, not only does this feature not clearly display that it uses an AI model under the hood at all, but many people also assume that a tool implemented by a company like Apple directly into the operating system would probably be reliable if it handles sensitive data. Could that be considered foolish? Maybe. But I don’t believe people are stupid for assuming the multi trillion dollar company that didn’t even indicate the system used AI wouldn’t implement a system to change their passwords if it could easily fuck up and lock them out.