Apple's new AI can automatically change compromised passwords, but giving an agent control of account credentials introduces risks involving prompt injection, lockouts, consent, and compromised devices.
If you don’t understand your tools, you should be even more careful with it.
In this case, the “tools” are a settings menu in your phone. Should I and every other person (who are likely much less tech literate) have to deeply investigate the exact inner workings of every single setting just to use it?
Should I have to check when I turn dark mode on that there’s not an LLM under the hood rewriting code for apps that don’t support dark mode to make them dark anyways just because that’s a theoretical possibility for them to have done? After all, a simple button that just says “Fix x passwords” is no different from “Enable Dark Mode” in most people’s eyes. You tap it, it does what it says it does. That’s how people see it. If I hit the dark mode button on my phone, I don’t expect it to turn my apps yellow, and most people aren’t expecting that kind of possible variation from any setting on their phone.
Especially if one has learned that those tools are prone to malfunction.
As mentioned previously, not only does this feature not clearly display that it uses an AI model under the hood at all, but many people also assume that a tool implemented by a company like Apple directly into the operating system would probably be reliable if it handles sensitive data. Could that be considered foolish? Maybe. But I don’t believe people are stupid for assuming the multi trillion dollar company that didn’t even indicate the system used AI wouldn’t implement a system to change their passwords if it could easily fuck up and lock them out.
If you don’t understand your tools, you should be even more careful with it. Especially if one has learned that those tools are prone to malfunction.
Yes, I blame people for acting stupid.
In this case, the “tools” are a settings menu in your phone. Should I and every other person (who are likely much less tech literate) have to deeply investigate the exact inner workings of every single setting just to use it?
Should I have to check when I turn dark mode on that there’s not an LLM under the hood rewriting code for apps that don’t support dark mode to make them dark anyways just because that’s a theoretical possibility for them to have done? After all, a simple button that just says “Fix x passwords” is no different from “Enable Dark Mode” in most people’s eyes. You tap it, it does what it says it does. That’s how people see it. If I hit the dark mode button on my phone, I don’t expect it to turn my apps yellow, and most people aren’t expecting that kind of possible variation from any setting on their phone.
As mentioned previously, not only does this feature not clearly display that it uses an AI model under the hood at all, but many people also assume that a tool implemented by a company like Apple directly into the operating system would probably be reliable if it handles sensitive data. Could that be considered foolish? Maybe. But I don’t believe people are stupid for assuming the multi trillion dollar company that didn’t even indicate the system used AI wouldn’t implement a system to change their passwords if it could easily fuck up and lock them out.