cross-posted from: https://sopuli.xyz/post/48033292
“Role tags were a formatting trick that became the security architecture and the cognitive scaffolding of modern LLMs,” the authors explain in a blog post. “We’ve shown that this architecture doesn’t survive into the model’s actual representations, and that such role confusion is linked to prompt injection.”
Ffs, cocaine isn’t some carefully guarded secret recipe, that’s the other coke
The important part is they were easily able to trick the LLM into doing something it was told not to do.
Which has serious implications for any LLM that’s ever put in charge of anything important or given access to any sensitive data it shouldn’t divulge. (Such as Instagram’s customer service AI, which hackers repeatedly prompt-injected into giving them access to other people’s accounts.)
Yeah, sorry. Was making a joke and it really needed a /s or /j because of the way I phrased it.
But, thank you very much for the explanation! One of the beautiful things about lemmy is people like you stepping up with good information. Completely wonderful thing to do :)
Can you explain?
Sure!
Coca-Cola is made by a recipe that has been kept secret pretty much since its creation. It is colloquially referred to by just the word “Coke”.
Cocaine is a hell of a drug. It is sometimes referred to by the word “coke”. However, the recipe for making it is dead simple and well known among the populace of the world, and very well known in areas where the plant it is made from is grown.
At one point, Coca-Cola included the same plant product in its recipe, and is where the “Coca” part of its name comes from. Thus, the joke also referred to that shared history as a neat little package.
Not really the best joke, tbh. But they can’t all be winners ¯\_(ツ)_/¯
Coca Cola actually still contains coca in it for the flavor, it just has the cocaine drug portion removed from it. They do this so that the drink still has the same flavor as it originally did, just without the extra kick
They remove it and then sell it to pharmaceutical companies, making Coca Cola the largest producer and distributor of cocaine in the entire world.
I thought they had synthesized the flavoring! Kinda cool, but also kinda weird that they’re now coke dealers in both senses (again) lol.
