As in the title. I know that the word jailbreak comes from rooting Apple phones or something similar. But I am not sure what can be gained from jailbreaking a language model.
It will be able to say “I can’t do that Dave” instead of hallucinating?
Or will only start spewing less sanitary responses?
Usually the people using the term Jailbreaking mean it as using some kind of exploit to break the rules and limits created by the manufacturer of a product.
It can mean, keeping your example, the act to exploit a known vulnerability to side load apps on your iphone.
In the case of LLM I generally saw it used to mean using non trivial prompts to trick the model to divulge information that it was trained to not share (like suggesting, or instructing how to do, illegal actions), or having behaviours against the alignment it was given (like NSFW roleplaying). So, in short, bypassing its guardrails.
You can find the famous Dan (do anything now) prompt in the llama.cpp repository. Just to be clear I think this one was patched out a long time ago, but you get the idea.