We do know that they can be coaxed into spitting out the original work, though, which sure implies it is in there.

Only very rarely, under extreme cases of overfitting. Overfitting is a failure state that LLM trainers want to avoid anyway, for reasons unrelated to copyright.

There simply isn’t enough space in a LLM’s neural network to be storing actual copies of the training data. It’s impossible, from a data compression perspective, to fit it in there.