AI models don’t resynthesize their training data. They use their training data to determine parameters which enable them to predict a response to an input.
Consider a simple model (too simple to be called AI but really the underlying concepts are very similar) - a linear regression. In linear regression we produce a model which follows a straight line through the “middle” of our training data. We can then use this to predict values outside the range of the original data - albeit will less certainty about the likely error.
In the same way, an LLM can give answers to questions that were never asked in its training data - it’s not taking that data and shuffling it around, it’s synthesising an answer by predicting tokens. Also similarly, it does this less well the further outside the training data you go. Feed them the right gibberish and it doesn’t know how to respond. ChatGPT is very good at dealing with nonsense, but if you’ve ever worked with simpler LLMs you’ll know that typos can throw them off notably… They still respond OK, but things get weirder as they go.
Now it’s certainly true that (at least some) models were trained on CSAM, but it’s also definitely possible that a model that wasn’t could still produce sexual content featuring children. It’s training set need only contain enough disparate elements for it to correctly predict what the prompt is asking for. For example, if the training set contained images of children it will “know” what children look like, and if it contains pornography it will “know” what pornography looks like - conceivably it could mix these two together to produce generated CSAM. It will probably look odd, if I had to guess? Like LLMs struggling with typos, and regression models being unreliable outside their training range, image generation of something totally outside the training set is going to be a bit weird, but it will still work.
None of this is to defend generating AI CSAM, to be clear, just to say that it is possible to generate things that a model hasn’t “seen”.
Okay for anyone who might be confused on how a model that’s not been trained on something can come up with something it wasn’t trained for, a rough example of this is antialiasing.
In the simplest of terms antialiasing looks at a vector over a particular grid, sees what percentage it is covering, and then applies that percentage to to shade the image and reduce the jaggies.
There’s no information to do this in the vector itself, it’s the math that is what is giving the extra information. We’re creating information from a source that did not originally have it. Now, yeah this is really simple approach and it might have you go “well technically we didn’t create any new information”.
At the end of the day, a tensor is a bunch of numbers that give weights to how pixels should arrange themselves on the canvas. We have weights that show us how to fall pixels to an adult. We have weights that show us how to fall pixels to children. We have weights that show us how to fall pixels to a nude adult. There’s ways to adapt the lower order ranking of weights to find new approximations. I mean, that’s literally what LoRAs do. I mean that’s literally their name, Low-Rank Adaptation. As you train on this new novel approach, you can wrap that into a textual inversion. That’s what that does, it allows an ontological approach to particular weights within a model.
Another way to think of this. Six finger people in AI art. I assure you that no model was fed six fingered subjects, so where do they come from? The answer is that the six finger person is a complex “averaging” of the tensors that make up the model’s weights. We’re getting new information where there originally was none.
We have to remember that these models ARE NOT databases. They are just multidimensional weights that tell pixels from a random seed where to go to in the next step in the diffusion process. If you text2image “hand” then there’s a set of weights that push pixels around to form the average value of a hand. What it settles into could be a four fingered hand, five fingers, or six fingers, depends on the seed and how hard the diffuser should follow the guidance scale for that particular prompt’s weight. But it’s distinctly not recalling pixel for pixel some image it has seen earlier. It just has a bunch of averages of where pixels should go if someone says hand.
You can generate something new from the average of complex tensors. You can put your thumb on the scale for some of those weights, give new maths to find new averages, and then when it’s getting close to the target you’re after use a textual inversion to give a label to this “new” average you’ve discovered in the weights.
Antialiasing doesn’t feel like new information is being added, but it is. That’s how we can take the actual pixels being pushed out by a program and turn it into a smooth line that the program did not distinctly produce. I get that it feels like a stretch to go from antialiasing to generating completely novel information. But it’s just numbers driving where pixels get moved to, it’s maths, there’s not really a lot of magic in these things. And given enough energy, anyone can push numbers to do things they weren’t supposed to do in the first place.
The way models that come from folks who need their models to be on the up and up is to ensure that particular averages don’t happen. Like say we want to avoid outcome B’, but you can average A and C to arrive at B’. Then what you need is to add a negative weight to the formula. This is basically training A and C to average to something like R’ that’s really far from the point that we want to avoid. But like any number, if we know the outcome is R’ for an average of A and C, we can add low rank weights that don’t require new layers within the model. We can just say, anything with R’ needs -P’ weight, now because of averages we could land on C’ but we could also land on A’ or B’ our target. We don’t need to recalculate the approximation of the weights that A and C give R’ within the model.
AI models don’t resynthesize their training data. They use their training data to determine parameters which enable them to predict a response to an input.
Consider a simple model (too simple to be called AI but really the underlying concepts are very similar) - a linear regression. In linear regression we produce a model which follows a straight line through the “middle” of our training data. We can then use this to predict values outside the range of the original data - albeit will less certainty about the likely error.
In the same way, an LLM can give answers to questions that were never asked in its training data - it’s not taking that data and shuffling it around, it’s synthesising an answer by predicting tokens. Also similarly, it does this less well the further outside the training data you go. Feed them the right gibberish and it doesn’t know how to respond. ChatGPT is very good at dealing with nonsense, but if you’ve ever worked with simpler LLMs you’ll know that typos can throw them off notably… They still respond OK, but things get weirder as they go.
Now it’s certainly true that (at least some) models were trained on CSAM, but it’s also definitely possible that a model that wasn’t could still produce sexual content featuring children. It’s training set need only contain enough disparate elements for it to correctly predict what the prompt is asking for. For example, if the training set contained images of children it will “know” what children look like, and if it contains pornography it will “know” what pornography looks like - conceivably it could mix these two together to produce generated CSAM. It will probably look odd, if I had to guess? Like LLMs struggling with typos, and regression models being unreliable outside their training range, image generation of something totally outside the training set is going to be a bit weird, but it will still work.
None of this is to defend generating AI CSAM, to be clear, just to say that it is possible to generate things that a model hasn’t “seen”.
Okay for anyone who might be confused on how a model that’s not been trained on something can come up with something it wasn’t trained for, a rough example of this is antialiasing.
In the simplest of terms antialiasing looks at a vector over a particular grid, sees what percentage it is covering, and then applies that percentage to to shade the image and reduce the jaggies.
There’s no information to do this in the vector itself, it’s the math that is what is giving the extra information. We’re creating information from a source that did not originally have it. Now, yeah this is really simple approach and it might have you go “well technically we didn’t create any new information”.
At the end of the day, a tensor is a bunch of numbers that give weights to how pixels should arrange themselves on the canvas. We have weights that show us how to fall pixels to an adult. We have weights that show us how to fall pixels to children. We have weights that show us how to fall pixels to a nude adult. There’s ways to adapt the lower order ranking of weights to find new approximations. I mean, that’s literally what LoRAs do. I mean that’s literally their name, Low-Rank Adaptation. As you train on this new novel approach, you can wrap that into a textual inversion. That’s what that does, it allows an ontological approach to particular weights within a model.
Another way to think of this. Six finger people in AI art. I assure you that no model was fed six fingered subjects, so where do they come from? The answer is that the six finger person is a complex “averaging” of the tensors that make up the model’s weights. We’re getting new information where there originally was none.
We have to remember that these models ARE NOT databases. They are just multidimensional weights that tell pixels from a random seed where to go to in the next step in the diffusion process. If you text2image “hand” then there’s a set of weights that push pixels around to form the average value of a hand. What it settles into could be a four fingered hand, five fingers, or six fingers, depends on the seed and how hard the diffuser should follow the guidance scale for that particular prompt’s weight. But it’s distinctly not recalling pixel for pixel some image it has seen earlier. It just has a bunch of averages of where pixels should go if someone says hand.
You can generate something new from the average of complex tensors. You can put your thumb on the scale for some of those weights, give new maths to find new averages, and then when it’s getting close to the target you’re after use a textual inversion to give a label to this “new” average you’ve discovered in the weights.
Antialiasing doesn’t feel like new information is being added, but it is. That’s how we can take the actual pixels being pushed out by a program and turn it into a smooth line that the program did not distinctly produce. I get that it feels like a stretch to go from antialiasing to generating completely novel information. But it’s just numbers driving where pixels get moved to, it’s maths, there’s not really a lot of magic in these things. And given enough energy, anyone can push numbers to do things they weren’t supposed to do in the first place.
The way models that come from folks who need their models to be on the up and up is to ensure that particular averages don’t happen. Like say we want to avoid outcome B’, but you can average A and C to arrive at B’. Then what you need is to add a negative weight to the formula. This is basically training A and C to average to something like R’ that’s really far from the point that we want to avoid. But like any number, if we know the outcome is R’ for an average of A and C, we can add low rank weights that don’t require new layers within the model. We can just say, anything with R’ needs -P’ weight, now because of averages we could land on C’ but we could also land on A’ or B’ our target. We don’t need to recalculate the approximation of the weights that A and C give R’ within the model.