LLM AI

Model collapse is not coming to save you

Recently I was sitting in a hotel restaurant, talking with a few people from my profession about AI and software development. As usual, opinions were all over the place. At some point somebody brought up model collapse: eventually these things would stop getting better, perhaps even get worse, because they would increasingly be trained on their own output.

I hadn't heard that argument in a while. It got a lot of attention about two years ago, when research showed how models trained repeatedly on generated data could lose the rare and unusual parts of what they had learned. The weird edges disappear first, and with each generation more of the quality goes with them.

What struck me back then was how relieved some people seemed to be. They were uncomfortable with AI and what it might mean for their work and the world around them. Here was an argument suggesting the whole thing would eventually stop working anyway. I think a lot of people found that reassuring and latched onto it without asking how likely that outcome actually was.

My reaction then was much the same as it is now. Yes, model collapse can happen. But I don't buy the jump from a training process that degrades a model to an entire industry inevitably making its products useless.

Where does the real data go?

One way to get into trouble is to keep replacing training data with the output of the previous model. Information gets lost along the way, and the next generation has no way to recover what it never sees.

But why would we keep doing that once we know it's a problem?

The older human-generated material doesn't disappear when a model produces something new. We can keep it. And people haven't stopped producing new material either. Research gets published, people argue on forums, software breaks in ways nobody anticipated. We write code, compile it, test it, deploy it, and find out what we got wrong. All of this produces information about things that actually happened.

Research on accumulating real and synthetic data examined an interesting distinction. Replacing the training data with each generation's output led to collapse, while retaining the original real data and accumulating successive generations of synthetic data alongside it avoided collapse in the settings tested. This didn't even require a continually growing supply of fresh human material. The paper also derives a bounded error under its mathematical assumptions.

That doesn't prove every mixture of real and generated data is safe. Keeping some real data in the mix isn't automatically enough to prevent degradation. It does show that how we construct the dataset matters. A prediction about the future of AI has to account for our ability to change that process.

Some output survives, some doesn't

I think the comparison with evolution is useful here. Evolution produces plenty of failures and doesn't guarantee improvement. But variation and selection matter. Not everything gets passed on equally.

We already do this with generated output. Something useful gets incorporated into a project and built upon. Something useless gets rejected or corrected. Most ideas people come up with aren't particularly good either. We discard a lot of them after trying to do something with them. I don't see why that process would stop because some of the ideas came from a model.

Of course, being copied or linked doesn't make something true. Wrong answers can spread very well, and useful but unusual information can get overlooked. The web is a terrible curator, but it is still a curator. And we have better ways to judge some kinds of output than counting how often they appear.

Take generated code. Does it compile? Does it pass tests that check the intended behavior? Does it actually solve the problem when someone uses it? Those checks don't catch everything, but they give us reasons to keep or reject the result beyond another model finding it plausible.

That is still contact with reality, even if AI wrote the code. We don't have to accept every generated answer and feed it straight back into the next model.

Money is a pretty good filter

Training a frontier model costs a lot of money. I don't expect people to keep spending it on progressively worse results without changing what they're doing.

A lab that trains mostly on generated garbage is competing with labs that invest in fresh data, filtering, curation, and human feedback. I expect the latter to do better. If a particular training process loses useful information, that's a problem to investigate and a reason to change the process. The incentives don't guarantee a solution, but assuming nobody finds one is quite an assumption too.

This goes beyond the labs. People using these systems have reasons to care whether they work. Platforms hosting content have reasons to care whether readers can find anything useful. A forum full of generated garbage becomes less useful to its audience. Spam filters, moderation, and deliberate curation are responses to that problem, even if they don't solve it completely.

Some platforms will handle it badly. Some models will get worse at particular things. I don't need every company to make good decisions for this argument to hold. The ones that do have an advantage, and the rest have a reason to catch up.

Model collapse is a real failure mode. Generated slop can also make the web worse, even while the best models keep improving. Those problems deserve attention. But I haven't seen the general collapse people were hoping for, and I don't expect it to arrive just because synthetic data becomes more common.

Research can show us how a process fails. Even good research doesn't establish that we'll keep following that process indefinitely. People keep making things, checking things, rejecting things, and learning from what happens. That includes the people building the models.

I expect the successful models, much like successful people, to keep learning from fresh contact with the world. Things will change, and we'll have to deal with that.

I wouldn't count on model collapse to save us the effort.

Comments

Comment on this article
Names are not protected without a secret.
Identifies your name and lets you delete comments.
or drop anywhere in this form JPEG, PNG or static WebP, up to 1 MB.
Loading security question...
No comments yet. Be the first to comment!