Diffusion models are becoming sophisticated enough that they can reproduce an image even when they don’t have access to the original.
In a series of ‘what if’ scenarios, researchers associated with MIT’s Computer Science & Artificial Intelligence Laboratory (CSAIL) swapped out different training datasets to test the impact on image outputs when original image data was completely removed.
It turns out that, at sufficient scale, nothing changed.
The researchers call the phenomenon “attribution decay”: The more data a diffusion model is trained on, and the larger it gets, the less individual inputs matter.
“If you take away a piece of data and the output of the model doesn’t change, then that piece of data didn’t affect the output,” Zheng Dai, lead author on the work, explained in an MIT blog post.
These findings could have significant ramifications when it comes to resolving growing concerns about intellectual property (IP) and copyright infringement.
Models can recreate images even if they’ve never ‘seen’ them
Modern generative diffusion models essentially replicate statistical patterns in large training datasets to create realistic reproductions. These powerful tools have achieved “remarkable results” in a wide array of applications, the researchers noted, notably image, video, and audio generation.
But they are increasingly under scrutiny by creatives, companies, and policymakers, who all want a way to assign responsibility for generated outputs. Models sit at the center of lawsuits, licensing deals, and proposed regulations around the world.
For instance, Stability AI (maker of Stable Diffusion) and Midjourney are embroiled in an ongoing class action lawsuit filed by several artists in federal court in California. The claimants argue that the popular image, video, and audio-creating models are scraping billions of their copyrighted images without their consent.
Getty Images also brought claims against Stability AI, but they were struck down by the High Court of Justice Business and Property Courts of England and Wales, although Getty did partly win trademark claims because some AI-generated images closely resembled its work.
Attributability, the MIT CSAIL researchers noted, would increase understanding of “machine unlearning,” data poisoning, model interoperability, fairness, and privacy, while also addressing ethical, legal, financial, and regulatory issues.
“Developing a method to attribute generated outputs to influential training data would greatly advance our understanding of and ability to regulate these models,” the researchers wrote.
In their experiments, they used ablation, which is essentially testing what happens when certain elements are removed by looking at what a model might have produced if it had never “seen” a particular image.
Typically, ablation is difficult because models need to be retrained after data is pulled out. But the MIT CSAIL researchers applied the method to a “diffusion ensemble” architecture of many different components trained on different pieces of data. These components could be swapped out to determine how much of an impact, if any, each one had.
“Our analysis is based on observing changes in model behavior, or lack thereof, upon omitting a part of the training set,” the researchers explained.
To do so, they trained 24 ensembles on datasets containing anywhere from 256 to 160,000-plus images. These were pulled from seven publicly accessible image datasets, including ArtBench (artwork), CIFAR-10 (generic colored images), Fashion-MNIST (clothing and accessories), CelebA (celebrity faces), and MetFaces (human faces).
In one example, they presented an image of a famous oil painting generated by a model trained on public domain artwork from 744 artists. It was shown side-by-side with hundreds of seemingly identical images that the model had generated, even when specific artists had been removed from training data.
The original was re-imagined in every possible variation, and the researchers quantified attributability by measuring the largest change they could induce by omitting training data. The radius became smaller as datasets became bigger, holding true across different measurements including pixel-by-pixel or semantic meaning.
In other words, single artworks by specific artists, or photographs of certain people, could be entirely removed from datasets, and the model could still reproduce that image or style. Essentially, tangible connections are lost, and linking to specific data points responsible for generated samples is “practically impossible,” or can even vanish, the researchers explained.
Their method is novel, they said, because prior work has focused on removing large swathes of data rather than targeting smaller pieces, what they called “leave-one-out style attribution.”
The impact on attributability
Because the experiment shows that, as Dai put it, it “doesn’t make much sense” to attribute a given output to a given piece of data, creatives and others may not be able to provide an audit trail tracing back to their original work.
Co-author David Gifford, an MIT professor and CSAIL principal investigator, said the findings have a direct bearing on legal questions around whether model outputs are actually derivative works.
“One way to think about this is that these models are creative,” he said. “They are not simply copying what they are fed, but creating brand new outputs.”
So if outputs can’t be correlated to individual pieces of training data, questions can be raised around fair use and whether, in fact, model-generated outputs are themselves copyrightable as “novel works,” Gifford said.
It could also shift the conversation about how original creators are compensated when what comes out of a model seems a direct recreation of their work, but can’t be traced back to anything on the internet.
Ultimately, producing outputs that are guaranteed to be unattributable is an “obligation for the industry, rather than a loophole,” he said. AI builders “need to revise their models to take advantage of the advances in this work, so they can show they’re not creating derivatives of individual people or items.”