How Does AI Create Images?

Jon Ippolito

Professor of New Media
Director, Digital Curation graduate program
The University of Maine
LearnWithAI.org

🐦/🧵 @jonippolito 🐘 @jonippolito@digipres.club 🦋 @jonippolito.bsky.social 🔗 https://www.linkedin.com/in/jonippolito

Today's outline

Creativity through averaging

Connect Random Things exercise

swift_taylor_eras_tour big_bang_diagram

Learning about LLMs through image generators

Leonardo.ai logo
Showcasing output variability
Demonstrating AI's probabilistic character
Illustrating AI's dependence on model
Calling attention to AI fabrications
Recognizing beauty stereotypes
The benefit of seeing "averaged" imagery
The challenge of visualizing "averaged" language

Generative AI creates by averaging

ang_berkeley_rect_37_xvga
Angelo Ippolito, Berkeley #37, collage
legrady_george_ai_collages_from_photos
George Legrady, AI-generated collages

What's behind the magic?

Harry Potter Magic Patronum Spell Xvga

How image generators depend on word associations

How image generators create hybrid images

Harry Potter Hagrid Balenciaga Meme Xvga
generative_ai_0_text_background_1080
generative_ai_0_text_background_1080
generative_ai_1_text_original_1080
generative_ai_1_text_original_1080
generative_ai_2_text_trained_1080
generative_ai_2_text_trained_1080
generative_ai_3_image_original1080
generative_ai_3_image_original1080

Associating images with words

Uploading images
Alt Tag In Wordpress Star Wars
Viewing images
Alt Tag In Html Star Wars
generative_ai_4_image_trained_1080
generative_ai_4_image_trained_1080
generative_ai_5_latent_average_1080
generative_ai_5_latent_average_1080
m

Generating hybrid images

If generative AI creates by "averaging," can we just average the relevant images in our dataset to create new hybrids?

martian_daisy_10
martian_daisy_10
martian_daisy_20
martian_daisy_20

AI can't generate hybrid images just by averaging pixels ☹️

A better approach to generating hybrid images

The solution turns out to be related to the problem of removing noise from a photo 🤔

The reason is that the best de-noising algorithms work by segmenting the image into shapes.

denoise_10
denoise_10
denoise_20
denoise_20
denoise_30
denoise_30
denoise_40
denoise_40
denoise_50
denoise_50
denoise_60
denoise_60
denoise_70
denoise_70

Finding noise means finding shapes

A de-noising algorithm can find shapes in noise 💪

💡 So instead of averaging the original photos, we could

  1. Add noise to both images
  2. Average the noisy results
  3. De-noise to find new shapes
martian_daisy_30
martian_daisy_30
martian_daisy_40
martian_daisy_40
martian_daisy_50
martian_daisy_50

The transporter analogy

Star Trek Voyager, Tuvix

An infamous Star Trek episode about a transporter malfunction offers an analogy:

Creating a hybrid of two people is easier while their molecules are scrambled than in their original form.

tuvix_10
tuvix_10
tuvix_20
tuvix_20
tuvix_30
tuvix_30
tuvix_40
tuvix_40

Results of averaging noisy images

Stv Tuvix Smiling
Harry Potter Balenciaga Meme Xvga

Why should we care about how images are generated?

Farid Hany Real V Ai Dog Noise Closeup🔍 Because lurking in the noise are telltale signs of AI

Farid Hany Real V Ai Dog Noise A
Farid Hany Real V Ai Dog Noise A

Real v. AI dog images by Hany Rarid

Farid Hany Real V Ai Dog Noise B
Farid Hany Real V Ai Dog Noise B

Real v. AI dog images by Hany Rarid: noise

Farid Hany Real V Ai Dog Noise C
Farid Hany Real V Ai Dog Noise C

Real v. AI dog images by Hany Rarid: Fourier transform of noise

Back to title screen
← →

/