What do you think about the implications of image generating AIs to game development (and the world)?

[vozka]

I assume all of you have at least heard about it already or seen some pictures. Neural networks that generate decent looking images from a text prompt and do some other things (like modifying existing images) are becoming mainstream. They cannot generate photorealistic hi-res images yet, but the development pace is incredibly fast, so they will probably get reasonably close within a few years. 

The big thing that happened recently is that Stability AI released their implementation of such neural network, Stable Diffusion, for free as open-source. The quality is not the best, but it's good enough and all the other ones are behind a paywall, waitlist or just a closed source service and not good enough. You need a GPU (preferably Nvidia) with at least 4 GB of RAM to run it, so nothing special. 

The potential implications to society are big (and not necessarily positive), but honestly I don't really want to start the discussion because the internet is already full of it and so many people are surprisingly emotional about it. Go to Twitter if you want to see (you don't).

I'm more interested in what you think about using it in game development. I know that in some commercial studios it's already being used by concept artists, because it's firstly a reasonably efficient inspiration machine and secondly the images it produces are not licensed (and it doesn't matter that much in pre-production anyway), so if you manage to create something good, you can just paint over it to finetune and use it. But it can do other things.

It can do inpainting, which can be used to remove seams or blemishes in phototextures. 

It can also generate seamless textures (there's a switch that causes every generated image to be periodic). There are no examples of people using them in games or renders yet because that feature is less than a week old. 

Personally I tried using it to add dirt and imperfections to a render that was too clean, and that was a success as well. Adding dirt to a render is not exactly useful in most of games, but I bet it could be adapted to add some grime to game textures as well. 

Any ideas? Have you tried it already?

[chakkman]

I can't imagine this getting very popular in professional game development. Maybe for very low budget projects. For everyone else, it will surely not be custom enough. Sounds very generic. I know, generic world creation is a thing, but, not so much when it comes to artwork.

[AluminumHaste]

Yeah, it's not high quality enough to be used for asset generation, but like all things, it will be a tool in an artists toolbelt in the future.

[AluminumHaste]

Also:
 

Quote

For now, Stability.Ai recommends that you have a GPU with at least 6.9GB (Nice) of video RAM. Unfortunately, only Nvidia GPUs are currently supported. [But] if you own a powerful PC, you can take all the time you’d like to fine-tune your algorithmic art and come up with something truly impressive.

 

[jaxa]

Actually, Nvidia has had image models for generating photorealistic faces for years now. They have gotten better over the years.

https://developer.nvidia.com/blog/generating-photorealistic-fake-celebrities-with-artificial-intelligence/

Applying 2D text-to-image algorithms to modern games seems unlikely, with the exception of making lots of textures and maybe 2D portraits for UI/character creation, but there are many other algorithms being worked on. Maybe a similar approach could be used to replace procedural generation techniques. Like making a cave/dungeon in Skyrim, or that thing Tels was working on a decade ago. Rather than making a raster image, it could make geometry, place textures, and design a whole city.

Bring on the negative societal implications of bots invading art, I'm all for it. But one thing to watch out for is the copyright question. These image models can be trained on a superset of copyrighted images or a smaller focused subset (to mimic an artist's style) and produce images that could lead to novel legal questions and expensive copyright lawsuits. This is not a problem for people making memes and shitposting online, but it could be a massive problem for game developers, big or indie. Save a few bucks on art, get sued into oblivion. Maybe we'll see Business Software Alliance style shakedowns of game developers? "Where'd you get these sprites, EH?"

28 minutes ago, AluminumHaste said:

Yeah, it's not high quality enough to be used for asset generation, but like all things, it will be a tool in an artists toolbelt in the future.

Sounds like vozka has made some textures with it? These can be scaled up from small 512x512 sizes to higher resolution with a separate upscaling algorithm, that's what people have been doing to make stuff presentable along with touchups in Photoshop/GIMP. Whether the results are any good is another story, maybe vozka should post their results.

Edited September 8, 2022 by jaxa

[vozka]

27 minutes ago, chakkman said:

I can't imagine this getting very popular in professional game development. Maybe for very low budget projects. For everyone else, it will surely not be custom enough. Sounds very generic. I know, generic world creation is a thing, but, not so much when it comes to artwork.

I do not have permission to say names, but the concept artists I mentioned work for a mid-sized company and they use Midjourney for scifi themed concept art. The results surprisingly are not generic looking at all if you learn to use it, but it's true that the "text to image" workflow does not give you enough control to straight up generate assets (and the quality is not good enough either). Midjourney is a paid proprietary service, but they specialize in their results being slightly less photorealistic but more creative and artistic.

 

But the important thing is that the "image to image" workflow exists.

Normally with text to image you describe what you want, the the neural net starts with noise and gradually rearranges it into something that resembles the description. With "image to image" you start with some actual image that you feed to the neural net instead, and then have it transformed to something that fits the description. You can change how far the program is allowed to deviate from the base image. This gives you a lot more control and it could potentially be used to generate textures for example. 

Some guy just posted an example on Reddit. He drew a rough scheme of a "scifi wall" using basic shapes, overlayed a random scratched metal texture and had SD generate several versions of a scifi wall from that. Then he edited his favorite result a bit and fed it back into SD, generating the final version. This is how it looked: 

 

I used the same workflow when I was adding grime to a final render (so not straight to a texture) and it worked well too. 

[vozka]

37 minutes ago, AluminumHaste said:

Also:
 

 

That's the beauty of it being open-source. An unofficial AMD port already exists (although I have no idea how well it works) and people did several optimizations that reduced the RAM necessary to run it down to about a half (with almost the same computation speed, less than half with reduced speed). 

 

14 minutes ago, jaxa said:

Actually, Nvidia has had image models for generating photorealistic faces for years now. They have gotten better over the years.

https://developer.nvidia.com/blog/generating-photorealistic-fake-celebrities-with-artificial-intelligence/

Applying 2D text-to-image algorithms to modern games seems unlikely, with the exception of making lots of textures and maybe 2D portraits for UI, but there are many other algorithms being worked on. Maybe a similar approach could be used to replace procedural generation techniques. Like making a cave/dungeon in Skyrim, or that thing Tels was working on a decade ago. Rather than making a raster image, it could make geometry, place textures, and design a whole city.

Bring on the negative societal implications of bots invading art, I'm all for it. But one thing to watch out for is the copyright question. These image models can be trained on a superset of copyrighted images or a smaller focused subset (to mimic an artist's style) and produce images that could lead to novel legal questions and expensive copyright lawsuits. This is not a problem for people making memes and shitposting online, but it could be a massive problem for game developers, big or indie.

Sounds like vozka has made some textures with it? These can be scaled up from small 512x512 sizes to higher resolution with a separate upscaling algorithm, that's what people have been doing to make stuff presentable along with touchups in Photoshop/GIMP. Whether the results are any good is another story, maybe vozka should post their results.

My results are just a proof of concept and not very useful for game assets, but I was pretty happy with them. I made a very simple 3D render with unnaturally clean PBR materials, and then added some dust and dirt on them using Stable Diffusion.

I had to overlay some black noise texture over the clean materials because the AI needs something dirt-like to work with, it will not generate dirt on something that is completely clean unless you let it deviate from the base imagea lot, and in that case it tends to change the shape of items and distort things. I also had to render the bench and garbage bin separately (but keep their shadows in), the AI tended to change them too much.

The images are 832 x 832 px in size because that's the maximum that currently fits into my 6 GB of VRAM (I'm not using the latest optimizations), but when doing something like generating noise (which doesn't need much continuity), you can easily just split a large render into smaller tiles and do it tile by tile. 

 

Before AI dirt:

 

After AI dirt:

[Xolvix]

I'm OK with it when used appropriately. The current AI image generators are good starting points for various things, but there are often imperfections that one would still expect an artist to fix before using it. You don't just use Stable Diffusion to make what you want and you're done, especially if there are human characters in the image (fine details like faces, hands and such tend to appear slightly deformed and it's very noticeable). I'm also not a huge fan of using it to upscale textures for older games because the end result can shift the intent of the artist with some of the finer details, but again, that's only if the replacement is left untouched. If it's then modified by the artist to more closely match the original then that's fine I guess.

A part of me is a bit uncomfortable with such AI because I tend to feel that art is the domain of the human rather than the machine, and I've heard the idea that AI will be able to basically replace all humans in the art-making process eventually. Maybe that will still happen, but if am aware that a bot made some artwork instead of a human then there's something lost. Art is one of the few things that still has some soul attached to it (excluding mass-market stuff), so I don't know if I'm being old-fashioned but I'd feel better if I knew at least the base of the work was still mostly under the creation of a human.

[vozka]

10 hours ago, Xolvix said:

A part of me is a bit uncomfortable with such AI because I tend to feel that art is the domain of the human rather than the machine, and I've heard the idea that AI will be able to basically replace all humans in the art-making process eventually. Maybe that will still happen, but if am aware that a bot made some artwork instead of a human then there's something lost. Art is one of the few things that still has some soul attached to it (excluding mass-market stuff), so I don't know if I'm being old-fashioned but I'd feel better if I knew at least the base of the work was still mostly under the creation of a human.

This will undoubtedly happen with large parts of digital art. But there's a ton of art in the physical world that is not in danger. Even oil paintings - you can print digital art on canvas, but it won't be able to mimic some specific painting techniques that create a physical structure on the canvas, or specific types of pigment. Or various types of combined media. Then there are other even more physical media like sculptures, that's an infinite world.

Honestly from the various art school graduates/teachers that I know none of them are threatened by this because none of them ever did normal digital art like anything an AI can produce.

[OrbWeaver]

I always watch these panics about AI bots "invading" this or that with an attitude of detached hilarity. Given that humans are nothing more than a biological machine, it seems self-evident to me that given enough time and technological advancement, electronic machines will be able to do everything that humans can.

When humans create art, they do so by a process of generating ideas based on existing art styles they have seen or been taught, along with various sources of "inspiration" from their everyday life or past experiences. There is no magic, there is no "soul". It's just recombining various ideas in their heads in a way which matches culturally-specified criteria for what is aesthetically pleasing and what isn't. This is exactly what the AI bots are doing.

Sure, if you're a junior graphic designer whose job is to create customer-specified images on demand, you'd probably be threatened by these developments, just as people who made candles were threatened by the invention of electric lighting. But that's progress for you.

[chakkman]

2 hours ago, OrbWeaver said:

I always watch these panics about AI bots "invading" this or that with an attitude of detached hilarity. Given that humans are nothing more than a biological machine, it seems self-evident to me that given enough time and technological advancement, electronic machines will be able to do everything that humans can.

No way. Machines work completely different to humans. Machines and computers can do stuff which they are more suited to do, and more capable, like stupidly calculating hundreds of things at the same time. Machine stuff. They're not capable of, say, inventing a computer. Even with the most elaborate AI routine, they are only capable to do stuff which they are supposed to do, nothing more. And, they will never be. An algorithm or a program can't be complex enough, and "human like" enough to do what the human brain can.

Especially art is also about emotions, about memories, and subjective impressions. How would a machine be capable of that? It can only do what it's programmed to do, a fixed routine. Even machine learning is a fixed routine. There's nothing spontaneous, creative, or anything in that regard. It's simply doing what it's been programmed to do. A human being doesn't do what it's programmed to do. It inherited certain traits, but, a lot is based on experience and individuality. The brain is much, MUCH more than a computer or a machine.

Edited September 9, 2022 by chakkman

[jaxa]

@chakkman

The human brain is a "machine", and it's entirely possible that human-like intelligence can be created using brain-inspired (neuromorphic) hardware or even in a software simulation on conventional hardware.

Obviously, that's not what the current "machine learning" algorithms are doing. They are finding and applying patterns using large datasets and brute force. This can be very useful even if there is no intelligence or emotions involved and it fails hard on edge cases.

If many jobs (drivers, call centers, concept artists, etc.) can be eliminated by mere rock-dumb algorithms, well, that says a lot. There will still be plenty of artists, but they could end up being paid less and having their work and art styles routinely "stolen".

[chakkman]

1 hour ago, jaxa said:

@chakkman

The human brain is a "machine", and it's entirely possible that human-like intelligence can be created using brain-inspired (neuromorphic) hardware or even in a software simulation on conventional hardware.

No, it isn't. That's just utter nonsense. The human brain is neither fully understood, nor is there any degree of complexity which could map the whole brain in software. Whoever claims that is either a complete brainfart, or deliberately lying.

I know, people are in science fiction these days, but, that's what it is really. Science fiction.

Again, tell a machine to invent a computer (or something similar). Or to come up with Einstein's theory of relativity. 

I'm not sure why people think that would be possible from a computer. A computer can do things which you command him to do, and even do some kind of "learning" based on the parameters you put into it. But, expecting any kind of human behavior is just ridiculous, because a machine doesn't think like a human being, or any other living being.

A computer can forget things? Yes, maybe when you program it to forget things, based on a predefined pattern. And that's the thing. The pattern is predefined. It's not random. Not analog. It's digital. A computer can neither just forget things, nor can it have a sudden flash of insight. It just can act in pre defined routines, put into it by its programmer. Even if it allegedly acts human, like im machine "learning", which is nothing other than, again, pre defined behavior.

Edited September 9, 2022 by chakkman

[OrbWeaver]

2 hours ago, chakkman said:

No, it isn't. That's just utter nonsense. The human brain is neither fully understood, nor is there any degree of complexity which could map the whole brain in software.

This is the argument from ignorance (or lack of imagination). "We don't understand X, therefore X can never be understood, and it must be magic."

The brain is made up of neurons, which are well understood. They "fire" in response to a certain threshold being reached by a required number of input neurons. Yes, there are millions of them, and we don't understand the full implications of how they are linked together, but there's nothing magical going on. A sufficiently advanced machine with a sufficient amount of processing power would be able to simulate the behaviour of a neuron-based human brain. And unless you quite literally believe in a metaphysical "soul", this simulation would produce the equivalent of a human mind.

2 hours ago, chakkman said:

Again, tell a machine to invent a computer (or something similar). Or to come up with Einstein's theory of relativity. 

Nobody is claiming that a present-day computer can do any of this. But this proves nothing about what future computers will be able to do. Once upon a time, people would have said that no machine will ever be able to beat a human at chess, but now they can (easily). Only a few years ago people would have insisted that a machine can never interpret an English sentence and produce an original work of art, but here we are.

2 hours ago, chakkman said:

But, expecting any kind of human behavior is just ridiculous, because a machine doesn't think like a human being, or any other living being.

I'm not sure why you group all "living beings" together in this way. As far as we know, there is nothing living which can think like a human, although elephants and some cetaceans (e.g. orcas) may come close. Living beings encompass everything from humans to dung beetles and amoebas, and even our present-day computers are capable of more advanced "thinking" than many of these.

2 hours ago, chakkman said:

A computer can forget things? Yes, maybe when you program it to forget things, based on a predefined pattern. And that's the thing. The pattern is predefined. It's not random.

Computers have been capable of true randomness for decades. That's how secure encryption works. Most modern CPUs even have built-in instructions to generate random data from a hardware (i.e. analog) random generator.

Not that it really matters, since there is no current evidence that consciousness or human-like thinking depends on randomness. We have very little understanding of how human memory works, or why we forget things (and then subsequently remember them again). But there is no reason to declare up front that we will never understand this, or be able to simulate it in a machine.

2 hours ago, chakkman said:

A computer can neither just forget things, nor can it have a sudden flash of insight. It just can act in pre defined routines, put into it by its programmer.

And how do you know that we don't act in pre-defined routines, "programmed" by biology and/or culture, acting in a way which (purely by accident) gives rise to what we consider human behaviour? Just because you can't tell where your creativity and human behaviour comes from, doesn't mean that it is magic, unknowable, or coming from somewhere other than the mechanics of your brain.

[vozka]

2 hours ago, chakkman said:

No, it isn't. That's just utter nonsense. The human brain is neither fully understood, nor is there any degree of complexity which could map the whole brain in software. Whoever claims that is either a complete brainfart, or deliberately lying.

I know, people are in science fiction these days, but, that's what it is really. Science fiction.

Again, tell a machine to invent a computer (or something similar). Or to come up with Einstein's theory of relativity. 

I'm not sure why people think that would be possible from a computer. A computer can do things which you command him to do, and even do some kind of "learning" based on the parameters you put into it. But, expecting any kind of human behavior is just ridiculous, because a machine doesn't think like a human being, or any other living being.

A computer can forget things? Yes, maybe when you program it to forget things, based on a predefined pattern. And that's the thing. The pattern is predefined. It's not random. Not analog. It's digital. A computer can neither just forget things, nor can it have a sudden flash of insight. It just can act in pre defined routines, put into it by its programmer. Even if it allegedly acts human, like im machine "learning", which is nothing other than, again, pre defined behavior.

You make some good points, but I personally think it's similarly naive to think that we cannot eventually get to a point, where an artificial intelligence produces results so similar to a human in some areas, that the fact that it functions differently becomes irrelevant. 

In fact I'd say that current neural networks are a big step forward both in terms of the results they produce and in terms of how similarly they work to a brain. We know that artificial "neurons" are a distant abstraction of how a brain works, but a neural net is not just a normal mathematical algorithm. Yes, it's "just" matrix multiplications, but at the same time the complexity of the computation is so far beyond what any human could analyze or design that it's a blackbox. That in particular is similar to a real brain. 

The output of a neural network is deterministic just like with a "normal" algorithm - two identical inputs give you two identical outputs. With the case of image generation algorithms part of the input is noise (in the shape of the final image), which dictates how will the final image look. Change the noise very slightly and the output can be completely different. Since we want the results to be reproducible, we use a pseudorandom noise generator initialized by a seed number. Same seed will produce the same noise pattern and therefore the same resulting image.

What if we instead use a noise generator that is truly stochastic? We are able to do that (using specialized hardware). The algorithm itself would still be deterministic, but it would be unpredictable because a part of the input would be unpredictable. Would that be "organic" enough?

And what if we stop computing the neural network on a GPU and instead use a chip that does analog matrix multiplication (already exists, although not powerful enough for this particular usecase, yet), which brings a stochastic element into the computation itself? The computation itself is no longer deterministic. You know what will roughly happen, but cannot predict the exact result even if you know the input. 

Would that be organic enough? It is certainly a very different thing now from how any normal computations work. 

[vozka]

3 hours ago, chakkman said:

A computer can neither just forget things, nor can it have a sudden flash of insight.

Neural networks can and do forget things during training, and their memory is very imprecise. And we're not entirely sure why that is, as far as I know. But it's one of the basic principles without which the whole concept wouldn't work. 

It is true that a neural network cannot have a sudden flash of insight, or more precisely it cannot realize that it just had a sudden flash of insight. Artificial general intelligence does not yet exist and is probably a couple decades away, and specialized AI is not self-aware. It can certainly act as a flash of insight for a human that uses it to create something unexpected though.  

[Xolvix]

If you really want to distill things down then sure, we are biological machines and AI's should be able to replicate whatever we can do, given time. Hell you could probably combine the various text and image generating AIs out there to create the assets for a map at this point, including the story. With enough time, probably the maps themselves could be created by AI.

But if I know that I'm playing a map created entirely by an AI with minimal to no direct human influence... eh. It takes something away from it. There's a reason people enjoy practical effects like actual jets doing their shit in a movie like Top Gun: Maverick as opposed to a complete CGI fest. Not quite the same analogue I know but still. Otherwise it gets to the point where all content is synthetic and that feels incredibly stale.

As a side note, we have enough AI tech to create fairly realistic fake articles that can be used to slander and manipulate to a degree not seen before. The issue of art being wholly created by AI is one thing, but the damage of this tech to manipulate society to a negative degree is much, much worse. Such is the progress of technology I guess. Maybe that's my bias to be cautious about it.

Edited September 9, 2022 by Xolvix

[demagogue]

I've been using Stable Diffusion a lot recently.

Something you have to note is the current way to use it, you should expect to make 20~30 versions. One of them is going to be awesome. The other thing is that you often have to re-do wonky parts, which most of these systems are starting to do. I mean maybe 60% of it will be right on, so you keep that and fix up the wonky bits until they fit. So you're not getting perfection on the first run. You're iterating the same image in bits in pieces, each block taking 20 iterations or so.

Someone was saying it before .... hands, feet, legs, eyes, will look a little wonky. So you just drag a box over it and start iterating, and eventually it will get it right. By the end of that you're going to start getting close to perfection. We have the tech for that already now. It may take a little time, but you can get perfection with what we have now if you block iterate it enough.

And the final thing is, look at what Dall-E was making just 1~2 years ago. It was trash, and what we have now is lightyears already beyond it. In another 3~5 years I think it'll meet the hype, and every thing churning out of it will be photo real or whatever you're looking for, and then some. I don't think it'll take that long to get there just looking at the differential slope of the progress we're seeing right where we are now.

I think of it like chess engines. There was that period where people were wondering if it'd beat humans, and then it did, and then it just kept going without looking backwards until at this point the entire chess world looks like it's going to run into a crisis of conscience because the game is out of human hands now. There's every reason to expect imaging software is going to do the same thing in short order.

For gaming, e.g., I'm making a card game, and this is helping me make the background and card images. Again it's taking like 20~30 iterations, and then I have to fix a few things by hand, but I can churn out a half dozen really good looking cards in a day where I wouldn't have been able to make a single one before. But like I said, I expect in 5 years I won't even need many iterations or the hand-fixing... I think the fixing will be automated well enough.

Oh another thing is, because it's still diffusion over a data set, the big limits are users not understanding the ontology embedded in the set. If you figure out good search terms, you can get what you want much better. Another thing I think will improve in the future isn't only the tech, but the search terms and other UI elements will work much better to make the system give you quality work that you want, by improving the data set, but also you can have a second-order of AI that parses text inputs into a form that works with the data set. I think that's part of the next generation of AI that's going to make it start much better delivering to us on a human level. I mean I think the tech is already there now in principle. Now it's a matter of making the tech work for humans, and that'll be the next big revolution coming soon.

[jaxa]

This video popped up on my YouTube homepage:

[Oktokolo]

As a non-artist, i definitely apreciate that AI will give people like me access to individualized non-rights-encumbered art/assets - be it for personal enjoyment or use in free or commercial products.

This should also enable indie game devs to finally abandon "pixel art" for good...

[jaxa]

16 minutes ago, Oktokolo said:

As a non-artist, i definitely apreciate that AI will give people like me access to individualized non-rights-encumbered art/assets - be it for personal enjoyment or use in free or commercial products.

This should also enable indie game devs to finally abandon "pixel art" for good...

People are going to get sued over this stuff, because of course they would.

Pixel art won't be abandoned. It's an aesthetic that (some) people like, and bad artists don't make good pixel art. Also, AI will be used to make pixel art:

Edited September 13, 2022 by jaxa

[demagogue]

Y'all should visit the MidJourney discord channel if you haven't already. The other big thing, the FOSS one, is Stable Diffusion. That's what I've been using (via the hugging space demo, which doesn't use a credit system), and it's typically awesome once you know what you're doing. But I don't know where people are posting it in a single place.

It's coming fast and thick now. I'd say half of it is awesome, and the other near half is awesome but there's some little screwy bit you could fix up in a half hour. Then maybe 10% are duds. Of course the posts are going to be biased on the awesome end just because they've been filtered.

The other thing I've noticed is suddenly a ton of art on Twitter and FB is AI art, and again it's typically great. Yeah, art belongs to the computers now. You can be in denial about it, and it'll take the next year or two to really settle in. But I think the die has been cast now.

[Shadow]

MidJourney does some really mind-blowingly phenomenal creations, stuff that can be Escher-esque, Dali impressionistic, Thomas Kinkade landscapes, etc. However it's only free for making around 20 or so images and has a tiered payment system for more. The Discord channel does let subscribers post their own creations, and many of them are quite creative at getting the MidJourney engine to render some awesome stuff. Definitely worth the scrolling if you are like me and like to download high quality images for a revolving screen saver.

[jaxa]

Going open source (technically "source available") meant going against the corporate grain of "responsible AI", but it seems to have instantly earned Stable Diffusion most of the mindshare. Future versions of the model (and others) will become even better.

https://www.theverge.com/2022/9/15/23340673/ai-image-generation-stable-diffusion-explained-ethics-copyright-data

It will be good for AMD and Nvidia too. I was content to stick with the GTX 970 and APUs forever, now I'm eyeing high-VRAM GPUs with more interest.

1 hour ago, Shadow said:

Definitely worth the scrolling if you are like me and like to download high quality images for a revolving screen saver.

https://en.wikipedia.org/wiki/Electric_Sheep

Edited September 15, 2022 by jaxa

[Springheel]

6 hours ago, demagogue said:

 Yeah, art belongs to the computers now. You can be in denial about it, and it'll take the next year or two to really settle in. But I think the die has been cast now.

Yep, professional artists and graphic designers will be without a job within a few years.  Modellers and animators won't be far behind, I imagine.