This looks like its just a quick front-end to the ControlNet scribble model (perhaps with a customized underlying model instead of base Stable Diffusion), with canned settings, presumable a canned negative prompt, and maybe some common stuff added beyond the user input to the positive prompt. Which is not to be dismissive, its a very nice demo of what SD+ControlNet scribble can do.
But for people who like it, the ControlNet scribble model (and the other ControlNet models, depth-map based, pose control, edge detection, etc.) [0] are supported in the ControlNet extension [1] to the A1111 Stable Diffusion Web UI [2], and probably similar extensions for other popular stable diffusion UIs. Should work in any current browser, and at least the A1111 UI, with ControlNet models, works on machines with as little as 4GB VRAM.
[1] https://github.com/Mikubill/sd-webui-controlnet (EDIT: even if you aren’t using the A1111 WebUI, this repo has a nice set of examples of what each of the ControlNet models does, so it may be worth checking out.)
I recently added support for it on ArtBot[1] and it supports additional options like models, steps, control type, etc, as well drawing on a canvas[2] and generating input from that. It’s pretty wild!
Free and open source [3].
Bonus: it works on Firefox (unless you’re using private mode — because Firefox doesn’t make IndexedDb available to web apps in private mode and things end up breaking)
You probably don't need canned positive prompt. ControlNet has a "guess mode" and in reality, basically just send the control to positive part with empty prompt while not sending control to negative part with empty prompt.
Edit: nvm, this particular demo does require you to type in positive prompt.
Thanks for your kind words and feedback. I love seeing all these links to your scribbles.
I'm an engineer at Replicate, which is a place to run ML models in the cloud. [0] We built Scribble Diffusion as an open-source app [1] to demonstrate how to use Replicate.
This is all built on ControlNet [2], a brilliant technique by Lvmin Zhang and Maneesh Agrawala [3] for conditioning diffusion models on additional sources, in this case human scribbles. It also allows for controlling Stable Diffusion using other inputs like pose estimation, edge detection, and depth maps.
ControlNet has only existed for three weeks, but people are already doing all kinds of cool stuff with it, like an app [4] that lets you pose a stick figure and generate DreamBooth images that match the pose. There are already a bunch of models [5] on Replicate that build on it.
I see a few bits of feedback here about issues with the Scribble Diffusion UI, and I'm tracking them on the GitHub repo. If you want to help out, please feel free to open an issue or pull request.
What a simple, but excellent concept! I kind of expected it to crash after being posted to HN, but surprisingly it is still going strong.
What does it cost to host something like that? What does it cost to generate each scribble image?
If you haven’t played with Stable Diffusion based stuff before, note that as well as telling it what (in terms of subject) you want in the prompt, you can also tell it style things. Compare:
Sorry about the trouble. The Firefox incompatibility is the result of a bug in the underlying npm package we're using to render the drawing tool and canvas.
Maybe put a banner notifying users about this? That was a pretty bad experience while I struggled to make it work & thought the author/you had shipped something totally broken lol.
I think he is referring to “fork the repo and fix the bug yourself” as the solution, but given the existence of far more featureful publicly available Web UIs for Stable Diffusion + ControlNet, if you just want to get something that works on Firefox, you don’t need to expend that much work if you are wiling to host it yourself rather than just using something someone already has up on the web.
Will be nice to be able to upload a picture/sketch instead of having to sketch something out in the scratchpad. I tend to sketch block diagrams from time to time and hate having to use some tool to draw/drag/align things.
Very cool idea though!
If you already have your own images, you can use the Replicate model directly: https://replicate.com/jagilley/controlnet-scribble -- you can upload your image using the Replicate web UI or do it programmatically using the model's HTTP API.
I run a custom Stable Diffusion bot for a small community that has generated many tens of thousands of images. The community wanted to know what it was costing me, so I dug into it a bit and have a fairly "literal" answer to that question (generally). With the RTX3060 I'm running it on, using a kill-a-wat, I very roughly calculated that generating a 512x640 image consumed ~ +170 watts for about 6 seconds (on top of the baseline power consumption of the PC when idle). This comes out to a little over 1,000 Watt-seconds or 0.000295 kWh per image. I'll leave it as an exercise for the reader to look up their current cost per kWh and work out what that comes out to. This is extremely, extremely rough, but helped me wrap my head around the amount of energy we were using to generate all these images and roughly how much of my power bill I had to thank my friends for, haha.
Of course running on rented/hosted GPU's, it's a simpler, but much more expensive story — basically however much you're paying for GPU instances to run Stable Diffusion divided by how many images you generate. :)
Uploading my own images was one of the reasons I created my own version of this. It works great with kids pictures! This is so fun to see my kids smile on their face every time they see the AI version of their drawings!
This is great. I presume it's made with Replicate? It looks like it's being updated, as I was getting endless loading indicators a moment ago.
Now trying to decide if this is healthy for kids.... who may lose motivation to ever pursue detailed art and drawing because "AI can do it for me based on rough sketch". But at same time, it may motivate them to make more rough drawings with interesting ideas.
Yes, it uses Replicate indeed (scribble-diffusion)
What I find is that kids (at least mine) are motivated to create even more drawings on their own and then see what it looks like with AI, which gives them more ideas, so they create even more drawings. So, so far, I'd say it has a positive impact. But I'm also pretty curious how this may end up impact on the long term.
Good to hear. Hopefully kids learn the value of original and thoughtful concepts that inform not only AI renderings, but the quality of whatever creative work comes after raw ideas, such as script writing.
I can see storyboarding for film will make use of this. Although having the characters and background remain consistent from scene to scene is something I don't know how can be done, if every submission is a roll of dice.
This is cool. At first I was skeptical that the sketch had nothing to do with anything and it was just feeding the prompt into an AI. But using the example "sun behind mountains"[0] it seemed to work alright. What really blew my mind was when I paired the sketch with a prompt that made no sense at all[1]. Somehow the AI cobbled together a picture that used the prompt, but also has all the same geometry of the sketch.
I use that prompt because it's something my preschooler draws on a daily basis with no qualms at all, but can often be hard for generative AI to imagine.
So it's getting close to the imaginative abilities of a 3 year old! Halfway there.
I firefox there's something wonky going on with the scribble area; the bottom of the image doesn't seem to show up until after you've drawn a lot more in the top of the image.
That use case is very LLM unfriendly, because you want very specific and precise structure of output, but the LLM knows lots of stuff that is absolutely not what you want at all.
If you expanded the prompt to describe the squirrel in redundantly verbose detail it would probably pick it up just fine. The more synonymous descriptors you can work in, and the more naturally you can phrase it, the better it will work.
But IMHO the interesting thing about controlnet is being able to use pre-rendered production art as a basis and allowing SD to respect the original proportions/model/etc. For rough sketches I prefer img2img without controlnet as it gives the algorithm leeway to fix, reinterpret, or "be inspired by" my input image without being too attached to it (since it's full of imperfections anyway).
Yeah, the results I got out of this immediately told me it was stable diffusion.
I have no idea how that (have to spend hours just getting a prompt that looks semi decent) can be considered state of the art when something like midjourney (any prompt will produce something great) exists.
I would love a wire framing tool that converts crappy sketches (ideally on an iPad with Apple Pencil) into low and high fidelity app mockups, and possibly other things like mind maps. Does his exist anywhere?
Can you place more than two objects in the sketch? I'm trying to sketch a house with a tree side-by-side, but it always drawing trees in front of the house.
I just tried to draw exactly the same thing. I'm no artist, so I thought a house, a tree, the sun and some clouds were my best bet (like I drew when I was 5; now 55). :-)
So, technically, and beyond that, theoretically, and even anatomically.
In the brain, visual processing, left hemisphere was found to contain details; a right hemisphere to contain structural relations. So a whole is composed of elements and relative positions.
In Convolutional Neural Networks, "near, direct" layers contain analytic detail and "far, abstract" layers contain synthetic shapes.
So, implementation-wise, you can take e.g. descriptions as abstracts and a "pre-acquired" memory of details as «graphic».
Edit:
About the "combination", well that the whole purpose of this new technology proposal,
"ControlNet"
- i.e., formerly you may have had some "transformer" from input to output, and now "conditional controls" are added (through a "zero-convolution" technique) - see Adding Conditional Control to Text-to-Image Diffusion Models - https://arxiv.org/abs/2302.05543
I think it is biased to the text and not the scribble. I scribbled a car and tried one with a the text car and bar. The first one resulted in a SUV and second resulted in home/Kitchen Bar. I don't even think the scribble matters.
You went to the trouble of registering a domain but you didn't go to the trouble of even testing on Firefox? Even if you didn't have the time to fix it, then a message acknowledging that it's broken on some browsers is a nice gesture.
But for people who like it, the ControlNet scribble model (and the other ControlNet models, depth-map based, pose control, edge detection, etc.) [0] are supported in the ControlNet extension [1] to the A1111 Stable Diffusion Web UI [2], and probably similar extensions for other popular stable diffusion UIs. Should work in any current browser, and at least the A1111 UI, with ControlNet models, works on machines with as little as 4GB VRAM.
[0] home repo: https://huggingface.co/lllyasviel/ControlNet but for WebUI you probably want the ones linked from the readme of the WebUI ControlNet extensions [1]
[1] https://github.com/Mikubill/sd-webui-controlnet (EDIT: even if you aren’t using the A1111 WebUI, this repo has a nice set of examples of what each of the ControlNet models does, so it may be worth checking out.)
[2] https://github.com/AUTOMATIC1111/stable-diffusion-webui