I was excited about this when it was all the rage. It speaks to me as a programmer. Some years later, I stumbled upon this excellent article by an artist considering the problem from the art perspective: http://www.dinofarmgames.com/a-pixel-artist-renounces-pixel-...
There's a ton of graphics research that has the same basic issue, I feel. The premise behind a lot of it seems to be how to do impressive art without an artist, how to automate our way to compelling imagery.
I am guilty of this too, as an occasional graphics researcher and full time tool maker, I'm implicating myself here too. I am with you, the algorithms speak to me as a programmer, and they are really fun to think about and code.
But I suspect we're missing out on some opportunities to do amazing things with an artist in the loop, that we're not yet achieving what we could if we built tools that try to enhance creative people rather than replace them.
Adobe is pretty good about taking Siggraph papers and implementing them with more artist friendly controls than the paper, but I can't help but wonder what we might have if all the Siggraph papers on average aimed to be more controllable and less automatic in the first place.
In their comparison http://johanneskopf.de/publications/pixelart/supplementary/m... the Vector Magic results appeal the most to me precisely for this reason. For most images they look more like an artists rendition than an attempt to trace the image. Which is good.
>Of course, the algorithmic approach lets you upscale any number of preexisting games.
This is the point of these algorithms, for sure. Hand-crafted graphics will always beat algorithmic upscaling of lower quality hand crafted graphics, given the same amount of time to do each, but that isn't possible for most examples of games with pixel art.
Bizarrely, the pixelated stuff in the images you linked look way better. You couldn't use the depixelized whale or dolphin or whatever (our link) in a game, it looks awful. Some of the full comparison is even worse:
But you can use literally any of the pixel art'ed images from your link - they all look truly fantastic and production-ready! They're ready to ship now.
I remember back in like 2010, someone uploaded and ripped a YouTube video 1000 times and then edited the natural degradation of the compression. Went from perfectly watchable video to horrible static-y mess.
Theoretically, it should be possible to recompress the video back into the compressed file with no loss, provided the same codec is used, but it would be far more expensive computationally.
Like translating a phrase into a foreign language and back repeatedly, I inagine you’d end up with something fun and impressionistic. I’d like to see it too.
The best examples, those with more divergent results, are those with a lot of subordinate clauses, which get mixed-up and moved from different parts of the sentence. You can select complex sentences from the list below.
I always thought it was sad that it didn't get as much attention. I think it could also interesting to use in thumbnail generation based on regular photos, for example.
Going to ask in that reddit thread if the author is already aware of this algorithm. Might be able to take some cool new ideas from it if they don't! :)
Huh, that's actually quite interesting - but for an entirely different purpose.
My better half enjoys a bit of cross stitch. For a cheapo little gift I've played with outputting custom cross-stitch patterns based on pixellating images and reducing the number of colors to something more manageable, but it's often a little janky in GiMP and requires a bunch of manual tweaking - for anything remotely detailed it quickly becomes more hassle than it's worth.
This was my first thought as well. I've gotten decent at using photoshop to do this with generous use of cutouts and indexed color modes to custom forced palettes, but it's a very laborious process.
Playing with this, it's very effective for solid-color graphics as it quickly settles in on a nice limited palette. The line-work also does a reasonable job, limiting the manual fixing to areas where lines converge and so forth.
I'm working on writing up a post for /r/crossstitch comparing some different methods, so check over there in a while if you're interested.
Probably as a research project to see if it's worth using in HD conversions of old games, such as the Age of Empires remake that's dropping tomorrow. The algorithm seems to work well on standalone sprites, but kinda merges in a gooey way when two sprites are next to each other. The super mario example from the previous discussion shows this slightly: http://files.jjcm.org/upscaling.mp4
Very curious to see this as it progresses however.
I'm still impressed with the super mario example, I can't personally see anything negative about it and would enjoy playing the game with that "art style."
It's not perfect but seeing the alternatives it is a huge step in the right direction.
It looks like the transformation in being applied to the whole image at once in your example. If each sprite was vectorized individually, you wouldn't see the gooeyness. If you are a game developer, this change should be easy enough.
I don't think it was about videogames, more likely upscaling algorithm would have saved a TON of bandwidth for videos.
a a matter of fact some of the most recents algo available for video deinterlacing like nnedi3 are perfect for that, and of course also work for pixel art, and I think they blow away the example showed in their gallery - https://www.youtube.com/watch?v=0691zsXWbhA
I wonder if now, in the tensorflow era, a better automatic approach using generative nn could be devised to reach beyond 8x
waifu2x (http://waifu2x.udp.jp/) is another often-posted superresolution approach, focusing on upscaling low-res anime images instead of extremely tiny pixel art. It uses a totally different learning-based approach - however, their approach would not work with pixel art because it depends on having a set of high resolution ground truth images, and downsampling them to generate training data pairs. No such high-resolution examples exist for pixel art.
waifu2x seems to be based on Deep Convolutional Neural Networks, while the article looks like a pure handwritten algorithm and at least in this case the algorithm looks like a winner..
I am wondering whether in the future we may forget about writing algorithms altogether and instead rely on ANNs to do tasks that could have done better using hand written algorithms?
Image super-resolution is an ill-posed inverse problem: many possible high-resolution images would be reduced to the same low-resolution image. However, some of the possibilities are more realistic than others. Consider a grey pixel in a photograph from real life. It's more likely that it was downsampled from a 2x2 block of grey pixels than from a 2x2 black-and-white checkerboard pattern. We apply such knowledge by adding regularization to the problem, or using a prior distribution in a Bayesian formulation.
Deep learning is very good at memorizing those priors by looking at real world data. I think, in problems where the prior is important and complicated, data-driven approaches have a big advantage over hand-engineered approaches. That does not mean they will take over every kind of problem.
I think hq4x has a slight edge because it doesn't go all the way trying to make everything smooth, still preserving some pixelation. There's actually an analog to this in denoising: if you denoise too heavily you end up losing detail. Smart denoising manages to preserve edges, giving a better result. In this case, pixels are the same as noise, undesirable, but containing useful information.
Their method change too much on eyes and facial expressions in general. Also that broad sword is really blunt.
When comparing only the 4x results hq4x wins half the time. On the other half hq4x is a close second and generally looks alright. However in some cases the vectorized ones are pretty bad.
Boo and Bowser look great. Mario and Yoshi look terrible. The detailed pixel art and gradients just don't carry over. The shapes of the outlines do, however.
Scanlines IMO look the best for video games, because it also tries to simulate the medium they were originally presented in, not just the graphics data behind the images
A real-world example of this technique shows up in ColorDMD, an LCD to replace DMD displays used in pinball machines from the early 90's until just recently. Most of the displays were 128x32 with 4 (and later 16) brightness levels.
ColorDMD first had artists create color overlays for the images and animations, and then emulated the display with colored dots instead of the red/orange color from the original DMD.
A later firmware release included upscaling, with some impressive results. This thread on Pinside (a pinball discussion site) shows some examples from the game "The Simpsons Pinball Party". I've linked directly to a comment that shows an original rendered DMD frame along with colorized versions with and without upscaling.
Every time I see this paper I'm always disappointed that there doesn't seem to exist a usable (open source or otherwise) implementation of it anywhere.
My favourite all time game is Heroes of Might and Magic III. It was released in 1999, and the art is still good, but that maybe because I'm so used to it. Ubisoft did a great job, enhancing it for the HD release in 2015, there is video showing it: https://www.youtube.com/watch?v=AX_h3RIbSKk
Though, one lesson I've learned (as a game developer myself), was that enhancing may introduce more visual problems. On a port of a game I worked (Playstation1 -> PC) we hired an artist to enhance the eye textures, but once the team behind the original game saw this, they told us that since there was no EYE animation in the game, and having such high-fidelity texture (for it's time, back in 2000) made all the characters look like toys.. Now if we left (and we did) the eye textures blurry, then your own eyes can't focus, and accept this better (without eye animation). I'm not an artist (just a coder), but this taught me "less is more".
The biggest advantage of pixel art is that it takes up less resources to create a solid retro type game. Animations suddenly become a lot easier as it relies on the abstract nature of putting the imaginative efforts to the users.
By removing these artifacts, you essentially defeat that purpose. Animating vectors would take a lot more time to create something that seems professional, mostly because you have to be far more detail - analogous to moving from 2d to 3d. I personally don’t see the use cases for this in gaming.
Many type designers deal with this as part of their workflow if they work with traditional drawing tools like pen and ink. My understanding is that for classically designed fonts its often easier to redraw vector objects over the scan than to clean up what the automatic tools produce. However, for certain types of "distressed" fonts its a kind of necessity.
for both directions I would say a competent artist.
Clever algorithms get you some of the way, but you will still need someone with good sense to edit the result to get something usable, which is going to cost you in either time or money.
Downsizing isn't so bad now because of higher DPI displays (e.g. font hinting is obsolete now) but previously designers would need to make individual icons in a number of sizes removing elements as necessary.
It seems pretty trivial but I could be way wrong as this is in no way my area of expertise...but- pixelization filters in photoshop and other image editing softwares have been around for a very long time.
Even logically it is an easy leap as they're(the software/filters) most likely choosing the most common color in a given area and then converting that area to a square of that color..
Just pixelating an image doesn't really give high quality pixel art. No examples handy but I'm confident you'd see this if you tried to reverse the images in the post with the approach you outlined.
Could you take this a step further? For example, a human knows this thing isn't supposed to be lumpy. What would it take to make this look more like it was interpreted by a human?
Super resolution with deep neural networks has gotten some decent results that incorporate knowledge of what the image is. However, it's probably not worth it. Running this algorithm on a single image is much simpler than finding thousands of pixel art -> image pairs and training a superresolution network.
Filters have achcieved similar results, but not quite with the same technique!
Or, this algorithm wasn’t intended for real-time processing and creates an adjacency graph of similar pixels. Once it’s done that, it will try to fit/optimize splines over the regions. Emulator filters like hqx determine each pixel’s “upscaled shape” by the colours of its neighbours, usually indexed with a table.
I want to go through the research paper but am unable to load or download it. If anyone has managed to do the impossible please share a drive link. BTW which journal is this published in?
Gotta say, I really like those PhotoZoom 4 results, aside from some of the fringing artifacts. They seem to preserve a lot more of the detail, whereas the presented approach focuses more on eliminating noise.
It points to the problem with these upscaled versions: They aren't any good. In the end, the artist demonstrates how to get good results, draw it again: http://www.dinofarmgames.com/wp-content/uploads/2015/04/yosh...
Of course, the algorithmic approach lets you upscale any number of preexisting games. However, I have concluded that I like the original art better.