Game size is a perennial talking point and some games earn it and some games don’t.
There was a phase of games shipping with uncompressed audio in every language they support for no good reason that ballooned file size (Titanfall)
Some games have gotten around this problem by having higher resolution textures or additional languages being optional installs within whatever ecosystems DLC implementation is. I think this is the smart way to do it.
Examples of games shrinking I are hard to come by but special credit to IO interactive and Hitman 3.
The install size with all the content of the previous games included (If you own them) is less than the install size of the first game. That is impressive and commendable.
Games are weird and some people will look at the file size and think it means the game is better, just the way some people think games with more hours are better. The psychology of people who play video games is weird.
Duplication of assets on disk was a very weird byproduct of HDD file access conventions, but we shouldn’t need that anymore. There is still a shrinking, but valid crossover window between devs being able to assume all storage is solid state. I think things will get better but the debate of file size vs average/expects storage capacity vs Network speed will never end but there is certainly work to be done.
Blizzard us a bad example. Their games take incredible amounts of time to start. I think they dont play their own games and nobody gives a shit.
The launcher is bugged and downloads either empty or 137mb patches (maybe bad CRC at some server? Different CRCs at some?). Every time.
Starcraft remastered was bugged at some point and couldnt be played for a week.
If you statt Heroes of the storm, it takes few minutes to download some 600 byte ghost patch, then does some (antihack?) processing... then you can start the queue after like 10 minutes.
It's probably like the GTA5 that also took ages to start due to poorly done code and devs not caring at all.
Your experience is not my experience at all. Blizzard is the gold standard for me in starting games faster than any other launcher. In fact they are the only one I’ve seen that lets you play the game part way into the download. No other dev seems to have this technology.
I waited hours to play Witcher 3, Cuberpunk, Assassins Creed Odyssey. With WoW I was playing within minutes.
Other games have 'start with partial download' technology. In fact, the core tech of the team that eventually created Valve's Steam was downloading assets on-the-fly so that you could start playing a game before everything was downloaded.
I worked on Guild Wars 2, which has this feature. I made a first prototype of it that streamed all content on-the-fly. It's pretty easy to implement - you have an abstraction that asynchronously loads a file off of the disk, and you can just make that download from the network instead.
The tricky part is when you want to ensure all the assets are there for a specific area before you load in, or simply knowing what order to download things in. For example, there was a starter area of Guild Wars 2 that spawned monsters from many other areas, this meant that the manifest of what was needed was enormous for that area.
So the 'playability' threshold becomes a trade-off between game experience (assets popping in as you play) and quick entry.
Guild Wars 2 needs more respect from the MMO community. The ability to do testing on the live servers, or enable patches with a client reboot and no server downtime, is great.
Guild Wars 2, EverQuest 2 also allow launching after a relatively small (maybe 10%) base amount of data has been downloaded. Its never ideal IMO as it creates longer loading times to areas not already downloaded, but I appreciate that its an option.
In the Beyond All Reason (still "alpha") RTS, the game itself is only a couple Go, but maps are downloaded on the fly, because downloading all of the featured ones would take a dozen Go.
Assassins creed odyssey was apparently 44-50GB at launch. That’s 3 hours @ 45 megabits/second.
This is not an uncommon internet download speed even today (let alone 5 years ago when it launched) according to the steam download stats (it varies by country, check australia for example, US is a little over 100 average):
https://store.steampowered.com/stats/content/
Even at 100Mb, 45GB is 1 hour. Witcher 3 looks to be about 30GB but Cyberpunk is apparently 70-100GB.
Titanfall shipped uncompressed audio on purpose to cut out audio decompression CPU usage. You can disagree with the decision, but it wasn't an accident or negligence.
Why not ship the audio compressed and decompress it once when the game gets installed?
That wouldn’t help people whose issue with large game size is the amount of space used on their disk, but it would help people whose issue is the long download times.
For all we know it actually worked that way. Steam definitely decompresses game assets at install time. The people complaining about game sizes aren't looking at the depot files you download when installing the game, they're looking at their HDD (which is fair!)
My guess is Valve doesn't offer a 'decompress your ogg files to wav at install time' feature though.
You've already bought it, so the game studio got their money, despite having a "bloated" install. So the incentives aren't quite there to reduce the size.
I was thinking about buying Mass Effect yesterday, the recent update, but it requires 150GB and I don't have a Windows partition that big! (I'm Dual booting Linux) So I think money is being lost in people who aren't often gamers and haven't bought big disks for gaming.
> You've already bought it, so the game studio got their money, despite having a "bloated" install. So the incentives aren't quite there to reduce the size.
They may have some of my money, but not much (discount!), and they’re not getting money for DLCs, arguably the reason the base game was so cheap, and I’m not in a rush to buy more stuff that I can’t fit on my PC.
Prior to the era of the initial game purchase being just the entry point I’d have agreed with you, but many modern games have monetization strategies that make uninstallation a problem.
And if they have a strong enough incentive to care about CPU usage, they’re clearly putting effort into optimizing for some audience.
Are spare cores really such a scarce resource? I thought most games didn't come close to using all available cores. And audio decompression is not known for being that cpu intensive.
Meanwhile, over in metaverse land, there are petabytes of content in the big worlds.
I've been working on a faster Second Life / Open Simulator client. Those have a
reputation for being sluggish. That's rapidly changing.
All content is loaded on demand. Textures are stored in JPEG 2000 format.
JPEG 2000 decoders come in several flavors - slow and flaky, fast but need the GPU, and midrange but expensive. If you want to rewrite OpenJPEG in Rust, go for it. You can download part of a JPEG 2000 file and get a lower resolution texture, and that's what's done for distant objects. The general idea is to download textures at a size that gets one texel per screen pixel. (Yes, subpixel textures would be better.) Textures are locally cached to SSD. Background threads are frantically loading and unloading textures in the GPU at different resolutions as the viewpoint moves. The highest resolution is compressed losslessly; the others are lossy.
In metaverse systems with user generated content, there's not much instancing. So the texture load is much higher. Wander around Cyberpunk 2077 and you'll see the same railing show up in completely different contexts. This doesn't happen in worlds with user-created content.
There's no producer or art director to insist on reusing content.
If your game downloads content on the fly, you need substantial content delivery capacity.
Second Life front-ends Amazon AWS with Akamai. Open Simulator, on the other hand, often just hosts the content on a single game server, along with the the simulator itself. My own viewer pulls about 200Mb/s from the content servers for about 15 seconds when you first connect. Then it's done until the viewpoint moves.
Check out HTJ2K if you haven't already, it's J2K with SIMD instructions, it will have solid support because the medical industry is (slowly) moving towards adopting it. Sounds like you already saw NVIDIA's J2K decoder. There are FPGA solutions as well that are insanely fast (CAST and some others).
HTJ2K has support in OpenJPEG (master branch at least), might need to compile with a specific flag or something I can't remember.
I think JPEG-XL is ideal for these types of applications in my opinion, even if you ship the decoder with your client. Their progressive encoding/decoding is much better than J2K in my opinion.
I used to love Second Life back in the day. Can you tell us more about your "game" , and less about the technical details? I know this is HN, but what would you tell a prospective player about your game?
I'm interested in metaverses that work. The NFT industry has crashed and burned [1], and Facebook/Meta has bombed in that space [2]. With the clown car out of the way, it's now possible to make progress.
The lesson of these failures is that there's not much of a role for ads or brands in the metaverse.
They're distracting and don't fit in. The successes, from Roblox to Fortnite to Second Life, charge users a modest fee each month.
A metaverse is not a game. Games can be built within it by users. This has been done in Second Life, but the games are 1) sluggish, and 2) space-constrained, because land is expensive. Those are scaling problems which can be solved.
This requires solving the scaling problems that led to the original article here. You can't just download everything in advance. There's too much stuff in the larger games.
Open Simulator is an open source re-implementation of Second Life servers, written in C#. It's been around for a while, and now it's getting a bit more developer attention. There are multiple federated grids of Open Simulator servers, and content stores where you can buy items, all under different management.
Land is much cheaper than in Second Life, but servers tend to be under-resourced and slow. There are some people working quietly on trying to improve the Open Simulator technology to work better.
Other attempts to solve this problem include Improbable's system. Improbable managed to blow through $400 million on the scaling problem, producing a system that's too expensive to run.[3] (They're funded by SoftBank.) Some good indy games tried to use their system, but the server bill was too high. Their approach is a general-purpose distributed object manager, which seems to be the wrong tool for the job. Otherside uses Improbable, and they only turn on their world maybe twice a year for a few hours for special events.
Games want to be more photorealistic and the hardware to fake it is getting faster and cheaper.
I remember in 2008 when Intel demoed ray tracing[0] for Quake Wars. It ran on a quad-socket (4 CPUs x 2.7GHz) at around 25 FPS, and even Intel admitted that ray tracing is just "brute forcing" the complex optical physics of a scene.
Considering between then and now, we have much prettier looking games that do NOT require ray tracing for their visual effects (and can run on much fewer CPU cycles), thanks to much more "clever" techniques and algorithms (DoF, bloom, subsurface scattering, PBR, etc), and if it weren't for the RTX graphics cards setting the precedent, there would probably be more research into making things prettier without having to "brute force" things.
Nowadays, you can drop a 100gb mesh into UE5 Nanite[1] and skip a lot of the work involved in making things pretty. You don't need to generate or even understand displacement maps if you can just add more polygons to the model. You don't need to billboard the mountain way in the distance if Unreal does it on the fly. Who cares if that means including a 250mb mesh in your assets, versus a 5mb HDRI.
So yes, producers have either become lazy or are simply unable to meet the demands of the hardware by coming up with better algorithms. That's unlikely to change until either hardware stops advancing and either A) can't run the latest games or B) games become too large that they're impossible to download.
Lately I've become hyper-aware of games doing screen-space reflections and suddenly finding it distracting when I point my view down and the reflection of the scene at the horizon vanishes off the water/floor.
I upgraded from a 1070 to a 3080 right when the 3080 came out, and I actually have yet to play an RTX game.
> bloom
I hate that this ever became a thing.
Bloom just makes the scene blurry and bright parts end up blown out.
> and if it weren't for the RTX graphics cards setting the precedent, there would probably be more research into making things prettier without having to "brute force" things.
Not really. A lot of this research included making rudimentary versions of RTGI that could run on hardware of that age(see SVOGI). Of course you can bake everything and make amazing looking levels, that are completely static. But that doesn't work with open-world games, especially if they have different times of day. And it doesn't work in highly dynamic scenes. We basically hit the end of the road for scaling graphics without RT. And now we have hardware that can do realtime path-traced rendering of major AAA titles, and it looks miles better than anything smoke and mirrors could've ever delivered.
I'd rather have more RAM in the GPU and give up the ray tracing hardware.
The real problem in the industry is that GPUs are not getting cheaper. It cost about $250 for an 6GB NVidia 1060 in 2016, and it costs about $250 for a basic NVidia card today that can do slightly better.
Meanwhile, people are buying $100 WalMart laptops and trying to play games on them.
You can buy RX 6600, which is roughly 2x faster than 1060 and has 8 gigs of VRAM for 200 dollars[1]. If we take inflation into account, 1060 did cost around 300$ in today's money. So basically you are getting 30% more ram and 2x performance for 30% less money. In other words you are getting performance 20% better than GTX 1080 for 4x less money(launch price of 1080 was $600). And it uses less energy too, so TCO difference is even higher. And I'm not even talking about expanded feature set.
I mean, you could have a bunch of other things in the GPU instead of those cores, but the reality is that ray tracing serves multiple target markets for GPUs very well, and actually meets a demand people have: better graphical quality. In contrast, while I'm jealous of my friend with a 16GB 4080, versus my paltry 10GB 3080, it does not realistically impact anything I do outside of occasional ML retraining tasks. Neither did my old 8GB card. Gaming workloads just generally don't need it, because people there are mostly bound by compute, not texture or RAM limits -- and solutions like super-resolution/upscaling instead improve performance and memory usage all at once.
> and it costs about $250 for a basic NVidia card today that can do slightly better
Eh, the 3050 is like 30% more efficient in every single benchmark versus a 1060, with 30% more VRAM, and costs like $280. I don't think 30% is "slightly better". The Arc A750 also is around the same price point and is nearly 80% better, but I admit Nvidia has the best software stack on the market at the moment so the 3050 is a more fair comparison.
> Considering between then and now, we have much prettier looking games that do NOT require ray tracing for their visual effects (and can run on much fewer CPU cycles), thanks to much more "clever" techniques and algorithms (DoF, bloom, subsurface scattering, PBR, etc), and if it weren't for the RTX graphics cards setting the precedent, there would probably be more research into making things prettier without having to "brute force" things.
Unreal Engine 5 has a new lighting system called "Lumen". It seems to be visually almost on the level of normal brute force ray tracing, while not requiring any special ray traycing hardware and still running with decent performance on mid range hardware. It's a smarter approach which tries to accumulate lighting information on textures over time instead of computing everything again for each frame.
> and even Intel admitted that ray tracing is just "brute forcing" the complex optical physics of a scene
Modern ray traced games are much less "brute force" than what they were doing back in 2008. The use clever techniques and algorithms like temporal denoising and radiance caching to achieve high quality ray tracing effects with a tiny number of secondary rays per pixel (The budget is often under one ray per pixel).
It's not really fair to call it "brute force" at all, just a different set of clever algorithms.
> Considering between then and now, we have much prettier looking games that do NOT require ray tracing for their visual effects (and can run on much fewer CPU cycles), thanks to much more "clever" techniques and algorithms (DoF, bloom, subsurface scattering, PBR, etc), and if it weren't for the RTX graphics cards setting the precedent, there would probably be more research into making things prettier without having to "brute force" things.
Despite being a graphics researcher, I’ve been a little skeptical in the past about ray tracing in games. Recently I’ve started hearing for the first time from some hard-core gamers that turning on ray tracing makes a “massive” difference in the quality of the experience, and that they’ll use it as long as the frame rate can stay above 30 fps most of the time.
That said, with all due respect, your speculation about clever non-ray tracing techniques seems a little off-target from my perspective. There has been, and still is, a huge amount of research into these, and the entire field of techniques is hitting limits and failing to progress as fast as ray tracing. There are multiple reasons for this. One problem is that many of these one-off raster effects require their own memory & compute. Another is that they each require their own implementation, each require their own tuning, they impose constraints on the art and the artists, and they only work in narrow ranges of curated input. Shadow maps are a good example - needs a buffer of it’s own, and get too close or too far away from the shadows, or if the scene is too big, and they look terrible. You have to spend precious art+dev time just to make them work, you have to spend precious memory for this one effect, it eats into your rendering budget, it only takes care of the shadows, it’s quality has a hard ceiling limit, and you still have 13 other effects to work on that have the same story. With ray tracing, once you setup the memory for it, it can render all the effects at the same time, it’s the unified theory of rendering, and doesn’t require per-effect memory or much per-effect tuning, not like the grab bag of raster and screen-space effects does. Ray tracing has it’s downsides, and it’s certainly cycle-hungry, but there is a good reason that game devs are gravitating toward using ray tracing, it’s because ray tracing is fundamentally easier and fundamentally looks better than the one-off raster effects.
Regarding ray tracing being ‘brute force’, keep in mind that if you’re rendering many instances, or have lots more geometry than pixels visible to the camera, ray tracing can be more efficient than raster. Is usually isn’t in today’s games, because games have hard budgets that limit the amount of geometry - mostly due to raster algorithms! - but the trend of increasing geometric complexity is clear, and that ray tracing is the likely path forward. Nanite is very interesting and has made waves - it’s being adopted and used for the same reason that use of ray tracing is growing, primarily because it makes game development easier. Maybe we’ll see these two things merge in the future, they are not mutually exclusive. Nanite is carefully designed to make sure you’re always rendering polygons that are about 1 pixel in size, plus or minus, and Nanite has pushed scene complexity, but there are certainly open questions about whether this approach will continue to work as complexity grows.
>The crazy thing he mentioned is that if there was zero compression, the latest COD games genuinely had over 1 terabyte of uncompressed assets.
I think what this shows is there should be smart delivery of assets. Before installing perhaps you should be able to choose your graphical fidelity and only download assets related to that fidelity level and below.
This makes manifest building and delivery more complicated, and probably UX too, but I'm sure it's a problem that the infra/marketplace provider (Valve, Epic, Microsoft, Sony, etc.) could solve and make easy for devs.
> I think what this shows is there should be smart delivery of assets. Before installing perhaps you should be able to choose your graphical fidelity and only download assets related to that fidelity level and below.
War Thunder does this - in the game launcher you can choose whether you want the highest resolution textures and most detailed models or not. You can even get a minimal client without vehicle interiors as well (only 3rd person camera available).
It makes a lot of sense to me, to the point where it's surprising that all games aren't embracing this approach. I mean, I don't have a lot of storage, nor good hardware - why should I be forced to download high detail assets that I'll never even see because instead all of the models will be at lower LODs and all the textures will be of a lower resolution.
It feels like the game engine developers don't really care about bundling assets by fidelity and each game acts like it just owns your entire disk.
That's precisely why I don't really play AAA titles anymore and mostly go for pixel graphics and indie titles. There are actually surprisingly few well optimized games out there.
> I think what this shows is there should be smart delivery of assets. Before installing perhaps you should be able to choose your graphical fidelity and only download assets related to that fidelity level and below.
The Diablo 4 beta had this option, drastically reducing the download size if you didn't need 4K
Have you played a modern game? they have amazing levels of detail in the tiniest places. One game I play, Gran Turismo 7, claims that a single car model start to finish takes months to model for a single modeler. Some of the details in those cars that you can see in VR you can absolutely not see in-game normally, such as the door lock controls or the key-hole for the driver's door being exactly like on the real car, even for something so rare as a McLaren F1 longtail. There is so much waste of time and space dedicated to this stuff, but I love it personally!
Gran Turismo in VR is an amazing experience. I'm not even a racing car fan, but holy crap does that game really capture an experience. I'm somewhat afraid to find out just how much data goes into a single model.
The reality is a game has increasingly more content nowadays. Remember, there's a lot of "games as a service". So you have many "seasons" of additional content. You may not play all the content, but someone is. And when people want 4K and 8K textures, different variations so effects don't look the same twice, on top of all the other sizes, plus all the other effects ... it adds up. Never mind huge open worlds, where the asset footprint for just the map might be 100GB.
Having been in a few AAA studios myself (doing database stuff) and sitting next to artists and even art directors, the sheer amount of effort, headcount, etc. put into the art pipeline is genuine madness. In a large studio, art is over HALF the production cost, sometimes more towards 60-65%. Heck, one of the things sometimes brought up about Red Dead Redemption II is that there were three full-time employees in which all they were responsible for was horses. 6000 man-hours a year, on just horses.
To sum up what Chris said towards the end, "The only thing keeping these numbers down is the compression algorithms. If we didn't have these newer compression algorithms, we'd probably legitimately be at a terabyte. I can actually tell you for a fact, that there are over a terabyte of assets in the COD engine."
He completely misses the point. Games don't take hundreds of gigs because they have sub-optimal compression. They take hundreds of gigs because they don't use ANY compression. Jedi Survivor takes 139 gigs on Series X and mere 44 gigs on Series S. It didn't come out yet, but I'm 99.99% certain that this difference is just from inclusion of 8k textures into Series X version. That kind of scaling means that they don't use any compression beyond something like BC7. And some games are not only without any compression, they ACTIVELY duplicate their assets on disk so they load a bit faster on HDD. And these builds end up on modern consoles that have ludicrously fast SSDs.
Agree - OP even links to this[1] PC gamer article which calls out Hitman 3 as having a much smaller size compared to Hitman 2 (and Hitman 3 includes all of Hitman 2). How did they do it? Compression[2].
There was more to it than just compression. Hitman 2 allowed the player to install/uninstall each level individually, and the way that was implemented meant that each level had its own copy of every asset it contained. Hitman 3 removed that option and simply shipped every level in one unified package, with assets used in multiple levels only stored once.
Hitman 2 had 6-8? locations. They could have pretty easily packed the assets into a hundred files based on use and made each level install a different subset.
In 2003, I found out that America's Army (the US Army videogame) had compressed and decompressed all their textures at some point, so they were paying the quality hit for compression and the space hit for RGBA. This was a time when the game still took a day to download, even on a half decent connection, so >200% footprint inflation kinda mattered. The more things change, the more they stay the same!
I have fond memories of that game. 80% of the time I would play very patiently and carefully. Occasionally, though, I would go Rambo and take out the whole enemy team in 30 seconds.
If you compress data, it will be smaller and require less i/o to load, so it will load faster. That's ESPECIALLY true on HDDs, but even on SSDs there are decompression algos that run faster than SSD bandwidth. The uncompressed asset cargo cult is hurting everyone who plays their games, no matter what kind of disk they use.
Compression does not guarantee faster loads anymore. Uncompressed assets can be DMA'd directly to their destination. This is part of why GPU decompression (gdeflate like in directstorage, or the special decompression tech in the PS5) is on the rise because it allows you to DMA resources to their destination on the GPU and then decompress them there to avoid the loading time hit involved in decompressing them on-CPU after copying them disk->page cache->userspace app memory.
For HDDs, yes, compression is generally a win. HDDs are dead as far as game development is concerned, all modern targets have flash storage.
Some compression codecs that produce smaller files are also MUCH slower to decompress, which swamps any benefits from the reduced i/o. This is part of why sometimes people pick less efficient formats for use in games.
Jedi Survivor isn't shipping on any platforms constrained by an HDD, so it would be odd if they were making any optimizations for spinning rust, especially when they come at the cost of precious SSD space.
I remember Fallout New Vegas was almost 50 GB because it was a parallel install of all language/localization files. Once you manually deleted the languages you didn't want it went back to like 10 GB.
As I recall, this was a purposeful choice. Titanfall came out in 2014. The minimum requirements were a Core 2 Duo at 2.4 GHz or Athlon X2 at 2.8 GHz. I think that puts it on processors dating back to 2006 or so.
Games sometimes have a lot of sound effects. It's easier to stream 35 GB of data from disk than it is to decompress, like, 10 streams of audio at the same time with whatever CPU you can dedicate to audio.
I do think they could have figured something out, like compressing the right subset of audio assets, but I just want to give the context for why they made that choice.
That's like 50 hours of uncompressed stereo audio at CD quality. (2×16bit@44.1kHz) Do games really have that much stuff in them these days? I mean, I wouldn't doubt it, but it's surprising.
For story based games (with voice acting) absolutely yes. Dozens of story characters, hundreds of supporting npc and all of them talking as either part of your missions or just to fill in the world.
Maybe 50 hours is a lot for a non story based game but even then once you factor in all the sounds and all their variations to reduce repetition you might be up there.
MP3 isn't slow. I was playing MP3s on a 100 Mhz 486 DX4 in 1996. Certainly a system in 2014 can handle decompressing MP3s on the fly.
Even if you didn't want to do it on the fly during game play, they easily could have been decompressed at load time and it would have taken only a couple seconds. It's not like ENTIRE 35 GB of audio needed to be loaded all at once, nobody had that RAM back then, and very few people (especially gamers) have that much now.
Let me tell you a story about playing The Settlers IV on a 233Mhz 64MB RAM Voodo 3 2000. This game was cool in that it had two audio features: a soundtrack on the CD itself, played using the CD drive's DAC [^1], and you could also drop MP3 files into a folder where the game would play those instead. It was common practice to use a NoCD crack when playing online, because the CD check took long enough you could time out of the lobby, and if you forgot the CD you got booted. That meant most online gamers had MP3 files, and no CD.
The minimum requirements to play this game were a 200MHz CPU w/ MMX and 64MB RAM - I was pretty close to this baseline. So anyway, I discovered that the game played at much better FPS (like 30 instead of 5) if you turned the music off - but only when playing MP3 - CD Audio had no hit. Now perhaps the game used a sub-par audio codec, but that single MP3 decode stream was enough to make the game unplayable.
Anyway that's not to say that I would expect MP3 decoding to be a problem in 2014, in fact you can likely play audio with no noticable increase on CPU usage, but when you have multi-stream audio (think voices, background music, sound effects from various channels - guns, explosions, etc.) I can see it starting to add up - especially when the CPU is already constrained for the graphics, game logic and perhaps of course everyone's favourite anti-piracy/anti-cheat logic.
[^1] For younger readers, yes, CD Drives used to come with built-in DACs and a special cable you could hook directly into the audio card, allowing you to listen to CD Audio on PC for "basically" free in terms of CPU cycles.
Yeah CD drives were also CD players - I had rigged my computer with a separate tiny power supply for the drive so I could have it on without the computer being on - put a disk in and it would start playing audio through the headphone port on the front and out the special wires on the back, and if you had the right audio card that would even play out the speakers with the computer off.
I've benchmarked it myself in my game. It adds a measurable amount of time to loads if you're decompressing all the audio up front to reduce latency, and during streaming it increases CPU usage. Obviously a modern machine can handle a few ogg/mp3 streams in the background during gameplay, but some games are playing dozens or hundreds of sounds at once.
"It's not like ENTIRE 35 GB of audio needed to be loaded all at once" misunderstands the nature of games fundamentally. Some games do potentially need the ability to access any of thousands of sound effects at a moment's notice with the lowest possible latency.
This is likely where some of this comes from - trying to keep the PC and console versions in sync and having to do tricks like swapping space for speed.
What formats are better suited? To my knowledge most games use mp3-like compression for all their audio. Console hardware often has decompression support, it’s reasonably compact, quality/bitrate can be tuned by the sound engineers and supports a wide range of content.
It seems like on PC one would want to use Opus instead of the older MP3 or Ogg Vorbis. It has a number of performance-related improvements in addition to better audio per bitrate, although the higher-quality codec out of its two internal modes is still conceptually similar to MP3 & Co. so I figure it wouldn't necessarily solve all performance issues either.
Ok, how much uncompressed audio does need to be resident in RAM, do you think? Last gen hardware had 8GB total. I think the XBONE could only use 5GB for games. That was shared for CPU and GPU. Is it worth kicking out a lot of texture data so you don't have to stream audio?
This feels wrong. In that, I'd be surprised if they are not using compression. I think folks are just drastically underestimating how many resources are under discussion. And how big they are.
Heck, just the geometry is probably far larger than folks anticipate.
Thinking more on it, I am not too shocked that they would use uncompressed assets "at rest" on the machine. This is especially true for games mailed out on disks or cartridges. I'm still a little surprised, so the links are neat to read on.
I am kind of curious on why they wouldn't use compression in transport, as it were. I'm still mostly assuming that they do.
This list appears to be largely indie games, many with sizes under a gigabyte. I'm not sure why at-rest compression would be a high priority for a small team when their game fits on a $5 thumbdrive and will be compressed during the install process by Steam or Itch.
It's true. I've seen tons of games use basically uncompressed audio, badly compressed textures, etc...
People often repack video games, making programs to compress and decompress them so they are easier to pirate. It's not uncommon to see a 2-3x reduction in size, for bit-identical results.
Yes, that's the problem. You can compress them for size at download time, and recompress them to a GPU-optimal format at the time of installation. You may even be able to do it at runtime with modern hardware, shortening loadtimes and sometimes even improving performance indirectly.
You can combine the approaches! KTX2, for example, supports "supercompressing" GPU block compressed formats with LZ4 or Zstandard. The project I have been working on ships BCn textures compressed with zstd and it has been pretty dang good.
Essentially any modern image compression format is going to be limited, or close to, by storage speed (for decompression). So unless the user has insanely fast internet (and the servers are also able to saturate their connection), it will be faster in the end.
Gamers being gamers though they’re not going to say “well, we on reflection I’ve not had to wait so long”, they’re going to slam you with one star reviews for the fact there’s an hour long loading screen on first start.
The recompression/decompression logic could (and I would argue, should) be packaged separately for distribution platform instead of being embedded as part of the game's logic. In the case of using Steam, it would be no different than how it shows separation of time spent downloading vs time spent installing.
This allows it to be ran automatically after download without having the user need to launch the game.
You can add lossless compression, like Kraken, on top of that. PS5 has hardware accelerated Kraken decompression. And you can encode textures for GPU in a way that is better compressable with lossless algorithms.
Hardware accelerated is still not zero cost. Plus if you are publishing for the other console and/or PC there is a question as to whether you want to do the extra work for one platform.
That is a little surprising to me. Especially if we are talking about games that are delivered online. Thinking on it, I'm actually only surprised for online delivered games. I don't see any incentive to reduce size on delivered disks.
I'd expect that they use a compression that is good for transport, at transport time. I'm not too shocked that they have the assets in a way that is easy to use at runtime.
This is exactly it - and once released there is rarely the motivation to go back and optimize side things. The only game I know that has done anything like that is Factorio, or the open source reimplementations of various engines.
The crackers who remove copy protection often repackage (or someone else does) to greatly reduce size; since they’re in there poking around the code they can see what isn’t used.
This isn’t new, either, ancient console games have unused assets and even entire levels in the code but disabled.
I can see some benefits to having uncompressed assets in that it would help with asset loading. I'd still be surprised to not have any form of compression at all happening. I'd also be curious to know just how much the assets could be compressed.
Um, no. It's not speculation. Many games have been shown to use uncompressed assets, that's an objective fact.
The thread might be speculating about _why_ they do that, or whether this particular game does that, but that's a non sequitur, that isn't what I was talking about.
Your statement that some games in the past have used uncompressed assets is a non-sequitur.
Has any game in the past 5 years that has an install size >100gb used uncompressed assets? I would be very very surprised! The likelihood of this being the case on PS5 or XSX hardware is approximately zero.
Yes I’m familiar with the infamous uncompressed audio in Titanfall.
Hitman was a special case of bundling older games designed designed for spinning disks.
Fortnite didn’t publish what they did. But I promise it’s not “bEnableCompression=true”. Compression is used up and down the stack with the utmost of care. Even the older games that didn’t use compression did so only after great consideration of the trade offs.
“Games as a Service” games like Fortnite, Destiny, and Call of Duty have often bloated and then been optimized. The fix is never bEnableCompression. It’s more about how the content is sliced and packaged. How to efficiently serve new content is a really tricky problem. And a somewhat new problem that didn’t exist 10 years ago. At least for consoles.
Jedi Survivor having uncompressed textures is a speculation, but I'm extremely confident that it is this way. The fact that games nowadays ship without LOD systems, without texture compression, without adequate streaming is not just speculation. It is a fact.
By texture compression I meant something like LZ4 or Kraken on top of block texture compression, sorry for ambiguity. Of course they use block level compression. But it seems that it's the only thing they use, as I said in my top level comment.
You'll find that's pretty common since block compression already reduces sizes pretty effectively. At install time the files will be compressed further over the wire before being decompressed, so I imagine the payoff is not that big in many cases. I do wonder how much of it is that a Kraken license costs money and it takes time to integrate, though...
The geometry would be hard to compress, but I suspect there's some clever tricks with some kind of fractal compression[1] that could drastically reduce texture sizes for camo, foliage, etc. without any effect on aesthetics.
Encoding is the hard part, but only need be done once when building the final release.
> there's some clever tricks with some kind of fractal compression
It's a shame that IFS hamstrung the adoption of fractal compression with their expensive license fees - wouldn't necessarily have been The Future Of Images but at least we'd have had the option.
When I was messing around with this (~1995?), there was a Yuval Fisher paper floating around which did a good job of explaining it. Although it looks like it now costs ~$40 to get that paper or ~$100 for his book. Helpful!
Saying "They take hundreds of gigs because they don't use ANY compression." is absurd when you immediately have to caveat that codecs like BC7 are used for texture data, which makes up a large portion of the shipped data.
Sony included dedicated compression support in the PS5, licensed from a leading middleware company, specifically to make it easier to use compression in games. Games are using it.
I don't know why you're trotting out this misinformed BS.
You can also simply watch the statistics when Steam is installing a game - the amount written to disk is larger than the amount downloaded over the network, because they also compress assets in flight to reduce bandwidth usage (the most important thing, since raw assets often load faster from an SSD as long as you have available storage.)
You can argue every single file should be compressed at rest on persistent storage after install, but that's a losing argument. People really hate longer load times, so compression at rest is only appropriate if it improves them.
Yeah, I could've indicated better that I'm talking about on-disk compression of files.
>Sony included dedicated compression support in the PS5, licensed from a leading middleware company, specifically to make it easier to use compression in games. Games are using it.
Not really if we look at 3rd party games. Jedi Survivor is likely not using that since it takes 147 gigs on PS5 and only 139 on Series X. Warzone doesn't use it, Fortnite doesn't use it(despite Epic's acquisition of RAD), Hogwarts Legacy doesn't use it, Elden Ring doesn't use it, FIFA doesn't use it. These are some of the most popular games on the platform. My point was that he is talking about something beyond LZ compression when most games aren't using tools that are available off-the-shelf.
>You can argue every single file should be compressed at rest on persistent storage after install, but that's a losing argument. People really hate longer load times, so compression at rest is only appropriate if it improves them.
We have hardware-accelerated decompression in all current gen consoles and GPU decompression on PC. We would be in much better world if everyone used that.
> Fortnite doesn't use it(despite Epic's acquisition of RAD),
You're talking nonsense. I worked on Fortnite and have direct experience with it and the HW decompression on PS5. I can even point you to the patch [0] where Fortnite enabled oodle compression and shrunk the game by 60GB.
I find this extremely unlikely. The packaging tools by default will compress files when building a package. Decompression is transparent and done by the hardware; the game doesn't have to implement anything or link any libraries. You would have to explicitly select "no compression" during packaging. Warzone definitely uses it - I worked on Cold War, Vanguard, and MW2 specifically on streaming and decompression across our supported consoles.
Sorry, the fact that Xbox version is smaller confused me. What might be the reason for Xbox install being 20 gigs smaller than PS5 one given that PS5 compression is generally better?
Imagine you have a 1GB game, with a 500MB patch. To be able to play the game while downloading, you need the full patch space available. To apply the patch to the game, you need to grab the old 500MB, stuff it somewhere, and replace it with the new 500MB (in that order, otherwise if the update is interrupted, the game is screwed and you need to redownload the whole thing).
>(the most important thing, since raw assets often load faster from an SSD as long as you have available storage.)
>You can argue every single file should be compressed at rest on persistent storage after install, but that's a losing argument. People really hate longer load times, so compression at rest is only appropriate if it improves them.
Is this really true? I feel like you'd need an absolutely blisteringly fast SSD to match something like an modern, optimized lz decompressor's speed.
In terms of attracting engineering talent, how does the video games industry fare?
In a crunch situation, I can think of how easy it is for some to be able to circumnavigate various pitfalls while others fall into them and have no time to address it.
Very much a skill/experience thing. There’s countless examples of games that run horribly given their level of presentation, or have no business being so impressively performant on the hardware they’re running.
I can guess what execs will pick when faced with “delay the game or ship a massive bundle.”
It attracts truly extraordinary talent. One of many examples: https://www.youtube.com/watch?v=eviSykqSUUw . There is more technical ambition in a single slide of that project than an entire department at places I've worked.
However, it also tends to demand more and pay less, so there is churn and burnout and when it comes time to triage, the triage can cut very deep.
If I were to dive into the JIRA for any one of these "missing compression" incidents, I wouldn't expect to find an easy win, I would expect to find "missing compression" beneath a pile of much worse bugs like "level 2 crashes on AMD cards." So it goes.
> However, it also tends to demand more and pay less, so there is churn and burnout and when it comes time to triage, the triage can cut very deep.
Yeah, I've often heard that the advice for people who really want to make games is to get some friends together and try to put out something together as a side gig while you work somewhere boring to pay the bills.
SWE at a game studio, especially any AAA game studio, is an awful experience. The expectations are high and the salaries are low because there's a seemingly unlimited bunch of young adults that want to make games.
And yes, it takes extraordinary talent. Your code needs to be able to generate a frame in under 16 milliseconds in order to maintain 60 fps. If you want to appease the hardcore players with these crazy 360 hz monitors, you need to get your game loop down to a mere 3 ms.
The amount of shortcuts and optimizations needed to pull that off is insane. At that level, you're trying to figure out how to minimize cache misses and branch mispredictions. How many Node devs are thinking about those?
>>SWE at a game studio, especially any AAA game studio, is an awful experience.
Just as a counter point - I work at one of the largest AAA game companies and we have great work life balance(been here 9 years). Don't remember when was the last time I worked more than the required 7.5h/day, never worked on the weekend, the people I work with are all passionate about their work and want to deliver something great. The pay is "ok" - it's not exceptional but it allows for good quality of life. In comparison all my friends who went to work in traditional software development are paid somewhat more than I am but they all 100% hate their jobs, so......I don't think it's a bad trade off.
As for the infinite supply of people wanting to do this - it's only really true at junior level, as soon as you talk anyone above junior it's extremely hard to recruit and you might be getting a handful of candidates a year. We've been trying to hire a senior rendering programmer for over a year now and we've had like 5 candidates total.
Game programmers can do some amazing things, but they generally aren't given the schedules to do them "properly".
If most commercial software is a feature factory, games are "megafactories" because they are not really competing on the robustness of the implementation, just on delivering a certain kind of experience that delivers some interactivity and shows off some assets, and that tends to lead towards approaches that hack one feature on top of another and then lean on manual testing to catch "enough" errors. Many games also take characteristic shortcuts that eliminate some options for game design in order to simplify the tech.
The larger companies will often keep their tech teams separated from their game teams and treat the tech coders more generously. This comes with all the pluses and minuses of siloing the work - sometimes they solve the wrong problem, but do it well.
They can get great people but then choose to spend their effort solely on features and ignoring things like size and speed - for example, the GTA V loading time issue (https://nee.lv/2021/02/28/How-I-cut-GTA-Online-loading-times...) is a good example, where GTA V did many things really well technically but then completely ignored a huge loading performance issue for many years; things like that aren't "very much a skill/experience thing", the difference is whether the company cares.
> In terms of attracting engineering talent, how does the video games industry fare?
When I talk to people in the game industry, it sounds a lot like my experiences outside it. Plenty of programming skill to go around, but the skill is often allocated inefficiently, and everyone's under pressure to just ship stuff and move on to the next deliverable.
game developers work so much harder than enterprise developers, and enterprise developers look down on game developers as is they are sub-human or something. it's completely inverted to reality.
I'm not saying that you are doing this, I am saying that this is done.
watch some GDC developer videos on YouTube. these people work HARD for performance.
And, besides, his solutions aren't really good. Radical solution would be shipping image generating model with the game and generating textures on the fly. That's how all texture-related data goes from being measured in hundreds of gigs to being measured in hundreds of megs. So basically becomes non-issue.
8k texture with mipmaps takes around 100 megs uncompressed. Downloading it would be 8-10 seconds on 100 mbit connection. It is definitely possible to generate that image in 8 to 10 seconds on consumer GPU if you are using some kind of consistency models that are trained to generate specific game style and inferences in 8 bit.
I doubt you can do it in 8K at that speed and also why not just compress it? Any quality degradation from compression will be less than the artifacts produced by an AI image generator
All inference of Stable Diffusion is done in FP16 at best, with multi-step models, and models themselves are highly inefficient. I would be surprised if in May 2024 4090 isn't capable of generating 8k(8192x8192) Stable Diffusion quality image in less than 1 second. And also you wouldn't realistically need to
> also why not just compress it? Any quality degradation from compression will be less than the artifacts produced by an AI image generator
Because it allows for much more variance in content. I'm a bit sick of seeing tiled textures everywhere TBH. Also virtually all textures would be AI generated in less than 5 years.
I think this is exactly right. The next step is generated meshes/textures and generating entire worlds. You can do generation in a smart way so that performance is not an issue. Predict what you will need before you need it. Generally you would almost never need to generate an 8K texture from scratch, you would generate it from a 4K texture you already generated earlier, which would be much faster. You could have specialised asset generation HW on a GPU.
This is mostly true. The packages of assets for games are compressed but the individual assets are usually not. It’s mostly fancy zip of files with maybe a different header.
When optimizing load times, it’s usually console that’s the benchmark. PC is assumed to have faster or equal hardware so they optimize for the lowest denominator. Consoles aren’t PC’s but they are very close. They lack some instructions while gaining others. I can’t speak on current gen consoles but older ones like the Xbox 360, only could handle zip format and at a slow pace. You could show a loading screen while you unpack the data in memory or you can unpack on disk on installation saving some loading time, or it’s all uncompressed on consoles leaving the console to load the data as fast as console do.
The issue at hand around install size comes down to the engines, their ability to decompress asset bundles, the fidelity of the assets (8k vs 8k,4k,2k,1024) if no runtime optimization is done. WAV sounds vs OGG vs MP3 vs flac, there’s a lot of shit in those asset packs. You could compress it all like they do, into a folder structure you can index and load from, or you can write that game code to make boss fights cause camera shake, the decisions you make making a game are really all about making the game and little to do with how best to deliver your content.
Some texture formats like DXT are already compressed. Raw PNG’s could have compression. All these things are possible if the engine supports it.
There's no excuse for Jedi Survivor not to compress it's assets. Respawn is skipping last-gen consoles with this game. The modern consoles have significantly more CPU power at their disposal, and dedicated hardware for decompressing texture and sound data.
There is no excuse for them to be shipping a 150GB game on the PS5 when so many cross-gen games are 40-60% smaller on PS5 than PS4
It's easy to say something like that, but as mentioned above, games are generally duct taped together at the literal last minute and it's easy to miss things that, to outsiders, seem obvious. The difference is that we aren't looking at literally every piece of the game and rushing to fix as many bugs as possible, like the devs, we're just looking at the file size.
Of course, the other possibility is that they were aware of it, but it was somewhere in a long list of things that they would have liked to fix, but did not have time for.
that IS his point. the problem is not the compression, the problem is the size of the assets. compression can not solve this. alleviate it a bit, yes, but not solve.
he's saying that game asset paradigms are headed in the wrong direction rapidly.
The problem is that it wouldn't help. We have good enough solutions right now, but not many are using them. Better compression will be nice, but it will yield less results than adoption of something that is already here.
Compression optimizes for wall clock time which is (roughly speaking) sum of time it takes to get the compressed data and time it takes to decompress it. Downloads over the network are almost always bottlenecked on the network speed, so the heavier compression you have the better. However, when loading data from the local disk, the I/O can be so fast that the decompression time becomes the bottleneck and slows down loading.
Because of that optimization for download time and startup time end up optimizing for different things.
With loading speeds so different in these two cases, I wonder if the best solution would be to recompress the data with a different compression level during download and installation. For example, download textures compressed with a good but slow and non-random-access image compression format, and then convert them to a lighter GPU-friendly compression format locally.
Lossless transfer compression is good to have, but it's not enough. Lossy compression can give you over 10 times smaller files, but it's format-specific and needs to be carefully applied by the developer.
I do think in this case he misses a bit of the point, goes too far:
> They are designed for fast loading and streaming off SSD's. Who cares about saving off a few hundred ms (or a second) when it takes hours or days to download the product onto the SSD?
Most games will have only one two or three downloads over the user's lifetime, and it'll never be instant, will always require some wait.
But we load textures many times if we keep playing a game.
Rich's overall point remains, but I think some balance of optimizing load speed is also very appreciated & very necessary.
> Most games will have only one two or three downloads over the user's lifetime, and it'll never be instant, will always require some wait.
Are there any statistics on that? Because I have deleted and redownloaded games a whole bunch of times because you can't fit more than two or three games on a single SSD anymore.
I've played large games off of hard drives because I simply didn't want to delete 240GB of other files off of my main drive. The game experience was clearly much worse than the developers intended, but their space demands are unreasonable. Especially considering I'm only playing at 1080p!
Give me a 1080p installer that shaves off 75% of the texture size and you'll win more than a few milliseconds because the stupid thing can actually fit on my drive and doesn't need to come off the internet again.
That can be solved on first load with a texture cache. Shader caches are pretty universal at this point. Texture caches aren't (afaik), but I think they should be.
So now we would have faster download & faster load, but we'd have significantly bugger hard drive space usage than we have? That also seems like a concern.
Well, whatever the repackers (e.g. Fitgirl) use seems to work a lot better than what the game publishers use (often no compression at all).
Original Size: 93.1 GB -> Repack Size: from 35.5 GB [Selective Download]
Original Size: 3.1 GB -> Repack Size: 844 MB
And these are lossless, bit for bit reproductions. In many cases assets (audio, video, images/textures) could be using lossy compression for even greater reduction in size with no significant loss in perceived quality (but maybe some loss of advertising/bragging rights).
That depends. If you're on a slow connection, the 40 minutes to unpack something are nothing in comparison to the week it took you to download it. If used by the game publisher, the decompression could happen during download (I believe this is how steam does it).
If you're on a metered connection (common in many of the more forgotten parts of the world), the bloated download costs you real money or is simply not possible (or you have to spread it over several periods of data allowance).
Guild Wars 2 and others solve this with progressive downloads. You may start playing after a fraction of the content is downloaded. There is effectively no downside: new players can't access most of the content anyhow. It also applies to updates: you can start playing after a big update with a partial download. The background process can be safely interrupted and resumed without glitches.
War Thunder kind of has that, except that the "progressive" bit means you need to download the full thing to progress in the game beyond a certain point.
Enlisted, which uses the same engine, is fine though.
The design of ArenaNet's GW/GW2 has always impressed me. 99.4% of all of the game is contained in one file: GW2.DAT, which is (currently) 66.7 GB. I/O for updates is extremely efficient.
I've seen many games create a vast file system mess, and they're forever plagued by "corruption" or something, complete with "fix" tools, "cache" clearing push-ups and players reinstalling after big updates that don't install correctly for whatever reason. Somehow ArenaNet surrounded this whole problem and solved it optimally.
A massive amount of engineering went into making it work and some of the stuff we did was downright evil :) in gw1, a lot of data was compiled into the exe as c constants so that we wouldn't have to do a bunch of file management and I/o to locate and load it. We ended up hitting limitations in the C compiler.
This happens to touch upon a pain I keep getting with Steam. Games like squad keep getting tiny patches (compared to the whole game size), which is great, until you see that steam needs to read and write the whole file again to finish the patch. It's not as simple as their "dumb allocation wearing out SSD" problem either -- there's no API to do this anywhere unlike the common posix_fallocate(). Linux FALLOC_FL_INSERT_RANGE comes close, but no studio will tend their files so they make exact filesystem block-sized units.
I suppose there's always the approach of running the thing like a DB, as in you don't care about the representation of data being the same and liberally mark parts as "unused", compacting once in a while. But then the integrity check & delta update code of your distribution platform needs to understand that too.
You wouldn't even need Linux extensions for that. You just do it like ZIP (or multisession CD-R, whatever) and put a new directory at the end. Just don't ask about streaming.
Slow downloads wouldn't be a problem if shipping actually finished games on Blu-rays were still in vogue. And the disc limitation would make some publishers think twice about shipping 250 GB on 3 or more discs.
I recall a lot of hype more than a decade ago about all the textures going away and being replaced with math formulas to generate everything on the fly. Games were supposed to get dramatically smaller. What ever happened to that?
And there was also the opposite, John Carmack in particular didn't have a problem with using huge amount of data, "Rage" is an application of this idea. The game uses a single, huge, non-repeating "megatexture", with elaborate caching strategies to keep it playable even on slow media (HDD/optical). The idea was to give complete freedom to artists and not require them to be mathematicians. Further down the line, he even considered using sparse voxel octrees, i.e. even more huge 3D textures.
On the opposite end of the spectrum, you have Inigo Quilez ( https://iquilezles.org/ ). He literally draws with maths, and he is famous for his 4k demoscene productions. The problem is that about no one else is capable of doing what he does.
I see that Inigo Quilez work is more in the "render a scene not necesarily in real time" realm. Like, you could make a raytracing program, make it render off camera a shiny scene, and only after rendering it shows you the image.
Because of this, its very suited to math and procedural generation. You just change some parameters, render and show. Games instead have to do this in real time, with less latency and CPU/GPU usage. And procedural/math generated content tends to be CPU heavy (more usage+latency), if not also on the GPU (cant be cached, etc).
Shaders are more on the middle of this, because you want to apply certains transformations to and from rendering data.
It is an order of magnitude more expensive to produce art assets that way than to just hire people familiar with photoshop and maya. Those procedurally generated assets in tech demos like .kkrieger were just that: tech demos.
Asset production is already the bulk of the cost of AAA video game production.
Economics. Supply and demand. To create good procedural art you have to hire software engineers with artistic talent, which are more expensive than the texture and model artists you can find by the hundreds coming out of art schools.
> all the textures going away and being replaced with math formulas to generate everything on the fly.
Anyone who thought this was incredibly naive.
I look around my room and try to think about what textures could be generated procedurally. My walls, carpet, and ceiling? Yeah, sure. The stickers on my computer? No way.
Does it have to be one or the other and not both? I think your room could have textures for small items like stickers but the house itself could be a semi-static procedure? Maybe the tricky part would be group parties that have to share the procedure, maybe sharing a set of procedural seeds?
I think honestly the problem is small disks. Games are achieving almost photorealism on 4k with maybe 150gb. That's quite honestly not bad. You can't store the universe on a floppy disk.
I it is true : I was very agressive on this one. But still, thousands and tousands of triangle feels like so much overweight for me. Specialy for indoor scenes. Quality for the eye is a lot to do with texture, NOT the number of triangle.
And I do not even talk about the GPU work to handle those thousands of triangles.
With 300 points, my scene runs quick&fast even on budget Smartphones ! This is an important issues I can tell you. A video game that runs on all devices ...
I think in today's machine (be it PC, Mac or even Smartphones) the reading-and-transform of a png asset for the GPU is hard coded in the processor : so VERY fast. It works well.
There is no hardware acceleration for PNG in any consumer device that I know of. And there is such thing as VRAM usage, if you decompress PNG to bitmap it would use 3x-4x more VRAM than BC7. If you decompress it from PNG and then compress to BC7 it would increase loading times and basically destroys all point of using PNG instead of lossy compression.
Switching from PNG to basis for my game (currently with a pretty minimal set of textures) was like a 50% reduction in load times with no other changes.
Note that I'm not storing assets at rest in a ZIP file because there's no reason to. If I was doing that it might close the gap some.
Yes I agree but the PNG compresion size is so great ! The Textures in total reduce enormously my game file size. By 15 times : so good. And because it is small the web download is VERY quick. Very....
So in a way I loose a bit on decompresing textures time, BUT a gain a lot od seconds in dowloading reduced textures files.
- Step 1: download zip file containing my XML point file and my 90 PNG texture files.
- Step 2: open zip file and dezip in 91 filed
- Step 3: run 3D games with all the 91 Assets. The (small) game engine reads each 90 PNG files and - of xccourse- transform each one in a GPU compatible texture.
That seems impossibly black or white. I am not sure there is any product which is perfect in all dimensions. At some point you have to compromise with the state of the world and accept some bad with the good.
Rich is a compression expert who wrote a good post, and there are some good comments in this thread already, but a few key summary points to help people understand the tradeoffs here:
* Steam, Itch, etc all compress game files before sending them to you over the network for an install. The amount of stuff you download is usually much less than what gets stored on disk. There are still games with unusually massive updates though, especially ones where the files change enough that delta compression fails. This is difficult to avoid when using some middleware due to the design of its file formats. Compressed archives actually make this worse because you can't delta compress big archives as well as individual files.
* Smaller on disk doesn't mean faster to load anymore. In the era of spinning rust HDDs, smaller usually loaded faster, but there were extremes - some compression formats that produced massive size reductions were also 10x slower to decompress or worse, which could produce slower overall game load times. Now that most targets have SSDs, this means that less compression often produces faster loads. See https://ph3at.github.io/posts/Asset-Compression/ (from elsewhere in this thread) for one example.
* For image and geometry data that is headed for the GPU, it's best to pick compression formats that can either be natively consumed by the hardware (block texture formats like BC7, etc) or that can be efficiently decompressed on the GPU once you copy it there, like gdeflate. This means the file on disk will probably be bigger, unless you supercompress it using something like Basis. Basis adds measurable CPU time to loading, though, and as a result I'm considering doing an uncompressed texture cache in my game to improve load times even though I plan to continue shipping textures as basis KTX2.
* For streaming asset loads, the CPU usage to decompress the assets during gameplay could lead to frame drops, inconsistent frame pacing, and pop-in. My understanding is that Titanfall opted for uncompressed audio in order to keep CPU usage low when streaming in all the sound effects and voices that could play during a hectic match - at that point in time CPU usage for audio mixing was a serious concern; modern CPUs more headroom so it's less of a problem now. It made sense for them to choose this as a trade-off because high-framerate twitch online shooters prioritize low consistent latency.
* In some cases the install you get depends on the hardware you have. It's not uncommon for older platforms to see 'high res texture packs' that could be installed optionally. The Xbox platform has something called "smart delivery" where it will deliver a smaller game bundle depending on whether your particular console needs high-def textures. This takes real engineering work to do, so many games opt not to support it because that time is better spent elsewhere.
* Ultimately, unless your game is too big to fit on people's hard drives at all, there is usually something more important for you to work on vs implementing some complex compression algorithm for every one of your files at rest. You are probably also not as smart as the compression experts who wrote the compression codecs that Steam/Itch/Xbox/Playstation offer, either for install-time compression (to reduce network bandwidth) or texture compression, etc. So in practice you probably just pick an off-the-shelf compression solution and apply it where it improves load times and you leave everything else alone. It doesn't make sense to spend weeks shrinking your on-disk size by 5% when you could spend that time fixing crash bugs, improving accessibility with better control customization, or improving the game's framerate instead.
* Lots of people decide a game "isn't compressed" because you can shove it into an lzma archive and make it smaller. This misunderstands the issue. Many compressed or optimized-for-size formats can still be made smaller by using a really heavyweight compression codec like lzma or brotli, but critically, there are tradeoffs involved - slower compression means slower build times means longer development times, and slower decompression means worse load times. The best compression typically also requires you to bundle all your files together, which loses the ability to load only the data you want when you want it.
An anecdote: Back when I was working on the original WebAssembly spec, I spent multiple weeks doing lots of research into custom compression and how we could design the format to make it compress more efficiently. I even had some chats with the Brotli team. In the end, we barely did anything, because even months of hard work barely produced better results than just Brotli-compressing our files at maximum settings. We were able to improve gzipped sizes tremendously by filtering and rearranging the data, but there's no point doing it if you can just use the browser's built in brotli decoder instead.
Maybe I misunderstand something but to me it seems the premise is wrong.
What's most important about game textures is that they can load fast at a quality good enough not to disturb gameplay. Absolutely everything else is a secondary concern.
Saving a few ms during decompression actually matters when the game streams data from the disk in real time. People also hate loading screens and objects popping-in because the streaming pipeline is slow.
Some games do this by splitting off the highest resolution textures into a free "DLC" that the user can opt not to install if they want to save space. It's sometimes done for audio as well, e.g. optional Japanese voices for the English release of a JRPG.
Remedy's Quantum Break had an unusual spin on letting users choose the install size, it made heavy use of live action FMVs which could either be downloaded ahead of time or streamed on the fly as they played. Downloading the FMVs more than doubled the games footprint.
good idea : one why chitty etxture quality, one with average, one with quality texture.
They have done this on web-videos. Why not on video games files ?
Don't want to blame too much, but it seems like your internet got stuck in the past. I have 10gbits. I used to have 1gbits for a long time before. 100mbits is not that much these days. Switzerland is talking about a government mandated minimum speed of 80mbits in 2024. That's a 7 hour download on minimum speed for an unusually large game. Switzerland is not alone in this regard. That could be a view of privilege, but average internet speeds are still progressing forward fast.
It is an extremely privileged view. In my home country of Egypt, a population of 100 million lives on an average home internet speed of 30mbps with about 300gb monthly data cap, in 2023. I'm sure most of the world is similar.
I beleive it is your internet is a bit in a future. Good for you, but not a reason to blame others. See https://www.fastmetrics.com/internet-connection-speed-map-us... , for example. 30-50 Mbps was median in 2020, even with COVID-induced improvements I doubt things became significantly better today.
You're correct about the internet stucked the past, but for a lot of places in the world even with a lot of money you won't get fast speeds! My hometown is only getting 100/25Mbits max after the previous max 30/5Mbits 2 years ago!
It's not like we don't have the money, I bought a 10900k and 2080 super for my brother back then, it's just that the speed is unavailable. A "git gud" simply don't do that justice.
Renderers have been using compressed textures for literal decades. Not JPG, alas, because the compression formats have to support random access to individual texels.
BC1-5 have been standard practice since before the original XBox. I think when people here are talking about "uncompressed assets" they mean "not applying variable-rate zip style compression on top of BCn compression."
It used to be more common to have optional texture packs like this. It seems to have fallen out of favor, I can offer some informed speculation as to the reasons:
* Every additional configuration i.e. 'texture pack vs no texture pack' is more QA workload to verify that everything works as expected. So adding this option is more work for your already-busy testers.
* Adding DLCs to deployment systems like Steam or PSN is additional work and then when patching the game you would also need to patch the DLC. I can imagine the coordination overhead here being a pain.
* Some users will fail to install the DLC and then complain that the game is ugly or that the screenshots were misleading. This is ultimately a user education problem but it's also the case that the flow for installing DLC on most platforms is awkward.
* The DLC install process adds additional points of failure, so you need to handle all of that now too.
* Users installing a new graphics card will probably be annoyed to have to suddenly wait for a HD texture pack to download before they can test it out.
All pretty minor individually, but it probably adds up to explain why most studios don't do this.
I wonder how widespread developer support for Xbox Smart Delivery (which is basically what you describe) actually is at this point?
The article mentioned primarily download sizes and times. Honestly having +1Gb/S download speeds is quite nice. I also run a local 10Gb/S local network and again it makes large files seem small.
There was a phase of games shipping with uncompressed audio in every language they support for no good reason that ballooned file size (Titanfall)
Some games have gotten around this problem by having higher resolution textures or additional languages being optional installs within whatever ecosystems DLC implementation is. I think this is the smart way to do it.
Examples of games shrinking I are hard to come by but special credit to IO interactive and Hitman 3.
The install size with all the content of the previous games included (If you own them) is less than the install size of the first game. That is impressive and commendable.
Games are weird and some people will look at the file size and think it means the game is better, just the way some people think games with more hours are better. The psychology of people who play video games is weird.
Duplication of assets on disk was a very weird byproduct of HDD file access conventions, but we shouldn’t need that anymore. There is still a shrinking, but valid crossover window between devs being able to assume all storage is solid state. I think things will get better but the debate of file size vs average/expects storage capacity vs Network speed will never end but there is certainly work to be done.