development [Shrewds Dev Blog] Game Packaging

[ShrewdTactician]

For the last couple weeks, the team has been looking into the packaging of Renegade X. Specifically, the size and how the content is stored and downloaded. In general the team does actively try to avoid large patch sizes, (looking at you AAA games). We have a custom downloading solution, custom patching solution, our own collection of mirrors, etc. That being said, the game has been under development for a good number of years, and over time unused assets have crept into the hundreds of different packages that make up Renegade X.

For the last couple of weeks, we have been looking into 3 different things related to the game package as a whole. Which I will break down and explain.
1. The amount of assets we have.
2. The "compile" type of packages.
3. The download process.

The first one, the amount of assets. We grabbed all our assets, and also the list of 3rd party maps on our level download section and asked a simple question, how many assets are not referenced by any of these other assets or anything else? Well after touching 489 files, turns out about 3.13GB's worth! So we pulled out unused assets from each package and stored them in a separate folder,  that is not for inclusion in the public release. These files will be included in the sdk release, so that content creators can use them if needed. But in general, that should reduce the file size of the game by a large percentage. Now this wont translate into a direct saving of 3.13gb, due to a number of reasons, but its still going to be a shrinkage overall.

The second point, the "compile" of packages. We are a UDK based game, which means we use a fairly open version of the unreal 3 engine. The unreal engine works on a variety of platforms. PC being one of them of course, but there's also xbox, playstation, wii, ios, android, etc. In understanding that nearly all of these platforms run differently and are limited compared to PC, you can understand that the engine has mechanisms to deal with packaging the game for more resource limited environments. Over time, as these consoles are pushed to their limits, it meant that more optimizations are made to the engine to deal with these limitations. Eventually the guys at unreal worked out that some of these optimizations can be brought back over to PC.

So what does this actually entail? We'll traditionally, if you wanted to run your unreal engine game on a console, you would have to "Cook" the packages to work on that platform. Cooking does a good number of things, including, but not limited to:

1. compress content
2. convert the format to be native to the target platform
3. serialize the objects to be loaded from disk faster
4. compile everything related to the assets
5. update cross package dependency import/export/name tables
6. reduce disk seeking
7. create a startup package with dev specified common used packages, to save level loading time
8. other stuff

Some of these improvements can help PC as well, so we have been experimenting with cooking the packages for the PC platform. We have actually tried cooking before, a very long time ago, and packages that haven't been touched since then should still be cooked, which is why the game release currently stores its data in the cookedpc folder. That being said, our devs have noticed improvements when cooking the rest of the game, but it will be the more hard drive/cpu limited machines that should see the most benefit. For this coming patch, we are aiming to cook all of the content, except map files. The following patch should have the map files cooked as well. This is still being looked at, so we may decide not to use it, if problems arise.

Our 3rd item, is downloading. As mentioned before, our downloading system is a custom setup, and that is from the need to have a better downloading experience then what was available to us and you from stock setups. Recently, we have been looking into improving this system. Our biggest change looking forward, is to implement torrent style downloading, which is used in a variety of different games in this day and age. Torrenting is more resilient and natively supports multiple download sources, compared to traditional http downloading. Even if we implement torrent based downloading, there will still be a fail back option to http downloading, and as such, we are also looking into what can be done to improve the http download experience as well.

Now with these changes, even tho they are made to reduce the size of the game, and the speed of which it loads and downloads, and we generally try to keep patches from being bloated; because many files had to be touched, the actual size of the patch will be larger then typical, as even a slight change to a file means that the whole file needs to be updated. This larger then normal patch size should be a once off, or possibly twice off, as the maps will hopefully be cooked as well, in the following patch.

Overall, result of these changes should be a smaller sized, quicker to load and download Renegade X. Which I think most people would agree, is worth the investment.

[Ryz]

Good job, it's also nice to have frequent updates like this so we know what is going on. Keep up the good work!

[ShrewdTactician]

Thanks Ryz.

Believe it or not, I'm trying to space them out...

[Fffreak9999]

2 minutes ago, ShrewdTactician said:

Thanks Rhz.

lol

[TK0104]

You might want to look into maps, .ini files and preview videos too.

 

You might want to remove:

- CNC-Under OR CNC-UnderRedux (Depends which version is currently used)

- CNC-PTLevel. No one uses that in the game

- Valley and Training Yard .ini file since they are no longer part of the actual game

- (MAYBE) Unused Loadingscreens: Canyon Day/Night, Field X, old Field Day/Night, Fort Night, GobiRedux, GrassyKnoll, Mesa II/Day/Night, Training Yard, Valley, Under OR UnderRedux (depends on which version currently is used)

- (MAYBE) Launcher Preview Videos: TrainingYard and Valley

 

This may save like 1GB

Edited January 7, 2018 by TK0104

[Fffreak9999]

It may end up being one of the improvements to remove it (or it may be needed to reference).

 

Will see soon

[TK0104]

When time comes for a new SDK release, please remove stock maps from the file since they are one of the reasons why it's a lot of GBs.

 

 

 

 

[Fffreak9999]

They are handy for reference points if you want to see how one map handles it, then you can refer to it without potentially and accidentally overwriting your normal map.

[Fffreak9999]

Plus you can copy over assets which you like more easily, say for example, you like a particular rock and texture, you can copy it straight over.

[isupreme]

Thank you for the excellent reading material.   I really enjoyed it.    

I previously had a very limited amount of data each month. The size of downloads was a severe impediment.

  Investments in reduction sound Great.

[isupreme]

Does this affect Black Dawn?

I ask because i just got it working and i noticed some cooked pc files being downloaded.

 

[ShrewdTactician]

no, this only effects renx releases going forward.

bd was essentially an experiment in the udk for trailing making renx. I wouldn't be surprised if bd was a hodge podge of cooked and uncooked data.