Compressing savegame file

[stgatilov]

I personally don't like 80Mb savegame files

 

They contain a lot of redundant information. It is like a dump of gamestate... They can be packed easily. For example, after compressing with standard gzip savegames become about 20x smaller in size. Player has do it himself if he doesn't want to store Gigs of saves on hard drive. How about compressing/decompressing savefile "on the fly"? Just chain save/load file streams with some zlib stuff. I hope that it won't slow saving/loading too much.

 

I decided to try implementing it myself. I downloaded new 1.03 code and looked how saving is done. The problem is: all the data of savegame is written to idFile *f which is passed to idGameLocal::SaveGame from closed piece of Doom3 source. The implementations of all idFile classes are also closed. So there is now way to insert compression inside idFile.

 

So I switched to another idea: write savegame to memory buffer, then compress it and dump results to idFile parameter. Luckily there is idFile_Memory class which seems to do what I need: store everything that is written to it to memory buffer. The problem is: though I see idFile_Memory declaration I cannot create this object. I cannot reference constructors (undefined symbol). And I cannot find method for creating memory file in idFileSystem. So there is declaration of class idFile_Memory but no way to access it .

 

After some googling I found this version of filesystem.h. It contains method OpenMemoryFile which creates idFile_Memory. This address is referenced here in ET:QW forums, so I guess it is from Quake4 SDK. Has developers fogrotten to open access to Memory files in Doom 3 SDK? Is there any simple way to use idFile_Memory?

 

If the idFile_Memory is not usable, then the only proper way to implement on-the-fly compression is to create my own class with idFile interface. But in this case I'll have to implement all the bunch of functions idFile has for writing different types of data. I'll have to rewrite piece of closed Doom3 source...

 

Maybe someone knows how to solve the problem? Or he has any suggestions about savefile compression? Why is it not implemented yet? Anyone except me wants to see automatic savegame compression?

[Springheel]

I personally don't like 80Mb savegame files

 

Me neither. But my savegames are in the range of 20-50 Kb. A couple crack 100 Kb.

[Berny]

it would be nice if there was an option in the game menu (if it's not too much coding) - so everybody can decide if he wants to have faster save/load - or fewer store needed

 

 

 

 

R Berny

 

 

[Sotha]

Anyone except me wants to see automatic savegame compression?

 

I'm just a TDM mapper/player, but I do have plenty of hard drive space. Big saves do not concern me, especially now when you can really easily clean all unwanted old FM files in the menu. Who keeps old saves anyway?

 

Is compression worth it, especially if it is difficult to implement? (I got this impression from your post.) How much does compression slow saving and loading?

 

The team may have more pressing issues to resolve?

[nbohr1more]

From my understanding, the Quicksave crashes are due to uncertain memory states in the internal id filesystem. If that ET:QW file or another workaround can move this out of the proprietary area these crashes could be diagnosed and fixed more easily...

 

(Crossing fingers that this is not a dead-end that has already been tried...)

[grayman]

Me neither. But my savegames are in the range of 20-50 Kb. A couple crack 100 Kb.

 

Wow! How do you do it?

 

The smallest savegame I have is 20mb (Too Late) and the largest is 60mb (Alchemist).

 

A thousand times larger than yours.

[Aprilsister]

I certainly am seeing save games more in the range of double-digit MBs rather than KBs...

 

But I still would definitely want this to be optional -- at least for Quicksaves. If it could be implemented for stop-and-saves, sure that'd be cool.

[grayman]

Another option might be to uncompress savegames at mission load and compress them at mission end. During the game, save/restores wouldn't be slower, and when you're not in that mission, the savegames are compressed. It would add to mission load time, depending on how many savegames you have.

[Aprilsister]

Well, yes. That's cool too. The reasons I have for my suggestions are that when I do use Quciksaves (which is less often than most I'll bet, ratio-wise {i.e. I usually do a stop-and-save}) I want a quick save... and OTOH, for a stop-and-save, I am in a pause-mode-mindset anyway and often get up and shake myself, get/do something in the real-world, cry out Absalom!Absalom!Absalom!...

[Springheel]

Wow! How do you do it?

 

Whoops. Meant Mb. Most of mine are in the 20-50 Mb range.

[Serpentine]

I guess compressing "uninstalled" mission saves would be viable, I just don't think it's possible to do it on the fly. Also removes the performance related issues of packing/unpacking continually. While we're near the topic, perhaps something to purge old release saves...

 

gray, Tels or greebo etc should know more on the subject tho

[Tels]

I'm just a TDM mapper/player, but I do have plenty of hard drive space. Big saves do not concern me, especially now when you can really easily clean all unwanted old FM files in the menu. Who keeps old saves anyway?

 

Is compression worth it, especially if it is difficult to implement? (I got this impression from your post.) How much does compression slow saving and loading?

 

The team may have more pressing issues to resolve?

 

My HD is almost full, and when I discoverd that my FMs were roughly 1.6 Gbyte (Heart), 800 MB (NHAT) etc in side, despite me saving only a handful times, I wishes strongly we implement compression somehow during save and load (if possible, otherwise externally). Compressing during saving/loading will likely increase the speed of that operation, too, because instead of writing 100 Mbyte to the HD, you only write 5..10.

 

Modern CPUs laugh at compressing that data, but the HD is very slow in writing/reading it.

 

Not to mention that I needed to start to clean up my TDM installation because I simply did not have the roughly 20 Gbyte size it was last week

[stgatilov]

Compressing during saving/loading will likely increase the speed of that operation, too, because instead of writing 100 Mbyte to the HD, you only write 5..10.

I feel the same way. It is very arguable that compression will increase save/load time. Maybe it will decrease it.

 

Right now I'm trying to use CRenderPipe for intercepting the savegame data...

Edited December 24, 2010 by stgatilov

[stgatilov]

Yeah!!!

 

I finally made it! The savegame files are written compressed now (reading is not done though ).

 

I use CRenderPipe to create a pipe. This pipe is passed to idSaveGame for writing. I also create another thread which constantly reads from this pipe and stacks the data into std::vector. After all the game saving is done, I compress the resulting std::vector with zlib's compress and write the result to idFile *f parameter which is the read destination file.

 

So conceptually it is: dump the whole savegame to memory, compress that memory, write it to file.

 

The results are the following:

 

Uncompressed save: 87Mb size, 7 seconds time.

 

Compressed save: 4.4 Mb size, 2.5 seconds time.

 

So writing compressed savefile is MUCH faster than uncompressed. Even if we compress the whole save file at once.

 

 

 

 

The question is: what to do next?

 

The way I use CRenderPipe now is weird I'm afraid that all this mess simply won't work on linux implementation. I think it's better to create some common named pipe class. Then use it in compressed saving/loading. CRenderPipe serves a specific purpose which is not enough for saving/loading.

[rebb]

That does sound pretty cool

[Aprilsister]

So writing compressed savefile is MUCH faster than uncompressed. Even if we compress the whole save file at once.

 

Are you saying that the whole process of dumping to (available) memory AND compressing AND writing is 3x as fast? Or just the eventual writing of the 20x smaller compressed data is 3x faster?

 

...

 

Having taken a curiosity-piqued look inside one of the ~80MB saves from Heart: it's mostly full of code-comments! Maybe the whole save system needs an overhaul?

Edited December 24, 2010 by Aprilsister

[stgatilov]

Are you saying that the whole process of dumping to (available) memory AND compressing AND writing is 3x as fast? Or just the eventual writing of the 20x smaller compressed data is 3x faster?

 

...

 

Having taken a curiosity-piqued look inside one of the ~80MB saves from Heart: it's mostly full of code-comments! Maybe the whole save system needs an overhaul?

 

I measured the time from pressing "+" (quicksave key) to the moment screen unfreezes. So this is not just writing, but the whole saving process.

 

 

This text is not code comments. I think it is something like description of objects from DarkRadiant. I can try to understand what is it, but TDM team knows it much better anyway. It may be difficult and tedious to eliminate it. Though I hope it is not difficult... But compression is completely orthogonal to saving process anyway:).

 

 

P.S. Should I post to "I want to help" forum?...

[Fidcal]

Yes, post to the 'I want to help' forum please stgatilov. This sounds very promising and possibly your coding skills might be of help with other things?

[Sotha]

It is always nice and appreciated to have improvements. Maybe this could be the start of a beautiful and fruitful contributions?

[Tels]

I measured the time from pressing "+" (quicksave key) to the moment screen unfreezes. So this is not just writing, but the whole saving process.

 

 

This text is not code comments. I think it is something like description of objects from DarkRadiant. I can try to understand what is it, but TDM team knows it much better anyway. It may be difficult and tedious to eliminate it. Though I hope it is not difficult... But compression is completely orthogonal to saving process anyway:).

 

 

P.S. Should I post to "I want to help" forum?...

 

Yes, please, that sounds very good!

 

The test is likely the spawnargs, a lot of them are indeed texts. I wonder if we couldn't somehow during entity spawn delete spawnargs like "description" and "editorUsageFolder" (or what it was called). That would not only save memory, but also a lot of savegame space and time (the data wouldn't even get written).

[i30817]

If you control the save game streams, zipping and unzipping is a case of wrapping the input and output streams (at least on the java/c++ paradigm for these things).

[grayman]

Having taken a curiosity-piqued look inside one of the ~80MB saves from Heart: it's mostly full of code-comments! Maybe the whole save system needs an overhaul?

 

Can you give some examples of "code comments"?

 

I found a number of "editor_*" spawnargs saved in the savegames, and hacked the DLL to not bother reading these from the map when the mission starts, since to my knowledge they're of no use to TDM, just DR. If they're ignored, they won't end up in savegames. There's already a specific check in the light entity to not save "editor_*" and "parse_*" spawnargs to savegames, so someone's already been thinking along these lines.

 

This only reduced the size of an Alchemist savegame by 4%, so I don't know if this is a big win.

[LEGION]

The results are the following:

 

Uncompressed save: 87Mb size, 7 seconds time.

 

Compressed save: 4.4 Mb size, 2.5 seconds time.

 

So writing compressed savefile is MUCH faster than uncompressed. Even if we compress the whole save file at once.

 

That sounds great, gimme that!!!

[Aprilsister]

Contains the script name to call when this object is used by an object on the used_by lists. Different scripts may be called for different items by adding another used_action_script spawnarg followed by the item specifier as shown: item specifier as follows: used_action_script_<specifier> . The specifier may be the item's entity name, inventory name, inventory category or entityDef classname.!

 

Okay that is an example snipped from a huge chuck of like-wises taken from my first save in "Lockdown" (I used "Lockdown" instead of "Heart" to verify that this was not unique to "Heartr" and because the saves are corresponding with the FMpk4size proportionally smaller -- basically quicker to load into Textpad...)

 

It may not be technically something everyone wants to agree on as "code" comments, as many folk don't agree on what "code" is (like the term "hack" )... but that is all beside my point (to me Excel macros can be described as "code", hypertext can be as well, fricken' postscript... though, I still might not call those who practice these things "programmers"). Anyway, as I say it's beside my point. My point is as you have both agreed that this stuff is some sort of notation that is not addressed to the machine but rather to the reader of the "code".

 

And basically a whole lotta the savegame files is this kind of stuff.

 

Together with 'maxs' this can be used to use a primitive code-generated clipmodel for this entity. Mins/Maxs are only applied if the 'clipmodel' is empty/invalid and 'noclipmodel' is not set to 1. Mins defines the 'lower right' corner of the rectangular clipmodel bounds, measured from the entity's origin. Note that this will prevent the code from load a clipmodel from the visual mesh.

 

Now, of course with this excellent idea of stgatilov's ("let's compresss these save games") the textual stuff like this will be compressed bunches, but I was just suggesting that maybe it can be eliminated altogether to speed things up and allow for an even more succinct savegave system.

 

Spawnargs of the format 'set XYZ on ABC' are used to pass along and set spawnargs on entities that are spawned during runtime and then attached via 'def_attach' to the base entity. XYZ stands for the name of the spawnarg, and ABC the name of the attachement or the attachement position. For instance, 'set _color on candle' '0.5 0.5 1' would try to find an entity attached at the point candle and set the spawnarg '_color' on it. This is mainly useful for the pre-made combo entities like moveable and static light sources, so the mapper can influence their appearance.

 

IOW, I think you can see what I see, we may just be unnecessarily tripping over semantics.

[Aprilsister]

Quoting myslef from another thread:

 

BTW1: this relates to the savegame compression thing, and since you, grayman, are here I'll mention it here; I did a test saving this for uploading with best zip and best rar, and rar kicked zip's ass for what kind of data presently makes up a savegame --- some 2.2MB to 4.8MB (working with ~44MB raw). If you and stgatilov do decide to tackle the savegame thing maybe you can consider implementing rar libs instead?