Jump to content
The Dark Mod Forums

I'd like to implement savegame compression


stgatilov

Recommended Posts

I can see the problem. How does your CIDFileFrontend::ReadString() method look like?

 

Is it correct that when your CIDFileFrontend::ReadString() method tries to change the foreign idStr it still invokes the gamex86 idStr methods, as they are inlined and there are no actual symols to call?

 

 

Here is the code.

 



int CIDFileFrontend::ReadString(idStr &string)
{
string.Empty();

int len = 0;
int res = ReadInt(len);
if (res < sizeof(len)) return res;

string.Fill('@', len);
res += Read(&string[0], len);

return res;
}

 

Yes, you are right. it calls gamex86's idLib functions to allocate memory.

 

 

Regarding symbols: we have NO symbols about Doom3.exe at all as far as I know. All the coordination relies on virtual function calls which are not crashing because 1. declarations are identical hence vtables are identical 2. each object uses pointer to vtable stored in itself to understand where the function is. And the virtual functions are always prepared for call from outside. Each non-virtual function is uncallable, because: 1. We don't know its address 2. It could be optimized (even if not marked as inline). Calling it requires hacking each executable separately, writing asm helpers and bad stuff like this.

 

I'm trying to get access to Doom3.exe's heap. Maybe I can call some virtual function which creates proper memory and set this buffer to the idStr read.

Link to comment
Share on other sites

  • Replies 57
  • Created
  • Last Reply

Top Posters In This Topic

It's a huge pain either way you turn it. If that approach is proving to be too much hassle I guess we're better off using the pipe, as you had planned originally?

 

Yeah... ID decided not to expose their memory allocation to DLL. And finding virtual functions which allocate memory it also a problem. The only approach I can think of is to create a temporary file with some garbage and call idFile::ReadString routine to allocate memory for idStr :laugh:. Crazy.

 

 

Each instance of CRT in Visual C++ allocates memory in its own windows heap. It should be easy to find that heap for Doom3 and allocate memory from it. Can't find similar info for unix, probably C runtime uses global process heap there. Anyway it gets dirty.

 

Also I think it is not possible to avoid writing idSoundWorld to savegame.

 

 

 

 

Ok, then I'll continue trying OS-specific things. The pipes are already working now, but I don't like the code: it is cumbersome and error-prone (a lot of WinAPI calls, threads...).

 

It would be great to find some windows equivalent to /dev/shm if it exists. Any ideas? I'll try to play with FILE_ATTRIBUTE_TEMPORARY flag.

 

Also I think about taking FILE* object from idFile_Permanent and using windows-specific things with it (like getting HANDLE).

Link to comment
Share on other sites

Just a remark:

 

The "memory allocated in different places" could also be responsible for some crashes that I see with my LODE stuff. At first I thought it is because of double frees, but this is also a possibility. Good to know!

 

And I look forward to compressed saves, because the disk space needed for some FMs is ridioulous atm :)

"The reasonable man adapts himself to the world; the unreasonable one persists in trying to adapt the world to himself. Therefore, all progress depends on the unreasonable man." -- George Bernard Shaw (1856 - 1950)

 

"Remember: If the game lets you do it, it's not cheating." -- Xarax

Link to comment
Share on other sites

Now I'm sure about the problem.

 

In idGameLocal::RestoreGame the ONLY place where idStrs are loaded is a call to savegame.RestoreObjects();

 

This call in turn calls idRestoreGame::ReadSoundCommands() which redirects the flow to closed part of engine with gameSoundWorld->ReadFromSaveGame( file );

 

How feasible is it to replicate what that call does? It should be possible to see what it actually tries to read, then call our own routine to read that data, would that help?

 

*looks at IDA*

"The reasonable man adapts himself to the world; the unreasonable one persists in trying to adapt the world to himself. Therefore, all progress depends on the unreasonable man." -- George Bernard Shaw (1856 - 1950)

 

"Remember: If the game lets you do it, it's not cheating." -- Xarax

Link to comment
Share on other sites

:blink: I wish that someday I too will speak your tongue, coder people. I hail thee. :)

 

I know this rather cool routine, though:

 

5 Rem *** rather cool routine ***

10 print"happy new year"

20 goto 10

 

and it was even better as I recall if you added a semicolon to the end of the first line.

Where are the REAL brits?! The one's we have are just brit-ish.

Link to comment
Share on other sites

Just a remark:

The "memory allocated in different places" could also be responsible for some crashes that I see with my LODE stuff. At first I thought it is because of double frees, but this is also a possibility. Good to know!

 

 

 

To be honest, it is rather difficult to catch this bug unless you do something bad. The closed part of code is mostly isolated from gamex86. The only thing you should never do is: call idFile->ReadString :mellow: . I guess this is the reason why idRestoreGame::ReadString does not call it.

 

 

I think it's better to state it all in wiki? Several heaps, communication with closed part only with virtual functions and the like.

 

 

 

 

Now I think about reverting back to greebo's idea. I added debug output in saving process to know what parts of code write the most. Only 300Kb is written before idSaveGame::Close. It means that 99.6% of save consists of object data. All the objects call methods of idSaveGame and idRestoreGame which are opened. So I just need to rewrite their methods :laugh:.

 

Most of the data will be stored to buffer inside idSaveGame. The closed part of code will write their data uncompressed straight to the file. After everything is done, the cache would be compressed and dumped into save. No OS-specific things, no hacks, no heap mess. Great!

 

The only thing I cannot understand now is what eats time in save/load process. Writing 86Mb to hard drive is much faster that the whole saving process if done in one operation. I hope that when I rewrite SaveGame/RestoreGame everything will become clearer...

Link to comment
Share on other sites

Just a note, if you're using IDA or similar: the mac binaries (lib+executable) are both completely unstripped and make understanding and tracing through things far, far easier.

I have the latest ones lying around here, if you'd like to get your hands on them just pm me.

Link to comment
Share on other sites

To be honest, it is rather difficult to catch this bug unless you do something bad. The closed part of code is mostly isolated from gamex86. The only thing you should never do is: call idFile->ReadString :mellow: . I guess this is the reason why idRestoreGame::ReadString does not call it.

 

Well, but what happens if you do something like "new idRenderModel()" in your code (gamex86) and then let D3 clean it u (via deallocation) wouldn't that exactly trigger the bug (I get f.i. a crash when closing D3 whenever I used one of my special objects).

 

I think it's better to state it all in wiki? Several heaps, communication with closed part only with virtual functions and the like.

 

Yes, please, and I could draw pretty pictures if you like.

 

Now I think about reverting back to greebo's idea. I added debug output in saving process to know what parts of code write the most. Only 300Kb is written before idSaveGame::Close. It means that 99.6% of save consists of object data. All the objects call methods of idSaveGame and idRestoreGame which are opened. So I just need to rewrite their methods :laugh:.

 

Most of the data will be stored to buffer inside idSaveGame. The closed part of code will write their data uncompressed straight to the file. After everything is done, the cache would be compressed and dumped into save. No OS-specific things, no hacks, no heap mess. Great!

 

 

Sounds like a heck of a hack to me (that was a compliment!) :)

 

The only thing I cannot understand now is what eats time in save/load process. Writing 86Mb to hard drive is much faster that the whole saving process if done in one operation. I hope that when I rewrite SaveGame/RestoreGame everything will become clearer...

 

Calling many system calls instead of one (like writing to the file in 4 byte steps instead of one) could f.i. cause it. Who knows what the system is doing if you feed it 86 Mbyte in small bits?

 

Edit: One thing we might do in the future is also not to save so much data. Some of the data can be recovered quite easily, or can be read from the spawnargs. Or instead of caching a few strings for each entity, just cache an index into a string list. Granted, the compression will deal with this, but usually not as good as when you avoid the data in the first place.

"The reasonable man adapts himself to the world; the unreasonable one persists in trying to adapt the world to himself. Therefore, all progress depends on the unreasonable man." -- George Bernard Shaw (1856 - 1950)

 

"Remember: If the game lets you do it, it's not cheating." -- Xarax

Link to comment
Share on other sites

Well, but what happens if you do something like "new idRenderModel()" in your code (gamex86) and then let D3 clean it u (via deallocation) wouldn't that exactly trigger the bug (I get f.i. a crash when closing D3 whenever I used one of my special objects).

 

 

You cannot do it. idRenderModel implementation is closed. Therefore you can only call its virtual methods. Constructor cannot be virtual hence you cannot construct it (linker error). To create closed-source object you have to call virtual factory function of another closed-source object (like OpenFileRead). And it will create the object you need inside Doom3.exe. Destructor is virtual. so it will call deallocation in Doom3.exe module as well.

 

As you see, all the objects with closed source are black boxes for us. We cannot construct them, we can only call virtual methods for pointers we get.

 

The problem appears when we pass objects between modules for which we have source. For example idLib containers (idStr, idList etc). We must be sure that no allocation/deallocation can cross the module boundary.

 

 

By the way, I noticed that idDict::Clear and idDict::Set takes a lot of CPU time on loading. Maybe I'll write some internal saving/loading for them.

 

 

 

Just a note, if you're using IDA or similar:

 

 

That's not for me. I prefer ollydbg to IDA since the first loading :rolleyes:

Link to comment
Share on other sites

An important question:

 

I think there'll be a setting for whether to compress save or not (since uncompressed files are less prone to errors and are easier to study). Should I make game loading compatible with both variants?

 

On SaveGame the saves compression is regulated by some options. On RestoreGame the save should be loaded regardless of any settings. Am I right?

 

If so, I'd add 'compressed' flag :rolleyes: Now BuildNumber and CodeRevision are not compressed since they are important.

Link to comment
Share on other sites

An important question:

 

I think there'll be a setting for whether to compress save or not (since uncompressed files are less prone to errors and are easier to study). Should I make game loading compatible with both variants?

 

On SaveGame the saves compression is regulated by some options. On RestoreGame the save should be loaded regardless of any settings. Am I right?

 

If so, I'd add 'compressed' flag :rolleyes: Now BuildNumber and CodeRevision are not compressed since they are important.

Yes, if save compression can be switched on and off via CVARs that would be ideal. A simple boolean flag in the savegame should do the trick. This also helps people to check whether savegame issues will be due to compression or else.

Link to comment
Share on other sites

Looks quite good so far - very nice work, you even included the Number of the Beast. ;)

 

First few tests with small missions have been positive, I'll have a go at saving a larger mission now.

 

edit: ran into the first crash right after posting this. In NHAT Misison 2 (ANOOTT) I receive a crash in memcpy (debug build).

Link to comment
Share on other sites

Is it possible that the RawVector is running out of memory? When the crash occurs (in idSaveGame::Write()) the cache object holds a NULL m_Pointer member. m_Size is 67108915 at this point, and m_Capacity is 134217728. Maybe it failed to re-allocate the buffer and a "flush" operation is needed (compressing and writing what it holds so far to disk).

Link to comment
Share on other sites

Is it possible that the RawVector is running out of memory? When the crash occurs (in idSaveGame::Write()) the cache object holds a NULL m_Pointer member. m_Size is 67108915 at this point, and m_Capacity is 134217728. Maybe it failed to re-allocate the buffer and a "flush" operation is needed (compressing and writing what it holds so far to disk).

 

That's possible...

 

 

There is no flushing now. I decided not to mess with zlib internals. So all the data is written to that buffer and compressed afterwards. And the previous version (with pipe) behaved the same way.

 

Maybe some realloc error.

 

 

 

 

 

Link to comment
Share on other sites

I can confirm that realloc() is failing to return the amount of requested memory (128 MB, I think), the assertion I put in CRawVector::resize() fires due to a NULL pointer.

 

How big is your swap file, how much RAM do you have and how much memory is used by Doom3 process?

 

 

It sounds a shame that it failed to allocate 128MB :laugh: I can't believe it...

 

Can you please tell me exact steps... exact name of mission maybe?... HNAT is No Honor Among Thieves? Mission 2?

 

 

Link to comment
Share on other sites

Yup, this is No Honor Among Thieves, Mission 2. I just started the mission (via "New Mission") and switched compression off to create an uncompressed game, then hit Quicksave. The debugger fired immediately at that first save.

 

Doom3.exe is using 280 MB as private working set, and 1529 MB as maximum working set. Maybe it's hitting the 2 GB boundary for x86 applications when reallocating memory the n-th time. At any rate we probably can't really help it but capping the cache size. I've got plenty of free ram in my machine, that's not the problem.

 

Independently of this problem, one easy measure is of course to strip all the editor_* spawnargs from the entities at spawntime, these actually make up a considerable portion of string data written to the savegame.

Link to comment
Share on other sites

It can't allocate contiguous region in virtual address space. Maybe make buffer non-contigous?... Or I need to code buffered compressor stream? And the same decompressor stream. Or maybe it's better to include some C++ zlib wrapper (like gzstream)? Or even better: I've just found that there are boost zlib streams. But in case of streamed compression the file structure would become more complex. The compressed data will be interleaved with non-compressed data (which is closed-source).

 

Independently of this problem, one easy measure is of course to strip all the editor_* spawnargs from the entities at spawntime, these actually make up a considerable portion of string data written to the savegame.

 

Hmm... What's the problem here?

Link to comment
Share on other sites

Hmm... What's the problem here?

The TDM entityDefs hold a lot of editor_* spawnargs to assist mappers with descriptions and instructions. These are a lot of text and are most probably not needed for the game - it'd be an easy measure to have them removed at entity spawn time, such that they won't get written to the savegame file unnecessarily. I'll take care of it, so no need for you to worry about that.

Link to comment
Share on other sites

The TDM entityDefs hold a lot of editor_* spawnargs to assist mappers with descriptions and instructions. These are a lot of text and are most probably not needed for the game - it'd be an easy measure to have them removed at entity spawn time, such that they won't get written to the savegame file unnecessarily. I'll take care of it, so no need for you to worry about that.

 

I have added a note to the bug report that "comment" is similiarily superflous (although not an as-easy target as "editor_" :)

 

In the same area, looking into a savegame, I wonder two things:

 

* it seems *all* spawnargs are saved, even when there is an internal variable and the spawnarg is never read again (f.i. "nomoss"). I don't think this is easily fixable, tho, because how could the savegame code know that "nomoss" is a never-accessed spawnarg because there is an internal member variable used instead? (And script code could always fetch the spawnarg). The only solution to this I can think of is using more inheritance in spawnargs, so that the base class has "nomoss" and the individual entities inherit it. However, this is something we already have done a lot.

* It seems that for an func_static, the save code also saves *all* the entity data, even tho most of it is never used for a func_static. It might be worthwhile to add a few conditionals to idEntity::Save() that skips saving some data when the entity is of a special kind (light, static etc.) I have already done so f.i. for the LOD data - instead of always writing X strings, I only write a boolean and skip the strings if this entity doesn't have LOD. Saves a few dozend bytes per entity, which can be significant.

 

Of course, save game compression will hide such data waste quite a bit, but if we can easily avoid data getting saved in the first place, this should be a win regardless (less data to compress => faster).

"The reasonable man adapts himself to the world; the unreasonable one persists in trying to adapt the world to himself. Therefore, all progress depends on the unreasonable man." -- George Bernard Shaw (1856 - 1950)

 

"Remember: If the game lets you do it, it's not cheating." -- Xarax

Link to comment
Share on other sites

Of course, save game compression will hide such data waste quite a bit, but if we can easily avoid data getting saved in the first place, this should be a win regardless (less data to compress => faster).

Just cut that leEEEengthy editor_* args. Even without compression saving/loading will become mush faster and will take much less space. I think that hunting duplicates is superfluous. Even compression may turn out to be unnecessary after editor data is stripped.

 

 

By the way, did you consider opening read-only SVN access to public? Or it it is too dangerous? Some crazy people may try SVN builds... And sending patches will be easier. Though I think not many guys want use SVN builds and send patches...

Link to comment
Share on other sites

Just cut that leEEEengthy editor_* args. Even without compression saving/loading will become mush faster and will take much less space. I think that hunting duplicates is superfluous. Even compression may turn out to be unnecessary after editor data is stripped.

 

The editor data is certainly big, but if you save 1000 entities, the system also saves 1000 X (0|1) for each boolean, which can be compressed a lot!

 

By the way, did you consider opening read-only SVN access to public? Or it it is too dangerous? Some crazy people may try SVN builds... And sending patches will be easier. Though I think not many guys want use SVN builds and send patches...

 

Yeah, we are working on this. We could also give you SVN access, but greebo would have to decide this.

"The reasonable man adapts himself to the world; the unreasonable one persists in trying to adapt the world to himself. Therefore, all progress depends on the unreasonable man." -- George Bernard Shaw (1856 - 1950)

 

"Remember: If the game lets you do it, it's not cheating." -- Xarax

Link to comment
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.

Guest
Reply to this topic...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.


  • Recent Status Updates

    • Petike the Taffer  »  DeTeEff

      I've updated the articles for your FMs and your author category at the wiki. Your newer nickname (DeTeEff) now comes first, and the one in parentheses is your older nickname (Fieldmedic). Just to avoid confusing people who played your FMs years ago and remember your older nickname. I've added a wiki article for your latest FM, Who Watches the Watcher?, as part of my current updating efforts. Unless I overlooked something, you have five different FMs so far.
      · 0 replies
    • Petike the Taffer

      I've finally managed to log in to The Dark Mod Wiki. I'm back in the saddle and before the holidays start in full, I'll be adding a few new FM articles and doing other updates. Written in Stone is already done.
      · 4 replies
    • nbohr1more

      TDM 15th Anniversary Contest is now active! Please declare your participation: https://forums.thedarkmod.com/index.php?/topic/22413-the-dark-mod-15th-anniversary-contest-entry-thread/
       
      · 0 replies
    • JackFarmer

      @TheUnbeholden
      You cannot receive PMs. Could you please be so kind and check your mailbox if it is full (or maybe you switched off the function)?
      · 1 reply
    • OrbWeaver

      I like the new frob highlight but it would nice if it was less "flickery" while moving over objects (especially barred metal doors).
      · 4 replies
×
×
  • Create New...