stgatilov

The whole TDM relies on the fact that equal class declarations give equal virtual function pointer tables. Notice that there are no symbols/addresses declared or passed related to idFileSystem methods, for example. But we call them on both windows and linux completely without any linking or checking. Now consider the example. If you add your own virtual function into idFileSystem class declaration (into the middle of it), Doom will crash badly. Because the VFPT-s become different in Doom3.exe and gamex86.dll. Compiler/linker won't give you any error of warning in this case, so compiler indeed hopes that declarations are the same. I suppose that it is guaranteed by C++ standard or at least GCC and MSVC compilers. I'll look it up.

It is impossible. There is some data which must be saved but is unaccessible. That's why idSaveGame calls closed source methods like this: if ( ui->WriteToSaveGame( file ) == false ) { WriteToSaveGame is method inside Doom3 and it won't accept our classes - only idFile

Just tried doing as I proposed. savegame.ReadBuildNumber() calls readInt function of my class, so it works. Should I start implementing idFile this way? Maybe it is better to separate compression from the idFile implementation? Create for example idFile_Stream class which will call Read/Write methods of its CByteStream member. And create CByteStream implementation which actually compresses/decompresses the stream. To make further idFile implementations easier...

Not so easy. You cannot derive CFileCompressed from idFile. You won't be able to construct such an object because compiler will put a call to constructor of base class like this: //C++ must call constructor of base class: CFileCompressed::CFileCompressed() : idFile() {} Linker has no way to find where idFile constructor is and will output error with unresolved external idFile::idFile. Maybe this will be OK: class CFileCompressed { //Here are declarations of exactly the same virtual functions! } //Virtual function pointer tables should have the same format idFile *newfile = (idFile*)(new CFileCompressed()); newfile->Write(blablabla); //will be redirected to our code because of virtual function redirection

Sorry, this was not intended to get into patch. This block is from working copy (I was investigating the type of MakeTemporaryFile result). I copied files manually, so this block got into zip. Of course it should be deleted. Well, I tried this way but got linker error. All the methods of idFile are unresolved symbols if you try to create an object of class derived from idFile. So I decided that it was not possible. Also I didn't wanted to implement all the Read* and Write* methods of idFile. But now I think linker error can be resolved (with some hacks maybe). Something like create class with exactly the same virtual functions and C-cast its pointer to idFile*. I'll dig in this direction. Only in theory=) I hope that ordinary file in /dev/shm directory will be enough. It resides in memory when it is possible and it can have arbitrary size. So we

Here is the modified/added files. Delete renderpipe* files. http://www.mediafire.com/?3ceq80wxt021335 I hoped that hg and svn diffs were compatible. The patch looked like the usual diff...

Here is the patch... I'm not permitted to attach ".patch", so I renamed it to ".txt". savegame_compression_patch_v1.txt

Yes, it is currently working (at least on my windows XP machine ) Implementation is very big and dirty. It creates named pipe like CRenderPipe in windows. There are two modes: 1. For small file without threads. Works exactly like CRenderPipe. It assumes that all the data fits the pipe buffer. So in case of writing: ID writes to the pipe everything it wants (e.g. screenshot), then I read from the pipe everything it has written. 2. For big file with threads. I don't think windows will give me 2 x 100Mb pipe buffer for savegame compression. So likely file won't fit the buffer. That's why I have to create another thread. In case of writing this thread constantly tries to read from the pipe. After all the writes are completed (end of SaveGame for example) and file is closed this thread is joined. In both cases the contests of pipe are stored to std::vector. After writing the contests of this vector can be read. Reading works the similar way. Since the whole save/load with compression is working now, I'll soon post a patch. As you see, the way it is working now is very ugly. The problem is in limited size of pipe buffer. On linux /dev/shm can hold data of any size, so there is no need to create any threads. The best solution is to make idFile_Memory working. I think pipes and /dev/shm will become unnecessary in this case. The other option is to find /dev/shm equivalent for windows and use it. Then the thread ugliness will go away.

I have already posted here that I'm trying to implement savegame compression in TDM. I have implemented the class to capture ID file input/output by using named pipes but now I feel that it was not necessary. I've noticed the SourceHook code. So you have already hacked Doom3=) Seems that you can replace or add code to closed Doom3 engine functions. Is it possible to get access to idFile_Memory constructor or the idFileSystem member which creates it (if it exists)? If it is possible, then we can simply use ID tools to store data to memory (so CRenderPipe would be unnecessary and implementing compression would be easy).

Yes, I is the way to handle lightgem on linux and I hope it'll be the way to implement savegame compression. Does anyone knows windows equivalent of /dev/shm? It must be possible to open the file by filename with stdio functions. MSDN says that if FILE_ATTRIBUTE_TEMPORARY is set on CreateFile, the file is never flushed to hard drive. Maybe try it?

The problem with savegame compression implementation it not the compression itself. Zip compression of memory buffer is done with ONE line of code. The problem is how to intercept data before it goes to hard drive. I have almost finished this damn piece of code for windows (and I hope for linux it'll be easier). Choosing different compression is easy. Any compression library that can compress from memory buffer to another will be suitable. But remember that speed may become a bottleneck if you choose smth like 7z ultra . Default zip compression is good because it is fast. I've seen bzip2 compresses better that zip, but considerably slower.

Wow... I've seen exactly this candle from the training mission flickering! So this is not linux problem, this is some change from update 1.03.

I measured the time from pressing "+" (quicksave key) to the moment screen unfreezes. So this is not just writing, but the whole saving process. This text is not code comments. I think it is something like description of objects from DarkRadiant. I can try to understand what is it, but TDM team knows it much better anyway. It may be difficult and tedious to eliminate it. Though I hope it is not difficult... But compression is completely orthogonal to saving process anyway:). P.S. Should I post to "I want to help" forum?...

Yeah!!! I finally made it! The savegame files are written compressed now (reading is not done though ). I use CRenderPipe to create a pipe. This pipe is passed to idSaveGame for writing. I also create another thread which constantly reads from this pipe and stacks the data into std::vector. After all the game saving is done, I compress the resulting std::vector with zlib's compress and write the result to idFile *f parameter which is the read destination file. So conceptually it is: dump the whole savegame to memory, compress that memory, write it to file. The results are the following: Uncompressed save: 87Mb size, 7 seconds time. Compressed save: 4.4 Mb size, 2.5 seconds time. So writing compressed savefile is MUCH faster than uncompressed. Even if we compress the whole save file at once. The question is: what to do next? The way I use CRenderPipe now is weird I'm afraid that all this mess simply won't work on linux implementation. I think it's better to create some common named pipe class. Then use it in compressed saving/loading. CRenderPipe serves a specific purpose which is not enough for saving/loading.

I feel the same way. It is very arguable that compression will increase save/load time. Maybe it will decrease it. Right now I'm trying to use CRenderPipe for intercepting the savegame data...

I personally don't like 80Mb savegame files They contain a lot of redundant information. It is like a dump of gamestate... They can be packed easily. For example, after compressing with standard gzip savegames become about 20x smaller in size. Player has do it himself if he doesn't want to store Gigs of saves on hard drive. How about compressing/decompressing savefile "on the fly"? Just chain save/load file streams with some zlib stuff. I hope that it won't slow saving/loading too much. I decided to try implementing it myself. I downloaded new 1.03 code and looked how saving is done. The problem is: all the data of savegame is written to idFile *f which is passed to idGameLocal::SaveGame from closed piece of Doom3 source. The implementations of all idFile classes are also closed. So there is now way to insert compression inside idFile. So I switched to another idea: write savegame to memory buffer, then compress it and dump results to idFile parameter. Luckily there is idFile_Memory class which seems to do what I need: store everything that is written to it to memory buffer. The problem is: though I see idFile_Memory declaration I cannot create this object. I cannot reference constructors (undefined symbol). And I cannot find method for creating memory file in idFileSystem. So there is declaration of class idFile_Memory but no way to access it . After some googling I found this version of filesystem.h. It contains method OpenMemoryFile which creates idFile_Memory. This address is referenced here in ET:QW forums, so I guess it is from Quake4 SDK. Has developers fogrotten to open access to Memory files in Doom 3 SDK? Is there any simple way to use idFile_Memory? If the idFile_Memory is not usable, then the only proper way to implement on-the-fly compression is to create my own class with idFile interface. But in this case I'll have to implement all the bunch of functions idFile has for writing different types of data. I'll have to rewrite piece of closed Doom3 source... Maybe someone knows how to solve the problem? Or he has any suggestions about savefile compression? Why is it not implemented yet? Anyone except me wants to see automatic savegame compression?

It is a very important idea, it should be written down: Be sure to post some screenshots from a zombie mission on moddb the next year. The more frightening the better. Maybe it will force people to look TDM at least

This minor issue was already discussed here. But still... The savegames are very large and contain some strange unnecessary information (there is some text that looks like DarkRadiant docs... what is it doing in saves?!). Perhaps some data should not be written to the saves. But the easiest way to greatly reduce the size of saves is to zip them. I know that all the necessary zip libraries are already used in TDM code. The idea is quite simple: pack the whole FM directory when uninstalling mission and unpack it while installing if required. Maybe someone knows another ways of solving the problem? What are the plans of the TDM team about saves? (And of course this is not very important... )

This mission is completely nonlinear! The space is rather small and compact but it is all filled up. At the beginning it is even difficult to choose where to go There are many places that have no loot and no items. It makes the proper feeling that money isn't lying around in every corner. There is a room on the first floor of the hotel with two books. One of them has three pages. But the last page has a bent corner representing that it is not last. When I scroll to the next page the book closes automatically. What's wrong with this book? Scrolling through the last page causes the readable system to mess up a bit. After that I frob the another book in the room (which is OK) once - it opens. Then I hit frob again - it reopens without closing. After each frob while reading it the new instance is created and the old ones doesn't disappear.

As far as I understand, adding HDR will not affect anything axcept what you see on your screen. The gameplay will be the same. The lightgem value will be computed on CPU and will be the same with and without HDR. So the conditions in which the guard sees you does not depend on HDR because HDR won't affect internal calculations (AI and illumination of player are part of them). Ghostable mission will remain ghostable by same manipulations, and inghostable mission will remain inghostable. The following things should be handled carefully: The look of the lightgem will be changed. If you enter a darker room, then the overall brightness will decrease, and the autoexposure will make everything including the lightgem look more bright. Moreover, the look of the lightgem will change after simply turning your face to a dark corner without any movement. So the lightgem look will be messed in some way. I don't know for good of for bad . Anyway, I think that lightgem can be handled separately from the other scene, so its behaviour with autoexposure can be fixed to whatever you like... Another thing is the autoexposure principle itself. In the training mission there is a guide how to set gamma and brightness settings to get visually standard image. The thief spends a lot of time in shadows. And he shouldn't see anything in a very dark room. So the autoexposure should be introduced very carefully: the exposure adaptiveness should be bounded somehow. Because of the performance loss some players will have to play without HDR while others will turn HDR on. So the players will get conceptually different pictures in the same level (the brightness/contrast levels will be different) with and without HDR. (Please correct me if I'm wrong somewhere) P.S: The pictures are great. And the light is more important in thief game than in any other. I would be very pleased to see HDR in The Dark Mod.

This would greatly increase difficulty in this mission, but overally it is good. As for me guards' ability to relight torches is much worse that the potential ability to turn the lights on: you spend water arrows (limited supply) on torches but you can push the button infinitely many times. I use water arrows very seldom in TDM because I know that any guard passing by can easily restore the light. By the way, can any guard relight the torches or mapper can restrict a guard not to do so?

Well, I won't say that the mission is easy. I decided to go on apprentice (not fond of ghosting) in "KO them all" style. That torch guys are the real problem! I forgot to hid the bodies from them and soon I got to the situation: Two thoughts came to my mind: The place is guarded really well. So many guards want me dead.Where is my GRENADE LAUNCHER?!!I spent a lot of time hiding behind those crates. I realized how beatiful the night sky was! I noticed a bug in the my mission complete stats screen: "bodies found: 0" is a VERY wrong statement . They found bodies rather often and I am absolutely sure that I didn't quickload after that. By the way: The guards are clever enough to relight the torches. But I've never seen them turning the lamps on. It is not good that the guards patrol dark rooms after I turn off the light. Maybe the difficulty levels should be like this: Easy: killing is allowed + plenty of broadheads (just shoot them all).Medium: no killing restriction.Hard: no KOs + (optional) ghosting.There is no clear necessity in KO restriction in this mission (at least nothing was said about it in briefing). "no KO style" is just an act of "increased thief's coolness". Thanks for the mission! I really liked it. Maybe now when I know the way in I'll play through it on ghost.

Yo-ho-ho! TDM is in the best 100! Voted for it again. Now I know there is a Deus Ex mod... Already downloading Maybe I'll share my vote between Dark and Nameless.

Know what?! I changed Error to Warning in FreeCurrentPVS in pvs.cpp and rebuilt. I was almost sure that this won't give me anything as the problem is in Doom3.exe and not in gamex86.dll (or I don't understand the DLL system...). The crash is gone. The corrupted sessions are terminated more or less correctly (at least without a crash). Save/load works OK. I've finished the trapped mission from corrupted save and seen the mission completition statistics. I know that this kind of "fix" is just running away from some stability problem (memory leak or corruption maybe). But I prefer using the gamex86.dll with this hack.

I am interested in the arrow flight problem I read wikipedia about lift force and air drag. I tried to model the arrow in 2D affected by two forces: 1. gravity: F = g*m; directed downwards applied to center of masses 2. lift force: F = kl*V*V*sin(a)); directed orthogonal to arrow direction directed upwards related to arrow when a>0, downwards when a<0 applied to the tail of arrow a - signed difference between arrow angle and velocity angle. For example of the arrow flies horizontally but is oriented 5 degrees upwards, then a is equal to 5 degrees. There is another type of drag: 3. parasite drag (assuming turbulences): F = kp*V*V*(sin(a) + C); directed opposite to velocity direction the application point does not matter But parasite drag will just lower the arrow velocity. So it is not as interesting as the lift force is. I used kp=0 (no parasite drag at all). Maybe the drag for arrow in TDM should be lowered? It will eliminate the feeling of arrow falling like a cannonball in a far shot. There is no need for big air drag for arrows I think. Well, about the results 1. Even with very small coefficient kl the arrow direction is naturally aligned to velocity direction. This is because any difference in angles (variable a) creates momentum which returns arrow direction to normal state (near a=0). By the way, how is it implemented now in TDM? What makes the arrow direction to be the same as velocity direction? If the only force is gravity, then arrow orientation will never change at all. 3. All the other stats are the nearly the same with lift drag and without it. I wanted the arrow to save height for longer time. But now I'm not certain this effect appears in the real life. EDIT: I've launched the training mission and played a bit with archery. I feel that the arrow flies well and nothing should be changed Anyway, shooting too far is very rare event in TDM gameplay.