TDM Engine Development Page

[SteveL]

Ah I see what's upset it. The fixed copy depth doesn't always use the q prefix for its openGL calls. I'll put that in for now for the sake of consistency. Even if we adopt your glew approach in 2.04, we don't want 0.01% of opengl calls using a different interface in 2.03.

[revelator]

Might have been caused by me since the engine i based this off was allready based on GLEW hence the lack of q before the gl calls,

in a fork not yet using GLEW it should have the q prefix. Im actually amazed that it worked without it.

Edited November 18, 2014 by revelator

[SteveL]

small function you can implement at once here, fixes videoram detection.

 

/*
================
Sys_GetVideoRam
returns in megabytes
This function works but returned negative sizes.
Fixed now.
================
...
if (hr == S_OK && spInstance) {
 // Get properties from the object
 CComVariant varSize;
 hr = spInstance->Get(CComBSTR(_T("AdapterRAM")), 0, &varSize, 0, 0);
 if (hr == S_OK) {
  retSize = varSize.intVal / (1024 * 1024);
  if (retSize == 0) {
   retSize = 64;
  }
 }
}


if (retSize < 0) {
 return retSize = -retSize;
}
return abs(retSize);
#endif
}

 

This makes auto setting system spec viable again before this fix it still worked but returned negative numbers so if your card had say 2 GB memory it would return -2048 instead of 2048 causing the detection to fail.

 

Just did a quick check to see why this isn't causing us problems with texture downsizing. It seems it's only used if there's no config.spec file, and TDM provides an empty config.spec file. If this has been broken since doom3, I suspect that file must always have been distributed with it!

 

I just checked what happens on my system using the original function, and I get a negative result too, but it's looks like a harder problem is going on. I have 4gb ram, which is pushing the limit for the unsigned int retval, and the return value for this func is a signed int. So I can see why the positive retval inside this function might give a negative result. And when I put in the minimal part of your fix -- making retSize a signed int and wrapping the return in abs() -- it returns 2048 not 4096, presumably because of that bad conversion from maxed-out unsigned int to signed int is turning my 4gb into -2gb. So still not the right result. Did I miss an essential bit?

[SteveL]

I tested and committed the qgl fix for CopyDepthbuffer though I haven't had time to do new binaries as I have to start work. But Tels, you should be able to compile the engine from svn (latest) again, although that patch I gave you will still screw up 2.02 svn unless you replace the gl* calls with qgl*

[AluminumHaste]

If that's the case, using something larger will be required.

A Titan comes with 6GB of ram and the new GTX980M Mobile GPUs have 8GB of ram.

What would we need to hold 8GB in a variable? EDIT: int64?

[SteveL]

That would do it if it's portable enough, but I'd suggest for now we leave the function dormant. We'd have to recalibrate the entire thing to make it useful. The current set up only does anything if you have less than 256mb!

[revelator]

Mmm could not reply earlier today because the forums seemed to be down, but yeah int pointers would need to be changed to 64 bit size on cards with more memory than 4 gb atleast also the signed vs unsigned comment is valid and needs checking. My small fix only changes the returned negative values into positive ones and does not take into account the growing heaps of memory on modern cards.

Also since 64 bit OS are becomming more and more the norm we might need to think about porting the code to 64 bit (dhewm3 patches should prove valuable there).

[revelator]

Come to think of it this will probably also affect other parts of the engine if say we use huge high definition textures we would run into problems with the pk4 system not being able to hold the data (actually allready ran into that problem with the HD textures for Doom3) causing the pk4's to not work even though you can open them in 7z or other zip managers. Probably also other places that would begin to experience problems.

[nbohr1more]

Yes, if you wouldn't mind doing the Dhewm merges into your github branch this can stimulate the process of getting that rolling again.

Serpentine had a (mostly) working merge but was having a little trouble with his Intel GPU as I recall (another Intel victim, ha!).

 

Having a working Dhewm-merge build would go a long way towards instilling confidence in the process.

 

Otherwise, I'll probably start on my own merge... eventually. (I'm not sure where Taaaki has left of but he may chime-in with his own Dhewm comments too...)

[revelator]

Sure anything you dont want from dhewm3 ? changing to SDL would break my GLEW port so im not sure how i should react to that part, besides i prefer using native OpenGL and no other part of the engine

really needs anything special since all the libraries used in it are allready crossplatform. Changing to use cmake would be a great leap though and we get rid of a lot of deprecated stuff in the process .

Nothing agains scons its a great build system but it can be a little clunky at times and cmake works for all OS so its really a no brainer.

All devs who want to have access to my git repository to make direct changes are welcome to chime in the more of us the faster it goes.

[nbohr1more]

AFAIK, we were keen to gain EFX from that branch too so everyone could have good spacial sound features.

If you can accomplish that without going fully SDL, then all the better (less culture shock, etc).

 

We're also rather keen on keep the gamex86.dll if possible (good for prototyping, smaller compile times, etc).

Edited November 18, 2014 by nbohr1more

[Tels]

That would do it if it's portable enough, but I'd suggest for now we leave the function dormant. We'd have to recalibrate the entire thing to make it useful. The current set up only does anything if you have less than 256mb!

 

But -2Gb is less than 256Mb

 

I'd say fix it right, and use a long (int64) so it can hold more then 2 Gbyte. Also, code like this makes me cringe:

 

if (retSize < 0) {
 return retSize = -retSize;
}
return abs(retSize);

 

If the variable is less then 0, you make it positive, fair enough. But why "abs()" in the other case? It is already positive... Plus, there a wholly needless if there. A much simpler approach is:

 

return abs(retSize); 

 

And be done with it

 

Edit: If a size of >4Gbyte (or 3, for 32 bit processed) is causing problems in other parts of the engine, the right way is to use 64bit to query the value, then limit it to a safe value (and output both on the console).

Edited November 18, 2014 by Tels

[Tels]

Sure anything you dont want from dhewm3 ? changing to SDL would break my GLEW port so im not sure how i should react to that part, besides i prefer using native OpenGL and no other part of the engine

really needs anything special since all the libraries used in it are allready crossplatform. Changing to use cmake would be a great leap though and we get rid of a lot of deprecated stuff in the process .

Nothing agains scons its a great build system but it can be a little clunky at times and cmake works for all OS so its really a no brainer.

All devs who want to have access to my git repository to make direct changes are welcome to chime in the more of us the faster it goes.

 

A change away from scons would be real good. But apart from a few fruitles tries, we have nothing to show for yet.

[Tels]

git clone github address

 

Ok, and what is the address?

[nbohr1more]

Ok, and what is the address?

 

https://github.com/revelator/The-Darkmod-Experimental

[Tels]

https://github.com/r...od-Experimental

 

Does not compile:

 

 

 

scons: building `game/precompiled_game.h.gch' because it doesn't exist
g++ -o game/precompiled_game.h.gch -x c++-header -c -pipe -Wall -Wno-unknown-pragmas -fmessage-length=0 -fpermissive -fvisibility=hidden -m32 -O3 -march=pentium3 -ffast-math -fno-unsafe-math-optimizations -fomit-frame-pointer -fno-strict-aliasing -Wno-deprecated -Winvalid-pch -fPIC -DGAME_DLL -Igame -Ibuild/release/game/sys/scons -Isys/scons -Iinclude -Iinclude/zlib -Iinclude/minizip -Iinclude/libjpeg -Iinclude/devil -I. game/precompiled_game.h
In file included from game/../idlib/precompiled.h:109,
			 from game/precompiled_game.h:28:
game/../idlib/../sys/win32/win_qgl.h:41:19: error: glxew.h: No such file or directory
scons: building `build/release/game/game/randomizer/userintf.os' because it doesn't exist
g++ -o build/release/game/game/randomizer/userintf.os -c -fPIC -pipe -Wall -Wno-unknown-pragmas -fmessage-length=0 -fpermissive -fvisibility=hidden -m32 -O3 -march=pentium3 -ffast-math -fno-unsafe-math-optimizations -fomit-frame-pointer -fno-strict-aliasing -Wno-deprecated -Winvalid-pch -fPIC -DGAME_DLL -Igame -Ibuild/release/game/sys/scons -Isys/scons -Iinclude -Iinclude/zlib -Iinclude/minizip -Iinclude/libjpeg -Iinclude/devil -I. game/randomizer/userintf.cpp
scons: building `build/release/game/idlib/Base64.os' because it doesn't exist
g++ -o build/release/game/idlib/Base64.os -c -fPIC -pipe -Wall -Wno-unknown-pragmas -fmessage-length=0 -fpermissive -fvisibility=hidden -m32 -O3 -march=pentium3 -ffast-math -fno-unsafe-math-optimizations -fomit-frame-pointer -DXTHREADS -fno-strict-aliasing -Iidlib -Ibuild/release/game/sys/scons -Isys/scons -Iinclude -Iinclude/zlib -Iinclude/minizip -Iinclude/libjpeg -Iinclude/devil -I. idlib/Base64.cpp
In file included from idlib/precompiled.h:109,
			 from idlib/Base64.cpp:1:
idlib/../sys/win32/win_qgl.h:41:19: error: glxew.h: No such file or directory
scons: *** [game/precompiled_game.h.gch] Error 1
scons: *** [build/release/game/idlib/Base64.os] Error 1
scons: building terminated because of errors.

 

 

[SteveL]

All devs who want to have access to my git repository to make direct changes are welcome to chime in the more of us the faster it goes.

What's the plan? I mean is there a list of fixes that you plan to implement? If we find problems along the way do we stop to fix them, or do you prefer to wait till the end of the code integration to sift the lot out?

 

I'll gladly give it a test run with some TDM maps when I'm done with 2.03 issues. I'm only concerned that with so many fixes to problems going in without individual testing -- problems we're not sure we have like the partial VBO-write fix, when it might be bus bandwidth that's our bottleneck -- that some really valuable work might go to waste because it's mixed in with stuff that turns out to be hard to make play nice with our maps and assets.

[revelator]

I'd say fix it right, and use a long (int64) so it can hold more then 2 Gbyte. Also, code like this makes me cringe:

 

if (retSize < 0) {

return retSize = -retSize;

}

return abs(retSize);

 

 

Also made me cringe but at the time i could not get it working correctly with just abs :S not sure if its a bug with msvc's libraries but abs alone output wildly incorrect values for some reason.

my 2 gb card would report 800 mb so i resorted to using the value = -value; and keeping the abs as a fallback. Could probably just remove the abs completely instead.

 

Does not compile:

i suspect as much you need the glew developer package for your distro, or a precompiled linux version, should probably make a seperate linux specific version of qgl.h also,

looks rather strange to have something with win in the name as part of a linux compile hehe.

 

EFX should be easy its part of openal soft and a replacement for the old dsound based EAX backend no need for SDL there

[revelator]

Well its not really up to me but if i where to go about it i would focus on first routing out known bugs to vanillas code, luckily 90% of these have allready been fixed by projects such as dhewm3.

Next collect devs to hear what they would like to see implemented and discussing which part we should focus on first. This part could be finding optimization spots as well as effects but focus should be on keeping the engine in a running condition on hardware that may go some years back, after that we can focus on eyecandy. Some teams could then focus on game logic, while us rendering nerds could focus on

new effects, and the allround wizards could help a bit on everything including optimization.

 

Im mostly a render nerd but i have enough common knowledge to also do work on other parts of the engine (not so much game logic) but if i have an idea i will usually run it by the people who do before i break stuff.

[SteveL]

Did you strip out a lot of the unused crud? Support for pre-ARB2 cards etc? Working round that without breaking it is a headache but I'd never want to take a month off to try removing it from the trunk and anyway it'd be impossible to recruit testers to get a wide variety of platforms when the effect of the experiment is...nothing. An alternative engine build could be an opportunity to do that.

 

While I'm speculating wildly, has anyone ever split the renderer front end and back end across two processors, like it was originally designed to be done? Has the problem of 2 processes writing to the same gl context been solved by new tech? Or if not, perhaps by using gpu skinning so there's vastly less for the front end to write to the gpu, and so moving what's left to the back end?

[nbohr1more]

I think I know how to kill the pre-ARB cruft and there are a few branches that confirm my suspicions.

[Tels]

Does not compile:

 

 

 

scons: building `game/precompiled_game.h.gch' because it doesn't exist
g++ -o game/precompiled_game.h.gch -x c++-header -c -pipe -Wall -Wno-unknown-pragmas -fmessage-length=0 -fpermissive -fvisibility=hidden -m32 -O3 -march=pentium3 -ffast-math -fno-unsafe-math-optimizations -fomit-frame-pointer -fno-strict-aliasing -Wno-deprecated -Winvalid-pch -fPIC -DGAME_DLL -Igame -Ibuild/release/game/sys/scons -Isys/scons -Iinclude -Iinclude/zlib -Iinclude/minizip -Iinclude/libjpeg -Iinclude/devil -I. game/precompiled_game.h
In file included from game/../idlib/precompiled.h:109,
			 from game/precompiled_game.h:28:
game/../idlib/../sys/win32/win_qgl.h:41:19: error: glxew.h: No such file or directory
scons: building `build/release/game/game/randomizer/userintf.os' because it doesn't exist
g++ -o build/release/game/game/randomizer/userintf.os -c -fPIC -pipe -Wall -Wno-unknown-pragmas -fmessage-length=0 -fpermissive -fvisibility=hidden -m32 -O3 -march=pentium3 -ffast-math -fno-unsafe-math-optimizations -fomit-frame-pointer -fno-strict-aliasing -Wno-deprecated -Winvalid-pch -fPIC -DGAME_DLL -Igame -Ibuild/release/game/sys/scons -Isys/scons -Iinclude -Iinclude/zlib -Iinclude/minizip -Iinclude/libjpeg -Iinclude/devil -I. game/randomizer/userintf.cpp
scons: building `build/release/game/idlib/Base64.os' because it doesn't exist
g++ -o build/release/game/idlib/Base64.os -c -fPIC -pipe -Wall -Wno-unknown-pragmas -fmessage-length=0 -fpermissive -fvisibility=hidden -m32 -O3 -march=pentium3 -ffast-math -fno-unsafe-math-optimizations -fomit-frame-pointer -DXTHREADS -fno-strict-aliasing -Iidlib -Ibuild/release/game/sys/scons -Isys/scons -Iinclude -Iinclude/zlib -Iinclude/minizip -Iinclude/libjpeg -Iinclude/devil -I. idlib/Base64.cpp
In file included from idlib/precompiled.h:109,
			 from idlib/Base64.cpp:1:
idlib/../sys/win32/win_qgl.h:41:19: error: glxew.h: No such file or directory
scons: *** [game/precompiled_game.h.gch] Error 1
scons: *** [build/release/game/idlib/Base64.os] Error 1
scons: building terminated because of errors.

 

 

 

Solved this particular issue with that patch:

 

te@te:~/src/tdm_experimental/The-Darkmod-Experimental/src$ git diff sys/win32/win_qgl.h
diff --git a/src/sys/win32/win_qgl.h b/src/sys/win32/win_qgl.h
index 55875a5..8f078d4 100644
--- a/src/sys/win32/win_qgl.h
+++ b/src/sys/win32/win_qgl.h
@@ -38,7 +38,8 @@ If you have questions concerning this license or the applicable additional terms
#include "glew.h"
#include "wglew.h"			 // windows OpenGL extensions
#elif defined( __linux__ )
-#include "glxew.h"
+#include "glew/glew.h"
+#include "glew/glxew.h"
#endif

//

 

We probably should start a new thread for this.

[nbohr1more]

Done:

 

http://forums.thedarkmod.com/topic/16692-revelators-tdm-branch/

[nbohr1more]

Another tangent but I like collecting this stuff...

 

Sebbbi over at beyond3d once told of his fully GPU stencil shadows using DX8 hardware but he

never explained how that was possible.

 

He must've used something analogous to this...

 

http://aras-p.info/texts/revext.html

 

(I get about 2000fps on an integrated intel GPU at work for this demo... lol )

[revelator]

There are a few crossbreeds between directx and opengl so its not unheard off also pretty much any rendering code can be moved GPU side if your gfx card supports the OpenGL version needed, even bumpmapping (tesselation).