Revelator's TDM Branch

[nbohr1more]

Continued from Discussion here:

 

http://forums.thedarkmod.com/topic/15178-tdm-engine-development-page/page__view__findpost__p__358532

 

Github Repo:

 

https://github.com/revelator/The-Darkmod-Experimental

[AluminumHaste]

Love all the crazy activity! Gonna be a good Christmas <3

[Tels]

The branch fails to compile on Linux (after the include fix):

 

game/../idlib/../idlib/math/Curve.h: In member function ‘type idCurve_BSpline<type>::GetCurrentSecondDerivative(float) const [with type = idVec3]’:
game/../idlib/../idlib/math/Curve.h:1891: sorry, unimplemented: inlining failed in call to ‘float idCurve_BSpline<type>::Basis(int, int, float) const [with type = idVec3]’: recursive inlining
game/../idlib/../idlib/math/Curve.h:1902: sorry, unimplemented: called from here
game/../idlib/../idlib/math/Curve.h:1891: sorry, unimplemented: inlining failed in call to ‘float idCurve_BSpline<type>::Basis(int, int, float) const [with type = idVec3]’: recursive inlining
game/../idlib/../idlib/math/Curve.h:1907: sorry, unimplemented: called from here
game/../idlib/../idlib/math/Curve.h:1891: sorry, unimplemented: inlining failed in call to ‘float idCurve_BSpline<type>::Basis(int, int, float) const [with type = idVec3]’: recursive inlining
game/../idlib/../idlib/math/Curve.h:1902: sorry, unimplemented: called from here
game/../idlib/../idlib/math/Curve.h:1891: sorry, unimplemented: inlining failed in call to ‘float idCurve_BSpline<type>::Basis(int, int, float) const [with type = idVec3]’: recursive inlining

 

The last lines are then repeated a few hundred times.

[Bikerdude]

Let me make a test map, give me a minute.

 

wrong thread/pm.

[revelator]

recursive inlining sigh guess theres still some work to do on getting it to compile on linux.

[revelator]

And @teIs oki had no idea that you need to include the standard glew.h before the glxew.h so thanks ill commit it immediatly.

also notice that the glew lib is missing from the scons build so ill look at getting that one in as well.

 

As for removing the old render backends i was not sure if you wanted that but its pretty easy to do as long as you remember that there are a few things in some places you dont expect specifically written to support them that needs removing, i have completely removed all the code for the old backends in revelation so i could just copy of my changes from there. Also there are a few lines of code that needs to be

rewritten (just for overbrights).

[revelator]

Ok made a stupid mistake in inlining i had all inlines set to force inline thats why it failed on linux, fixed on trunk now.

added teIs fix for glew, also comitted.

[revelator]

Fixed another one that would break linux builds, since my GC interface was not prepared for linux yet i have made a few changes so that it only gets enabled if we are using MSVC.

Added replacements to public compiler defines in sys_public.h also added workarounds for some missing interfaces.

changed the PACKED macro into a unified version that works for all architectures. Had to make a few changes to sound_local.h as well as a result of this.

[revelator]

Btw anyone else noticed that there seems to be a bug in the script animation for the broadsword ? whenever i select that weapon the animation halts funny enough when it happens the soldiers cannot use there swords either so lol.

[SteveL]

We have swords? Heh not noticed that but I use my sword so rarely I've unbound the hotkey. Curious.

 

Don't let me encourage you to remove any back end yet, that's just a personal gripe on my part. I honestly don't know if it's wanted by anyone so let me ask around.

 

By the way, remember the compilation bug with vs2013 and lighting, where compiler optimizations would remove some parts of light interactions? greebo narrowed it down to one function R_CreateLightTris. Turning off inlining for that function cures it in my test map and in a real map that I ran through last night to compare the builds. Not tried to narrow it down further yet (if it's even possible) but I'd be interested to know if this fixes it in your test cases too. You wrap the function in this compiler switch:

#if defined(_MSC_VER) && _MSC_VER >= 1800 && !defined(DEBUG)
// greebo: switch off function inlining for this file in VC++ 2013 release builds
// Function inlining seems to cause lighting bugs (triangles are drawn very dark or black)
#pragma optimize("t", off)
#endif

With the block after the function turning the "t" option back on again. The pragma has to wrap an entire function, so we can't try putting it in the middle but I might try to stop individual functions being inlined to see whether I can narrow it down any further. Although I suspect that greebo will already have done that if it's possible.

[nbohr1more]

I posted this as an issue on the git but it's probably good to have this here too:

 

From the other thread:

 

 

While I'm speculating wildly, has anyone ever split the renderer front end and back end across two processors, like it was originally designed to be done? Has the problem of 2 processes writing to the same gl context been solved by new tech? Or if not, perhaps by using gpu skinning so there's vastly less for the front end to write to the gpu, and so moving what's left to the back end?

 

 

Here's John Carmack's solution:

 

 

John Carmack - This was explicitly to support dual processor systems. It worked well on my dev system, but it never seemed stable enough in broad use,

so we backed off from it. Interestingly, we only just found out last year why it was problematic (the same thing applied to Rage’s r_useSMP option,

which we had to disable on the PC) – on windows, OpenGL can only safely draw to a window that was created by the same thread. We created the window on the launch thread,

but then did all the rendering on a separate render thread. It would be nice if doing this just failed with a clear error,

but instead it works on some systems and randomly fails on others for no apparent reason.

The Doom 4 codebase now jumps through hoops to create the game window from the render thread and pump messages on it,

but the better solution, which I have implemented in another project under development, is to leave the rendering on the launch thread,

and run the game logic in the spawned thread.

 

 

So basically: it basically looks like "if you open a GL window with this thread, any draw commands must also come from this thread" that means that the

frontend render code can be moved to another (spawned) thread as long as no GL calls are done from it.

So I guess the gist is: Start GL, spawn game, spawn frontend, feed back to original GL thread to execute backend.

[Tels]

The patch for the Linux include was not properly applied, here is it again:

 

diff --git a/src/sys/win32/win_qgl.h b/src/sys/win32/win_qgl.h
index f26907a..a27f56e 100644
--- a/src/sys/win32/win_qgl.h
+++ b/src/sys/win32/win_qgl.h
@@ -38,8 +38,8 @@ If you have questions concerning this license or the applicable additional terms
#include "glew.h"
#include "wglew.h"			 // windows OpenGL extensions
#elif defined( __linux__ )
-#include "glew.h"
-#include "glxew.h"			 // linux OpenGL extensions
+#include "glew/glew.h"
+#include "glew/glxew.h"				// linux OpenGL extensions
#endif

//

 

Now it fails with:

 

 

 

In file included from game/../idlib/../idlib/Lib.h:163,
			 from game/../idlib/precompiled.h:112,
			 from game/precompiled_game.h:28:
game/../idlib/../idlib/math/Matrix.h: In member function ‘idMatX& idMatX::SwapRows(int, int)’:
game/../idlib/../idlib/math/Matrix.h:2438: error: ‘ALIGNPTR’ was not declared in this scope
game/../idlib/../idlib/math/Matrix.h:2438: error: ‘ALIGNPTR’ was not declared in this scope
game/../idlib/../idlib/math/Matrix.h:2438: error: declaration of ‘<typeprefixerror>ALIGNPTR’
game/../idlib/../idlib/math/Matrix.h:2438: error: conflicts with previous declaration ‘<typeprefixerror>ALIGNPTR’
In file included from game/../idlib/../idlib/Lib.h:172,
			 from game/../idlib/precompiled.h:112,
			 from game/precompiled_game.h:28:
game/../idlib/../idlib/math/Curve.h: In member function ‘float idCurve<type>::RombergIntegral(float, float, int) const’:
game/../idlib/../idlib/math/Curve.h:199: warning: there are no arguments to ‘ALIGNPTR’ that depend on a template parameter, so a declaration of ‘ALIGNPTR’ must be available
game/../idlib/../idlib/math/Curve.h:199: warning: there are no arguments to ‘ALIGNPTR’ that depend on a template parameter, so a declaration of ‘ALIGNPTR’ must be available
game/../idlib/../idlib/math/Curve.h:200: warning: there are no arguments to ‘ALIGNPTR’ that depend on a template parameter, so a declaration of ‘ALIGNPTR’ must be available
game/../idlib/../idlib/math/Curve.h:200: warning: there are no arguments to ‘ALIGNPTR’ that depend on a template parameter, so a declaration of ‘ALIGNPTR’ must be available
game/../idlib/../idlib/math/Curve.h: In member function ‘float idCurve<type>::GetTimeForLength(float, float) const’:

 

 

 

No idea why. Because ALIGNPTR is only defined for Win32, but used everywhere. Haven't tried to fix it, tho.

 

-----

Btw, how do I do an "svn update" with git? I couldn't get this to work,not even after reading the man page...

Edited November 19, 2014 by Tels

[nbohr1more]

 

-----

Btw, how do I do an "svn update" with git? I couldn't get this to work,not even after reading the man page...

 

I've been struggling with this too. I used svn2github to clone our SVN but I can't find a way to get newer SVN data

without either manually editing or making a whole new clone.

 

Gonna start diving into the CLI client:

 

http://rogerdudler.github.io/git-guide/

[Tels]

Apparently its "git pull". I still think the entire "distributed" repository stuff is overkill for our "we need a central repository anyway" approach.

 

But whatever...

[nbohr1more]

Ha!

 

The svn2github clone automatically retrieved the changes and I clicked Pull Request on my local repo and merged them in. Not bad. Now I'm working

a little cleaner.

[revelator]

error: declaration of ‘<typeprefixerror>ALIGNPTR’

 

agh yeah gcc hates it when a type or macro is defined in two places gotta have a look at the second macro and see if it matches the other one, if it does move it to sys_public.h and make it global,

That should take care of that, if not define it to something else and rename the instances where its used to the new name then hope i catched all of em.

[revelator]

Huh ok just checked and its was not even defined for the linux build because i forgot to apply it.

Fixed now comitting changes.

[Bikerdude]

Are you guys at the stage were you could show use some befroe and after screen shots..?

[nbohr1more]

Well, this is more about cleaning-up the framework and improving performance so an FPS counter shot would probably be the

extent of that.

 

That said, I'll be merging my customLight changes and I could do a demo map and collect a few screens for that.

 

You're probably waiting on the Penumbra Wedge shadows, which I too am eager to see in action but that'll need to wait until either

I "learn more about ARB" or SteveL is done with his other changes.

[revelator]

 

blackbog hollow on darkmod with GLEW backend and GC garbadge collector running, all effects on.

[Bikerdude]

@Revelator, can you put a 'before' shot next to it..?

[revelator]

sure hang on need to get the default engine back with the updater

 

P.s i removed a few of the old fallback backends like the arb and nv10 renderers as well as the hardcoded light and texcoord generation caches for cards not able to do shaders = nv10 and older.

I kept the r200 renderer because it still works pretty well even on my R270x though the shadows are non shader based with this renderer, i havent tested the nv20 render path as it doesn't apply to my card and would most likely crash with it, but if someone has a nvidia try out r_renderer nv20.

[revelator]

 

original engine shot.

[New Horizon]

Is that a performance increase from the changes or from the change of location in the screen shot? Sorry to be a nag, but would it be possible to get a side by side of the same location, just for easier comparison? It's hard to get a feel from the different locations.

[Tels]

No it compiles a bit, and then stops here:

 

 

 

scons: building `build/release/game/idlib/Heap.os' because it doesn't exist
g++ -o build/release/game/idlib/Heap.os -c -fPIC -pipe -Wall -Wno-unknown-pragmas -fmessage-length=0 -fpermissive -fvisibility=hidden -m32 -O3 -march=pentium3 -ffast-math -fno-unsafe-math-optimizations -fomit-frame-pointer -DXTHREADS -fno-strict-aliasing -Iidlib -Ibuild/release/game/sys/scons -Isys/scons -Iinclude -Iinclude/zlib -Iinclude/minizip -Iinclude/libjpeg -Iinclude/devil -I. idlib/Heap.cpp
idlib/Heap.cpp: In member function ‘void* idHeap::Allocate16(dword)’:
idlib/Heap.cpp:393: warning: suggest parentheses around ‘+’ in operand of ‘&’
idlib/Heap.cpp: In function ‘void* Mem_Alloc16(int)’:
idlib/Heap.cpp:1169: error: ‘_aligned_malloc’ was not declared in this scope
idlib/Heap.cpp: In function ‘void Mem_Free16(void*)’:
idlib/Heap.cpp:1190: error: ‘_aligned_free’ was not declared in this scope
include/boost/system/error_code.hpp: At global scope:
include/boost/system/error_code.hpp:221: warning: ‘boost::system::posix_category’ defined but not used
include/boost/system/error_code.hpp:222: warning: ‘boost::system::errno_ecat’ defined but not used
include/boost/system/error_code.hpp:223: warning: ‘boost::system::native_ecat’ defined but not used
scons: *** [build/release/game/idlib/Heap.os] Error 1
scons: building terminated because of errors.