A chance to learn the innards of a compiler

[revelator]

While still working on Doom3 i also pitch in on several other projects where i found i could do some good.

One of those projects i happened to chime in on was Open Watcom.

 

Open Watcom was in its days considered the best compiler avaliable and the price tag certainly showed that

At one point the comany was bought up by a company named sybase and updated for better support of windows (by that time it was win 3.11).

Sadly microsoft by the time of windows 95 caught up and started rolling out there MSVC compiler which was a chore to work with compared to the aging

Watcom compiler which btw was used to build Windows 95 in the first place, but Watcoms ide was rather hard to work with compared to MSVC's besides that it still produced

higher quality programs but in the end MSVC due to its ease of use won out and Watcom slowly bleed out. At some point sybase threw the blanket and opensourced the watcom compiler minus all the microsoft owned property and a new project was born named Open Watcom which had the goal of creating a good opensourced alternative to the microsoft versions. For a few years everything worked out and the compiler was updated considerably by using mingw's windows API it could once again create windows programs allbeit with some difficulty. A developer then created a new Windows API built specifically for the watcom from scratch unfortunatly it needed some housecleaning which i helped with while also learning a bit more about the inner workings of a compiler. Sadly the original site is mostly dead now but work is still ongoing and help is wanted and appreciated.

So if anyone here wants to help with the project here's an official invite .

 

Open Watcom as of now also supports 64 bit OS but still produces 32 bit executables due to the lack of a 64 bit API so thats one thing that could use some work.

Quake builds and works with it both dos winquake and OpenGL versions.

It has some pretty impressive optimizations compared to pretty much any other compilers like clang.

 

So if you have spare time and or would like to help on development visit here https://github.com/open-watcom/open-watcom-v2and become a member.

 

Btw Open Watcom supports other OS as well, like linux OS/2 haiku and it can still create programs for windows 9x and DOS.

[nbohr1more]

Heh, we hit a huge snafu with the MSVC 2013 compiler and had to rollback to 2010 because of precision issues

(MSVC 2013 only does 32-bit intermediate precision unless you specify higher precision with flags and there is a "flag zoo"

with some producing better performance than others depending on the code... MSVC 2010 defaults to 80-bit intermediate precision

which matches the typical float units on CPU's from P3 era and up AFAIK).

 

Conspiracy theory:

 

AMD doesn't want developers doing floating point math on the CPU since the Bulldozer architecture gutted it in favor

of more simple integer cores with the idea that anything floating point related would be "offloaded to the GPU". Microsoft

has agreed to go along with this since their compiler competes against Intel's and they have a console partnership with

AMD. So Microsoft has made it difficult and byzantine to do high performance floating point math in their compiler.

In return, AMD gimps their OpenGL performance to coerce developers to DirectX or *Mantle (*which is really a platform

to train PC developers how to code low-level for consoles and ease PC to Console development).

 

Is that out-there enough?

 

Interesting timing on this one Rev...

[revelator]

Hehe i swear im not a microsoft conspirator but yeah that same snag was a real pain in my butt to.

Sadly theres one point that is in serious need of work on Open Watcom, the C++ compiler and its standard library are not done, so Doom3 with open watcom will most likely not work yet

The C compiler is complete though and works like a charm but we do need help on the C++ STL and compiler. P.Chapin is the lead dev (also the only dev) on the C++ compiler, but since hees alone

work has been rather slow. Unfortunatly my C++ fu is not strong enough to help building a standard library from schratch :S so if you know someone with a lot of C++ knowledge we could use his help .

[revelator]

Heh i had a hunch microsoft had some inner workings with AMD, must have started early back in the AMD K6-2 era where AMD started to gain serious momentum over intel.

For one i noticed that the most hated microsoft OS of all time ME ran like a charm on all boxes with AMD hardware (coincidence ???).

[revelator]

After reading about others ramming thee heads against the same pitfall im starting to wonder if it's really an AMD / Microsoft thing ?.

The problem as of now is that microsoft has changed the floating point precision in all there newest compilers to SSE math which would normally be a good thing or would it ?.

See the problem and also the reason why SSE and later are faster than FPU math is that SSE is actually less precise (less strict checking = more speed) in fact its off by exactly 4 bytes

yes someone actually tested it. Normally floating point math allows a certain skew which is normally quite ok but not if your application which was built with FPU math in mind suddenly finds that the values are off by the aforementioned 4 bytes. In case of the Doom3 engine we are so screwed its not even funny, you see idlib actually uses its own floating point library built on the FPU or x87 standard guess what happens when the microsoft compiler tries to optimize it with SSE math yup booom right in your face the values the engine expected are now far off = broken engine.

 

The reason i suspect more than foul play between AMD / microsoft is that the SSE instruction is actually intels, AMD had a similar optimization way way back in the K6 days called AMD-3DNOW which is now pretty much all but extinct, instead they use intels algorithms for SSE so does intel also want to chime in on the conspiracy ?. Sadly microsoft has pretty much stated that the x87 instruction will be dead from MSVC 2013 and onward newer to return despite that it actually is more precise. So as they said fix your code to work with SSE and dont call us back we will not call you.

Looks like someone will have to port idlib from rbdoom3 or rewrite the old one or port the one from rbdoom3. Well atleast the parts concerning floating point math unless microsoft left other surprises for us to find.

[Hiradur]

See the problem and also the reason why SSE and later are faster than FPU math is that SSE is actually less precise (less strict checking = more speed) in fact its off by exactly 4 bytes

 

How come that 32 bit floats are the same speed as 64 bit double then? (EDIT: Although this is true, I didn't think it to the end)

I think the performance gain of SSE comes from vectorization/SIMD which is why e.g. an SSE sqrt on a single variable isn't much faster than sqrt().

Edited February 7, 2015 by Hiradur

[revelator]

SIMD and vectorization are pretty much the same thing -> storing mutiple data in an array, theres some optimization going on behind the scene also ofc but the problem stands its off by 4 bytes ergo vanilla breaks, because its built with the x87 instruction in mind.

Now the funny thing when one thinks about it is that idlib actually does use SSE but it uses an inhouse assembler based version of it built against the x87 instructions.

Its also a fact that the x87 instructions have a higher internal precision than SSE but SSE has the advantage that it might offload that data on a GPU instead of solely on the CPU

somewhere i sense an upcomming directx12 as the reason why they want to force this, with AMD's mantle gaining improvements every day microsoft wants to make sure that directx stays ahead of OpenGL

even if they have to lobotomize there own compiler. Funny thing though i have not yet seen an OpenGL game make use of mantle but i guess better safe than sorry heh.

 

Some discussion elsewhere on the matter http://stackoverflow.com/questions/3206101/extended-80-bit-double-floating-point-in-x87-not-sse2-we-dont-miss-it

[revelator]

http://techreport.com/news/26922/amd-hopes-to-put-a-little-mantle-in-opengl-next

 

Im not a prophet but that one struck me a bit hehe

[lowenz]

Heh, we hit a huge snafu with the MSVC 2013 compiler and had to rollback to 2010 because of precision issues

(MSVC 2013 only does 32-bit intermediate precision unless you specify higher precision with flags and there is a "flag zoo"

with some producing better performance than others depending on the code... MSVC 2010 defaults to 80-bit intermediate precision

which matches the typical float units on CPU's from P3 era and up AFAIK).

 

Conspiracy theory:

 

AMD doesn't want developers doing floating point math on the CPU since the Bulldozer architecture gutted it in favor

of more simple integer cores with the idea that anything floating point related would be "offloaded to the GPU". Microsoft

has agreed to go along with this since their compiler competes against Intel's and they have a console partnership with

AMD. So Microsoft has made it difficult and byzantine to do high performance floating point math in their compiler.

In return, AMD gimps their OpenGL performance to coerce developers to DirectX or *Mantle (*which is really a platform

to train PC developers how to code low-level for consoles and ease PC to Console development).

 

Is that out-there enough?

 

Interesting timing on this one Rev...

"not-so-conspiracy", man.

Edited February 21, 2015 by lowenz