Jump to content
The Dark Mod Forums
Sign in to follow this  
stgatilov

NOTICE: 32-bit executables deprecated

Do you play 32-bit or 64-bit build of TheDarkMod?  

24 members have voted

  1. 1. Do you play 32-bit or 64-bit build of TheDarkMod?

    • I play 64-bit build and think 32-bit build is useless.
      20
    • I play 64-bit build, but I think that 32-bit build is better in some regard.
      2
    • I play 32-bit build and cannot play 64-bit build at all.
      2


Recommended Posts

TheDarkMod offers 64-bit build since 2.06, and I can't recall any issues with with it since then.
I guess most of the people use 64-bit executable now. As far as I know, developers have long stopped testing 32-bit build ūüėĀ
Additionally, there is one recent gigantic FM which has too high memory requirements. As far as I remember, it cannot be dmapped with 32-bit build at all.

For the sake of clarity:

  • 64-bit executables are: TheDarkModx64.exe, thedarkmod.x64, tdm_update.linux64
  • 32-bit executables are: TheDarkMod.exe, thedarkmod.x86, tdm_update.linux

 

We plan to deprecate 32-bit builds after 2.08.

Here is exactly what would change:

  1. The website will offer 64-bit updater for Linux by default (now it offers 32-bit one).
  2. The updater will install only 64-bit game executables. 32-bit executables will not be packaged with the rest of the game.
  3. Developers will not test 32-bit build. Well, they don't test it now anyway ūüėĀ
  4. The 32-bit executables for the latest TDM release will be available as a separate download on website, without minimal-to-none testing.

Of course, we will not do anything to break 32-bit builds and will not remove any parts of it.
On developer side, we will still maintain 32-bit build, just with minimum effort. We will still ensure that it compiles, and probably test it from time to time.

 

I'd like to ask all the players with 32-bit OS to respond to this thread. You are the ones who will have to download the separate 32-bit executables to continue playing.

If you can play 64-bit executable but think 32-bit ones are useful in any regard, please post a comment too.

Share this post


Link to post
Share on other sites

In 2.06, the 64-bit executable had a performance deficit but as of 2.07 and later the two have parity.

The only thing that may be an outlier is Pentium 4 SIMD? Does any of the original id Software "P4" SIMD assembly exist in current 32-bit builds?

Should anyone on a Pentium 4 be playing TDM 2.08 or newer?


Please visit TDM's IndieDB site and help promote the mod:

 

http://www.indiedb.com/mods/the-dark-mod

 

(Yeah, shameless promotion... but traffic is traffic folks...)

Share this post


Link to post
Share on other sites
26 minutes ago, nbohr1more said:

In 2.06, the 64-bit executable had a performance deficit but as of 2.07 and later the two have parity.

More SIMD routines were reimplemented, so that's not surprising.

Quote

The only thing that may be an outlier is Pentium 4 SIMD? Does any of the original id Software "P4" SIMD assembly exist in current 32-bit builds?
Should anyone on a Pentium 4 be playing TDM 2.08 or newer?

I don't know what is "Pentium 4 SIMD".
The SIMD code contains a lot of assembly written by ID for Doom 3. They are enabled only in 32-bit build, there is no way to make them work in 64-bit build. But I suppose all performance-heavy functions out of them have already been reimplemented for 64-bit build.

Pentium 4 was the top CPU when Doom 3 came out. I think nobody has it now.

Share this post


Link to post
Share on other sites

Here is some background on how Doom 3 was tailored to the P4:

https://fabiensanglard.net/doom3_documentation/37729-293751.pdf

https://fabiensanglard.net/doom3_documentation/37727-293749.pdf

https://fabiensanglard.net/doom3_documentation/37728-293750.pdf

https://fabiensanglard.net/doom3_documentation/37726-293748.pdf

https://fabiensanglard.net/doom3_documentation/37730-293752.pdf

 

Not to impugn the esteemed J.M.P. van Waveren but I am fairly certain that a few coins landed in his pockets when he prominently mentioned

Pentium 4 in those technical reviews...


Please visit TDM's IndieDB site and help promote the mod:

 

http://www.indiedb.com/mods/the-dark-mod

 

(Yeah, shameless promotion... but traffic is traffic folks...)

Share this post


Link to post
Share on other sites
8 minutes ago, nbohr1more said:

Here is some background on how Doom 3 was tailored to the P4:

All of them are about SIMD code.

The reasons he mentions P4 are 1) It was the most recent Intel CPU at the moment of Doom 3 release, and 2) It was the first CPU to support SSE2 instructions.
Before P4, there was only SSE, which is very limited: it only supports working on floats. SSE2 extended SIMD to doubles and integers of all sizes, making it much more versatile.
Today SSE2 is required by TDM, and guaranteed in x64 mode.

Share this post


Link to post
Share on other sites

What does it entails to change a code base to 64bits? Just change the SSE and SIMD code?

About the pool, my case is not there, I play the 32bits version but I can also play the 64bits version just fine, I just chose to play the 32bits version because I assumed it to be the most stable version, but if is not tested anymore, and there's no performance diference than I will play with 64bits one, thanks for the info stgatilov. 

Share this post


Link to post
Share on other sites

Does really anyone play TDM on 32-bit-only hardware in 2020?

If not i would recommend dropping 32 bit support completely, so devs don't have to avoid breaking it every time they work on the engine.
Code is not only an asset but also a burden. It has to be maintained. Less code is easier to maintain.

Share this post


Link to post
Share on other sites
2 hours ago, Abusimplea said:

Does really anyone play TDM on 32-bit-only hardware in 2020?

The main concern is not people with 32-bit only hardware, but people who have installed 32-bit OS for some reason. This is pretty easy to do even today.

Quote

If not i would recommend dropping 32 bit support completely, so devs don't have to avoid breaking it every time they work on the engine.

If 32-bit build gets broken while 64-bit is not, this is most likely a sign of bad and non-portable code, which should be fixed anyway, and which most likely can hurt in 64-bit build too.
Perhaps other developers have different habits, but I can say for myself that I have tested 32-bit build for something like 10 times since 2.07, and it seems to work properly without much effort.
 

Share this post


Link to post
Share on other sites
On 5/24/2020 at 9:40 PM, HMart said:

What does it entails to change a code base to 64bits? Just change the SSE and SIMD code?

It was a pretty hard work to make engine work in 64-bits, because ID wrote code in very platform-specific manner, they never expected it to become 64-bit.
However, all that work was finished by 2.06, and no more effort was spent on it since then.

Quote

About the pool, my case is not there, I play the 32bits version but I can also play the 64bits version just fine, I just chose to play the 32bits version because I assumed it to be the most stable version, but if is not tested anymore, and there's no performance diference than I will play with 64bits one, thanks for the info stgatilov. 

It would be great if people do independent performance testing. To be honest, I did not check it for a long time already, and one guy (nbohr1more) is not enough to draw a final conclusion.
I believe the performance difference is not very high though.

Share this post


Link to post
Share on other sites
5 hours ago, stgatilov said:

The main concern is not people with 32-bit only hardware, but people who have installed 32-bit OS for some reason. This is pretty easy to do even today.
 

I guess you could pose the same question then: Is there really anyone who plays TDM on a 32-bit OS nowadays? :)

I'm sure there is. 1 or 2. AFAIC, you could easily drop 32-bit support, but, of course your decision.

Edited by chakkman
  • Like 1

Share this post


Link to post
Share on other sites

I'm not a coder so my opinion is arguably biased, but I think 32-bit is terrible.

Old, faulty, limiting. It's simply not worth it anymore, it was great back in the days because, heh, 16-bit, anybody remember that?

Everybody should use 64-bit and be forced to use it...Although I'm a bit of a hypocrite saying that because I hate Windows 10, yet I use it, yet I want Windows 7 back but I'm paranoid of hackers especially since I got trembled by them 9 times, 3 were successful.

Share this post


Link to post
Share on other sites

Worth noting that Microsoft is getting ready to kill off support for 32-bit Windows, and most Linux flavors killed native support for 32-bit installs years ago. Not Debian, which is what I use (but not for that reason).

 

My vote is to ditch 32-bit support if it makes things easier for the team. I used to play TDM on a high-end 3GHz Pentium 4, and the experience was less than satisfactory anyway. Only the simplest in-door missions ran well.

 

As time goes by, people will want to use larger textures and better maps/models, and the memory cap of 32-bit becomes more and more of a problem in this scenario.

Share this post


Link to post
Share on other sites

Even tho I do agree that TDM should go 64bits exclusively I most say this:

Using 64bits software doesn't mean 32 bits goes away, why, first because it is neither old nor faulty, afaik is just a size of a memory address (an integer), and second because even tho idtech 4 is or was 32bits software, it still uses 8bits and 16bits wide data, and TDM 64bits still uses 8,16 and 32 bit wide data.  

Going 64bit, does make a game faster, just because the team has the option to use bigger memory address numbers and so, access more RAM, but this is only important if your game is memory starved.  Other thing that 64bits gives is the option to have more precision on some CPU/GPU calculations, that does mean better looking effects but also mean heavier calculations and so slower calculations, talking in global terms, is that what you need for your game? Do you lose much by doing the 32bits calculations instead? This is what game/engine devs need to think about.

Having the option to do more, also means you can be more wasteful about memory usage/optimization, the game can end bloated very fast and slower to many people using low end hardware. 

Edited by HMart
  • Like 1

Share this post


Link to post
Share on other sites

So many misconceptions here, I guess I should say something.

13 hours ago, HMart said:

Using 64bits software doesn't mean 32 bits goes away, why, first because it is neither old nor faulty

Yes, 32-bit is perfectly OK in general.

Quote

Going 64bit, does make a game faster, just because the team has the option to use bigger memory address numbers and so, access more RAM, but this is only important if your game is memory starved. 

There is limitation of 2GB of memory in 32-bit mode. And that's a hard limit: a process cannot use more than 2 GB even if you have 16GB RAM physically installed and 32GB pagefile for swap. When trying to allocate more, process will simply crash on 32-bit platform.

If your workload fits well into this limit, then there is no harm from it. If it does not, then it becomes an unavoidable blocker. If you have a very big map, you simply cannot dmap it on 32-bit, but if your map is small/medium size, it does not matter for you if executable is 32-bit or 64-bit.

Quote

even tho idtech 4 is or was 32bits software, it still uses 8bits and 16bits wide data, and TDM 64bits still uses 8,16 and 32 bit wide data.  

Speaking of variables in code and 64-bit mode switch, some of them stay the same size, but some of then switch from 32-bit to 64-bit.

All the pointers inevitably become two times larger, and all sizes in STL types do the same. Also, 64-bit process has more overhead for heap allocations. It means that the process consumes more memory in 64-bit, and consequently can work slower since the cache size does not change. You can probably see increased memory consumption yourself: take a big brushy map, and dmap it both in 32-bit and in 64-bit modes, looking at memory consumption in process explorer.

While this point is not very strong for native applications and games, it is a major problem for managed languages like Java, where pointers are everywhere. For instance, modern JVM still uses 32-bit pointers internally, as long as you don't request for more than 32 GB of heap memory. They say it is 30% faster ūüėā

Quote

Other thing that 64bits gives is the option to have more precision on some CPU/GPU calculations, that does mean better looking effects but also mean heavier calculations and so slower calculations, talking in global terms, is that what you need for your game? Do you lose much by doing the 32bits calculations instead? This is what game/engine devs need to think about.

64-bit mode does not offer any new capabilities except for more-than-2GB memory.
There are many other differences between 32-bit and 64-bit, but they are all about performance.

64-bit mode does not offer better precision. Floating point numbers are 32-bit and 64-bit both on 32-bit and 64-bit, working with same performance. In fact, 32-bit mode also offers 80-bit floating point numbers. Moreover, before something like TDM 2.06, this awful 80-bit arithmetic was used everywhere for intermediate values when computing complicated expressions. We got many precision problems when we disabled it, and we are still fixing them (e.g. there is a bunch of such fixes for dmap in upcoming 2.08). Now 80-bit floats are disabled both for 32-bit and 64-bit builds of TDM.
So it turns out that 32-bit mode offers more floating point precision, although these 80-bit numbers cannot be used universally in Visual Studio anyway.
(assembly geeks can also notice that 32-bit mode without 80-bit arithmetic has a tiny bit of overhead for passing floats into functions, due to old calling convention)

Speaking of integers, both 32-bit and 64-bit modes offer integers of size up to 64-bit. The 64-bit integers are emulated on 32-bit platform and are pretty slow, but TDM never uses them. 64-bit GCC also offers builtin 128-bit integer type, but Visual Studio does not, and so TDM does not use them. Anyway, I can hardly imagine what we could use them for ūüėĮ
(assembly geeks can also notice that using 32-bit integers in 64-bit mode occasionally have a bit of overhead, due to some additional extending instructions)

And of course bitness of executable does not affect GPU ūüėĀ

 

The main performance advantage of 64-bit mode is having 2x more registers in CPU: both GP registers and SSE/AVX registers.
When compiler goes out of registers in a tight loop, it has to "spill" into memory. This memory is surely in L1 cache, but L1 cache accesses are still slower than registers, plus more dependencies between instructions.
In TDM, there are many computationally heavy routines done in SSE/AVX (aka SIMD), which we had to rewrite for 64-bit mode. Because the original implementation by ID was written in inline assembly, and 1) you cannot use 32-bit assembly on 64-bit build, and 2) VC does not allow inline assembly at all in 64-bit build. So there are two completely different implementations of SIMD routines now: the first one (by ID) is used in 32-bit mode, and the second one (by us) is used in 64-bit mode. As far as I remember, the second one is often pretty low on registers, so if we start using it in 32-bit mode, it will work a bit slower. Some of the new routines written by us are already used in 32-bit mode, e.g. AVX code.

 

Quote

Having the option to do more, also means you can be more wasteful about memory usage/optimization, the game can end bloated very fast and slower to many people using low end hardware. 

Quite unlikely I would say.
Only that 32-bit build will start working slower than 64-bit build. People with low-end machines should use 64-bit build, so it's not a big problem.

 

As you see, performance difference is a very complicated question.
Out of all I wrote, I think the most difference comes from:

  1. 64-bit build wastes more memory (pointers, heap overhead)
  2. 64-bit build has more registers
  3. 64-bit and 32-bit builds have different SIMD implementations

To be honest, no idea which build wins now.

  • Like 3

Share this post


Link to post
Share on other sites
10 hours ago, stgatilov said:

There is limitation of 2GB of memory in 32-bit mode. And that's a hard limit: a process cannot use more than 2 GB even if you have 16GB RAM physically installed and 32GB pagefile for swap. When trying to allocate more, process will simply crash on 32-bit platform.

Interesting read, but this is not correct. There is the /LARGEADRESSAWARE flag with which you can allocate upto about 3.5 GB on a 32-bit process.

https://docs.microsoft.com/de-de/cpp/build/reference/largeaddressaware-handle-large-addresses?view=vs-2019

And we actually use the flag on TDM.

Share this post


Link to post
Share on other sites
26 minutes ago, STiFU said:

Interesting read, but this is not correct. There is the /LARGEADRESSAWARE flag with which you can allocate upto about 3.5 GB on a 32-bit process.

This is just a minor detail.
Yes, you can enable PAE and use 32 GB on a 32-bit Windows server OS, as long as you are ready to live with segmented addressing (like in 16-bit times).

Share this post


Link to post
Share on other sites
On 5/31/2020 at 6:45 AM, stgatilov said:

So many misconceptions here, I guess I should say something.

Yes, 32-bit is perfectly OK in general.

There is limitation of 2GB of memory in 32-bit mode. And that's a hard limit: a process cannot use more than 2 GB even if you have 16GB RAM physically installed and 32GB pagefile for swap. When trying to allocate more, process will simply crash on 32-bit platform.

If your workload fits well into this limit, then there is no harm from it. If it does not, then it becomes an unavoidable blocker. If you have a very big map, you simply cannot dmap it on 32-bit, but if your map is small/medium size, it does not matter for you if executable is 32-bit or 64-bit.

Speaking of variables in code and 64-bit mode switch, some of them stay the same size, but some of then switch from 32-bit to 64-bit.

All the pointers inevitably become two times larger, and all sizes in STL types do the same. Also, 64-bit process has more overhead for heap allocations. It means that the process consumes more memory in 64-bit, and consequently can work slower since the cache size does not change. You can probably see increased memory consumption yourself: take a big brushy map, and dmap it both in 32-bit and in 64-bit modes, looking at memory consumption in process explorer.

While this point is not very strong for native applications and games, it is a major problem for managed languages like Java, where pointers are everywhere. For instance, modern JVM still uses 32-bit pointers internally, as long as you don't request for more than 32 GB of heap memory. They say it is 30% faster ūüėā

64-bit mode does not offer any new capabilities except for more-than-2GB memory.
There are many other differences between 32-bit and 64-bit, but they are all about performance.

64-bit mode does not offer better precision. Floating point numbers are 32-bit and 64-bit both on 32-bit and 64-bit, working with same performance. In fact, 32-bit mode also offers 80-bit floating point numbers. Moreover, before something like TDM 2.06, this awful 80-bit arithmetic was used everywhere for intermediate values when computing complicated expressions. We got many precision problems when we disabled it, and we are still fixing them (e.g. there is a bunch of such fixes for dmap in upcoming 2.08). Now 80-bit floats are disabled both for 32-bit and 64-bit builds of TDM.
So it turns out that 32-bit mode offers more floating point precision, although these 80-bit numbers cannot be used universally in Visual Studio anyway.
(assembly geeks can also notice that 32-bit mode without 80-bit arithmetic has a tiny bit of overhead for passing floats into functions, due to old calling convention)

Speaking of integers, both 32-bit and 64-bit modes offer integers of size up to 64-bit. The 64-bit integers are emulated on 32-bit platform and are pretty slow, but TDM never uses them. 64-bit GCC also offers builtin 128-bit integer type, but Visual Studio does not, and so TDM does not use them. Anyway, I can hardly imagine what we could use them for ūüėĮ
(assembly geeks can also notice that using 32-bit integers in 64-bit mode occasionally have a bit of overhead, due to some additional extending instructions)

And of course bitness of executable does not affect GPU ūüėĀ

 

The main performance advantage of 64-bit mode is having 2x more registers in CPU: both GP registers and SSE/AVX registers.
When compiler goes out of registers in a tight loop, it has to "spill" into memory. This memory is surely in L1 cache, but L1 cache accesses are still slower than registers, plus more dependencies between instructions.
In TDM, there are many computationally heavy routines done in SSE/AVX (aka SIMD), which we had to rewrite for 64-bit mode. Because the original implementation by ID was written in inline assembly, and 1) you cannot use 32-bit assembly on 64-bit build, and 2) VC does not allow inline assembly at all in 64-bit build. So there are two completely different implementations of SIMD routines now: the first one (by ID) is used in 32-bit mode, and the second one (by us) is used in 64-bit mode. As far as I remember, the second one is often pretty low on registers, so if we start using it in 32-bit mode, it will work a bit slower. Some of the new routines written by us are already used in 32-bit mode, e.g. AVX code.

 

Quite unlikely I would say.
Only that 32-bit build will start working slower than 64-bit build. People with low-end machines should use 64-bit build, so it's not a big problem.

 

As you see, performance difference is a very complicated question.
Out of all I wrote, I think the most difference comes from:

  1. 64-bit build wastes more memory (pointers, heap overhead)
  2. 64-bit build has more registers
  3. 64-bit and 32-bit builds have different SIMD implementations

To be honest, no idea which build wins now.

Hey thanks for the deep explanation, I'm still learning this stuff, so is no surprise to me that I add some wrong or incomplete ideas but i'm glad I said it, because it made you correct me and that will just improve my knowledge. :)

  • Like 1

Share this post


Link to post
Share on other sites
1 hour ago, HMart said:

Hey thanks for the deep explanation, I'm still learning this stuff, so is no surprise to me that I add some wrong or incomplete ideas but i'm glad I said it, because it made you correct me and that will just improve my knowledge. :)

Don't worry, it's a very common misunderstanding. We talk frequently of "32 bit integers" or "64 bit floats" so people naturally assume that when we talk about "32 bit code" or "64 bit code" we mean the same thing, but in fact there is no connection between the size of memory addresses (which is what "64 bit code" refers to) and the ability to choose arbitrary sizes for integer or floating point data.

8, 16, 32 and 64 bit integers were already available in 32 bit code and remain available in 64 bit code, as do 32 bit and 64 bit floating point values, and often extended data types like the 80 bit "long double" floating points that stgatilov mentioned. The choice of which to use is determined by the size of the values that need to be represented, along with performance considerations and/or the amount of memory available.

  • Like 1

Share this post


Link to post
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.

Guest
Reply to this topic...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

Sign in to follow this  

×
×
  • Create New...