Faster dmapping (testers requested)

[SteveL]

We're working on tweaking DMAP for 2.04 to speed it up by cutting down its console output. It's the console spam that makes DMAP take 4x as long when you have vsync turned on in the video settings. And even when you have vsync turned off, DMAP is significantly faster without the excessive console output.

I'm hoping for some feedback from current mappers who use Windows (I don't know how to compile for Linux, sorry). Did I suppress messages that you want to see? I'll post a patched TDM 2.03 engine in this thread with the dmapping changes. If you're using DMAP right now, you can use this version and it'll be faster.

https://www.amazon.co.uk/clouddrive/share/ELHMA5EQbln-K72wMZt6FKzRBc5BodbL5Qcgc-CICkE?ref_=cd_share_link_copy

To install: save the linked file in your darkmod folder overwriting the current file with the same name (TheDarkMod.exe).
To uninstall: run tdm_update.exe from the same folder

Notes:

• With the patch installed, you can reactivate the original console output for comparison by using option 'v', e.g. dmap v mymap
• Some console spam still comes out of other parts of the code during dmapping (renderer code used to remove triangles, AAS and collision world generation), but not enough to slow things down.
• The console will now stay open throughout, so you'll see the output from the CM and AAS stages too instead of getting a frozen TDM at the end of dmap.
• Your first DMAP after opening TDM will take longer than later ones. The first one loads and parses materials and decls that won't have to be done again until you restart the game. So when comparing, ignore your timings on the first run.

Let me know if any of the messages that I've hidden should be put back again.

[grayman]

I'm using SVN, and dmap is considerably faster. I'm not missing any messages, and I'm thinking the "degenerate triangles" warnings could be suppressed, because that's internal to dmap and there's nothing a mapper can do about them.

[SteveL]

That's true. The only reason I left them in is that they come from general renderer code, not the dmap-specific code that I tweaked in this patch. I don't know whether that code is used by anything except dmap itself. But I'll look for a clean way to suppress the messages. It would be nice if dmap gave just the right amount of meaningful output, so that mappers will always want to read the text. I did some things in that direction in this patch... individual entities get mentioned selectively with their class and name, for example, if they take longer than 0.5 seconds to dmap. But that useful bit of output is still buried beneath messages from the code that's external to dmap, in the renderer, collision world, and AAS sections.

[Bikerdude]

Did a comparison run between 2.03 and svn using Moonbo's new map as a base -

 

2.03 vsync off

• 3:58 console disappears, mouse cursor fixed.
• 4:36 mouse cursor moveable, dmap finished.

SVN vsync off

• 4:28 console always visible, dmap finished.

SVN vsync on

• 4:36 console always visible, dmap finished.

New tdm.exe from OP, vsync off

• 4:28 console always visible, dmap finished.

Map info

• File size - 34mb
• Entities - 7009
• Brushes - 25171
• Patches - 10120

Copied over the existing tdm.exe in the svn folder, ran tdm.exe and got a CTD - dump file.

 

Copied over the 2.03 tdm.exe and the game loads up fine.

[Airship Ballet]

Dmappin' super speedy like

 

Speed of dmapping on an 11MB map probably doubled. I haven't timed it, but it's blatantly miles faster so there's no need.

Edited May 21, 2015 by Airship Ballet

[SteveL]

Copied over the existing tdm.exe in the svn folder, ran tdm.exe and got a CTD - dump file.

Use either SVN or the patch, but not both. The patch is for TDM 2.03, not the 2.04 WIP in the SVN repository.

 

Not sure I understand what results you got in your post above... Did you see no improvement over 2.03 by using the SVN build then? And when you say "dmap finished" do you mean you stopped the timer where that message appeared? That happens before the AAS and CM bits.

[Bikerdude]

• Use either SVN or the patch, but not both. The patch is for TDM 2.03, not the 2.04 WIP in the SVN repository.

Not sure I understand what results you got in your post above... Did you see no improvement over 2.03 by using the SVN build then? And when you say "dmap finished" do you mean you stopped the timer where that message appeared? That happens before the AAS and CM bits.

• Ah yes, I did wonder if that was the case, understood.

Yes SVN was faster than 2.03 with vysnc off by 8s, and with vsync on under SVN its was the same as 2.03. So good news as it is, the new .exe with vsync ON is as fast as 2.03 with vsync OFF.

[SteveL]

• Yes SVN was faster than 2.03 with vysnc off by 8s, and with vsync on under SVN its was the same as 2.03. So good news as it is, the new .exe with vsync ON is as fast as 2.03 with vsync OFF.

I tested it on a few maps and got much more impressive results than that. Hopefully other maps will give you better results too. It will vary by map, based on the amount of FS built in DR (which cause dmapping slowdown).

[Moonbo]

If it makes everyone feel better, it takes about 15-20 minutes for the mission to dmap on my computer :-P.

[ERH+]

If it makes everyone feel better, it takes about 15-20 minutes for the mission to dmap on my computer :-P.

I hate you.

[Bikerdude]

@Moonbo, didnt you win a new PC from the amazon contest? what are the soecs of your current pc..?

[SteveL]

@moonbo: that's really slow. you do have vsync off, don't you?

[Tels]

We're working on tweaking DMAP for 2.04 to speed it up by cutting down its console output. It's the console spam that makes DMAP take 4x as long when you have vsync turned on in the video settings. And even when you have vsync turned off, DMAP is significantly faster without the excessive console output.

 

I'm hoping for some feedback from current mappers who use Windows (I don't know how to compile for Linux, sorry).

I'd like to try it in Linux, unfortunataly, it doesn't compile here anymore. Is there anyone still testing TDM on Linux?

[Bikerdude]

I'd like to try it in Linux, unfortunataly, it doesn't compile here anymore. Is there anyone still testing TDM on Linux?

There should be a few people as I have seen more than the odd post from someone with linux doing DR/TDM related stuff.

[Goldwell]

I always find the first dmap after freshly booting up TDM to be the longest so I ran a dmap after a fresh boot and then another straight after to see if there was any difference. Overall I found it did trim down my dmapping speeds but not by a massive amount.

 

Mapsize 5.16mb

Without patch

dmap first time - 58 seconds

dmap second time - 20 seconds

With patch

dmap first time - 48 seconds

dmap second time - 15 seconds

[Bikerdude]

On the subject of doing multiple DMAP/MAP runs, this is greatly aided by having lots of system ram.

 

The more ram you have the longer you can go before TDM throws a memalloc error, for example with 16Gb of ram I can go for hours doing multiple runs with TDM/DR both open while working on a map

[Moonbo]

@Steve/Biker, I do have vsync turned off but I do all my mapping on my laptop which even chugs when loading the map in DR at this point :-).

[AluminumHaste]

Nice work Steve, I did have some requests if this is possible.

Once the first part of DMAP has finished, the console closes but now it's doing pathfinding and it's not done.

Could it be changed to keep the console open until Dmap is completely finished?

 

Also about the console output, would it be possible instead to change it so that every 2 seconds, a percent progress update shows instead of nothing?

`

[SteveL]

Nice work Steve, I did have some requests if this is possible.

Once the first part of DMAP has finished, the console closes but now it's doing pathfinding and it's not done.

Could it be changed to keep the console open until Dmap is completely finished?

 

Also about the console output, would it be possible instead to change it so that every 2 seconds, a percent progress update shows instead of nothing?

`

 

Console closing: I thought I'd fixed this. It works for me. I didn't try to stop the console closing -- which happens outside of dmapping code -- but I put in a command to reopen it immediately. I don't even see it close, and at the end, it's still open. What's happening at your end exactly?

 

Progress updates: We could add some kind of progress update. It'd be nice to have a line of output for each of the 10 major steps in dmapping, and a 1-liner account of what got processed. The longer running stages could report a counter at least that got updated every 10 or 100 items: 550 / 2435 func statics dmapped, for example, if we can get it to write on the same line every time rather than scroll the console.

[AluminumHaste]

Oh my bad, I was testing 2.03 lol.

Yeah that's better.

[SeriousToni]

It's the console spam that makes DMAP take 4x as long when you have vsync turned on in the video settings.

 

 

LOL, srsly?! Can't believe this!