stgatilov

Measured it with timewatch. Full build took 2:41 before Eigen, and took 3:15 after Eigen. I guess I should dig deeper what the numbers mean

Given that I do not work with DR on daily basis, for me all DR-related concerns are purely theoretical. Compared to some bad projects on my daily job (hint: better do not combine templates with automatic code generation) it is fast anyway. Do not consider this post as a complaint, better treat it as an interesting piece of information. It is up to you what to do with it. I will measure wall time with and without Eigen at the moment when it was integrated. I measured time of full clean build. Of course, more typical incremental builds should be much faster, but: Since vector math is almost everywhere, it would be correct to expect incremental compilation to become slower by about the same ratio. Linking time becomes a bigger problem for incremental builds. Many template instantiations make it slower too. One indirect way to estimate it is too look at the total size of .obj files. Precompiled headers will win back at most 10% of the time (at most 3% if only considering Eigen). The real problem is template instantiations everywhere. The only way to fix it is: Return back simple vector/matrix classes with trivial implementations of simple arithmetic operations straight in header (they should be inlineable). Include Eigen headers in only one cpp file and use it to implement whatever complicated operations you need. Expose such operations to headers as non-inlineable methods. In other words, either don't use Eigen, or set a compilation firewall between it and the rest of the codebase. Without compilation firewall, you won't get rid of the slowdown.

Most likely there is another problem with Eigen: very slow Debug performance. I have no idea what are typical CPU-intensive workloads in DR, but my impression was that they rarely have numeric nature. If this is true, then slow Debug performance is not a problem (and potentially faster Release performance is not a benefit).

Recently I have learnt about new cool toy for analyzing build time in Visual Studio: C++ Build Insights Basically, MSVC build tools now generate events for various things, which allows to see timelines, aggregated information, etc. The events are based on Event Tracing for Windows, which is a relatively known framework among some very cool guys. Since I was thinking about picking up the DarkRadiant side again, I decided to give it a run on DR projects. To make timings easier to understand, I limited parallelism to 1 project and 6 compiler threads at a time (I have 6 physical cores). I built vcperf from sources (the latest tag on GitHub), because stock VC2019 build does not include information about templates, and that's the main problem usually. I used VC2019 to build Debug x64 build. I think release build does not matter much, since developer usually work with Debug and have to wait for it all the time. I'll try to explain the findings. Here is the overview of stuff: it explains how much time each part of compiler takes: In most cases, I will only look at "Exclusive Direction". Inclusive stuff is too bad because it takes common stuff into account many times when we sum up the values. CPU Time is a better metric in general, but most of low-granularity timings don't provide it --- and that's what we want to analyze. Besides, I have specifically ensured that there is one thread per physical core so CPU time and duration should be more or less the same. So, the whole build takes 986s. Out of it, C1DLL takes 785s. That is the frontend part of compiler, which most likely contains parsing C++ code and doing template instantiations. At least, an image in this article says that template instantiation is part of frontend. And otherwise numbers simply don't match. The backend takes something like 175s --- most of the other time. Clearly, the 80% of the time is spent in frontend, so let's given it a closer look. First of all, let's look at C++ code parsing: The whole parsing takes 172s. By filtering items containing some substring, I learned that: Parsing Eigen takes 31.5s Parsing Wx takes 32.6s Parsing MSVC standard headers takes 34.5s Parsing Windows SDK headers takes 24s Parsing w32deps includes except Eigen and Wx takes 11.4s Parsing all the rest takes 38.5s The number for Wx is OK, since it is very important for DR. Standard includes are also understandable --- that's STL and windows.h. But isn't Eigen excessive here? Note that parsing time can be reduced almost to zero by using precompiled headers properly. According to the stats I see, windows.h is parsed 63 times, which gives 33s in total. If it is always in PCH, than it would be parsed at most 19 times.... Anyway, parsing takes about 17% of total build, so it is hardly a real problem. Now we come to template instantiations --- the root of evil for slow C++ builds I computed sum of all Durations in this list, and got 605s --- this approximately matches 785s - 172s (frontend - parsing). And that's 60% of whole build time. Quite obviously, Eigen is the king here: all the most expensive instantiations come from Eigen. Here is breakdown: Eigen::* takes 461s std::* takes 122s All the rest takes 22s Unless someone decides to use boost, STL is usually the most offensive part of template bloat. However, Eigen manages to top it in 4x times. So the conclusion is: adding Eigen slowed down build in at least two times. If it was wxWidgets, it would be OK, since GUI is the most important part of DarkRadiant. But such cost for 4D vectors and matrices? Seriously?! Does it even offer any added value in DR? UPDATE: Given that backend most of the time generates machine code for the template instantiations, I bet the real slowdown can be even 2.5-3 times. If someone remembers when Eigen was integrated, it can be checked directly.

Yes, texture overrides don't should not be a problem, here at least. As for objectives, I think they need major refactoring in TDM core game. As I wrote once, there are 4 instances of objectives list, and some of them are copy/pasted, some have name collisions. I guess I forgot to mention it: of course mission loading GUI is fully defined by FM --- nothing changes here.

Let's start with a hard example: "Volta 2: Cauldron". I'm afraid I need help from @kingsal here. There is a ton of changes and overrides in GUI files when I compare to 2.09, but the only difference that I can spot is the background. Also parchments are changed for objectives, but I think it is done directly by overriding images, no .gui files involved. It looks very weird of course, since some objectives parchment has new default style, some has old default style ("dirty" parchment), and some have customized parchment. I have added a "fixed" version, but I only removed all forbidden file overrides, and added the following to "mainmenu_custom_defs.gui": //stgatilov: disable briefing state after video #define ENABLE_MAINMENU_BRIEFING 0 //stgatilov: display custom background everywhere #define MM_BACKGROUNDS_MAINMENU_NOTINGAME BackgroundCauldron #define MM_BACKGROUNDS_EXTRAMENU_NOTINGAME BackgroundCauldron #define MM_BACKGROUNDS_MAINMENU_INGAME BackgroundCauldron #define MM_BACKGROUNDS_EXTRAMENU_INGAME BackgroundCauldron #define MM_BACKGROUNDS_BRIEFING BackgroundCauldron #define MM_BACKGROUNDS_DIFFSELECT BackgroundCauldron #define MM_BACKGROUNDS_SHOP BackgroundCauldron Of course, I defined "BackgroundCauldron" in "mainmenu_background_custom.gui" too. P.S. I also wonder if I should support something like MM_BACKGROUNDS_ALL, which would automatically override all backgrounds. Although failure and success screens are not changed here... so I guess it would be hard to decide what "ALL" should mean.

This is so-called "internal name", it matches the name of the subdirectory in fms directory. I'm writing it because it is the easiest for me to obtain with search scripts. Hah! There are mission authors who want to do font changes too. And backgrounds are de-facto controlled by mission authors. I'm afraid there is no way to "not intervene with mission modifications". In principle, you can unpack the guis directory and edit the files. Of course, if developers change these files, then you can get into trouble after update. Another approach is to expose something via cvar, but I think it is worth doing only if many people want it (yeah, it is fragile!)

I have done some proofreading: https://github.com/freyk22/tdminstallerwin/pull/1 Also, the following things went to my mind when I tested it: The option to "install OpenAL". First of all, it has absolutely no effect on TDM. Second, it installs some "OpenAL" from Creative website, which is a good way to get problems if user accidentally has Creative hardware. I think this option should be removed/hidden completely. The option to "install VC redistributable": it is not needed now. Of course, this thing won't do any harm (and every program does it today). However, now its description says "This installs required visual studio files", which is wrong. Your installer has its own LICENSE file and displays to the user. It is outdated and much shorter than it should be. Of course, official installer does not even show the license to the user... but I think it you show it, you should show the correct one. The installer shows that TDM will take 1.8 MB of space, which is... a bit underestimating That's not a big problem of course. Maybe it's better to show some rough estimate (e.g. I wrote 4-6 GB in the wiki article) ? Or simply hide this "space required" if NSIS allows that? I used my old VM for testing, with 32-bit Windows 7. Everything installed OK, but the desktop icon points to TheDarkMod.exe, which does not exist, because 32-bit TDM is deprecated now and such executables are provided in separate download. First of all, I think user with 64-bit OS should only get one icon to 64-bit TDM. Second, I think something should be done for the case of 32-bit OS too... blocking install or displaying a warning maybe.

(Now that I see the list of FMs , I guess I'll reserve one more post)

Yes, something triggers quickload after the level loading ends. No freaking idea what it is! That's the most hilarious technical issue I have even seen Try to bind quickload to something ordinary... like NumPad9 or something. Also, you can look through DarkmodKeybinds.cfg and DarkmodPadbinds.cfg and comment/delete all lines which have "loadgame" in them. Which language do you use in TDM? Which is the default language of Windows? Default input language? Which keyboard layout do you use?

Hey, memory dump can be useful only if done at the right moment! I'm not sure if there is such moment in this case. If only when the problematic message shows up but before closing it...

Known general issues: When clicking "Restart" in settings, the game does not land back to the same settings. In mission failed screen, the list of objectives does not show status of objectives properly. Missions to be checked and fixed: A) Overriding some mainmenu_*.gui which should not be overriden: ac1 (post) ac2 (post) bcd (post) bathhouse (post) briarwood_manor (post, update) claw1_31 (post, fix) cauldron_v2_2 (post) --- changes accepted by @kingsal elixir (post) gatehouse1_3 (post) good (post) hhi (post) hhta (post) hhvf (post) hhtlc (post) itb (post) lich_queens_demise (post) northdale1 (post) northdale2 (post) nowandthen (post) painterswife (post) snowed_inn (post) springcleaning (post) talbot (post) ulysses_genesis (post) u2_flock (post) volta1_3 (post) NEW: away0 (post) B| Videos controlled by SDK (both briefing and debriefing): ahouseoflockedsecrets (post) nhat3 (post, update) requiem (post) score_to_settle (post) -- acknowledged by @Springheel sneak_destroy (post) C) Custom title: hareinthesnare (post) marshofrahena (post)

Almost a year ago I raised concern about reckless overriding of main menu GUI files (which explicitly say they must not be overridden) by mappers. I have changed ton of things now, and I think I'm ready to show the new version for beta testing. If everything goes well, these changes will be released in TDM 2.10. You can obtain the game build for beta testing via tdm_installer. Just check "Get custom version" on the first page and select "test-menugui-N" version in test build. Note that it is close to the latest dev build, and thus includes all the changes done since 2.09. Here is the list of customizations in the new version that I know of: Change background everywhere. Change music everywhere. Disable mission-specific screens. Briefing and Debriefing videos (done the "old" way). Replace briefing screen with whatever mapper wants. Custom mission loading GUIs. All of the above can be set independently for each mission in a campaign. Customize the "Wait until ready" screen. I did not expose the ability to change stuff in Credits screen, since in my understanding that's credits for TheDarkMod itself, thus mission authors should not touch it. The instructions about how to achieve points 1, 2, 3, 4, 6 are given in "mainmenu_custom_defs.gui". Here are things which I have marked as deprecated and want to remove: Videos "controlled by SDK". The "old" approach can now be used both for briefing and debriefing, even in campaign. Custom title. As far as I remember, it has only been used to hack custom background into the menu. Why keeping it? Of course, all the missions which use deprecated features need to be fixed to rely on something else. The missions which override .gui files in unsupported way are most likely broken too and need fixing. All the other missions should work properly now. Here is the policy about fixing FMs: I will fix all FMs, unless mission author wants to do it himself. Fix will only change .gui files, nothing else. Fixed mission should work as intended in 2.10. Fixed mission should play well in 2.09, except that main menu can look differently, and probably briefing/debriefing video can be skipped. Every mission will be briefly beta-tested here. Not the gameplay of course (that would take too much effort), but various menu/gui interactions. Until 2.10 is released, the fixed missions will be stored on separate server and will only be available via special cvars tweaking. Ordinary players will still get non-fixed versions, since they most likely use TDM 2.09. In order to download fixed FMs, you should add the following line to "autoexec.cfg": set tdm_mission_list_urls "http://darkmod.taaaki.za.net/5323_menugui/missionlist.xml" If you want to join testing, I suggest copying TDM installation without FMs, then updating it to appropriate version and changing "autoexec.cfg". Trying to juggle with one installation will be confusing.

Which missions did you try? Did you try prebundled missions like "Tears of Saint Lucia" or "New Job" ? What happens if you install Saint Lucia FM, then open game console and execute "map saintlucia" ? You won't have any items this way, but at least: does it start the map properly? Do you have any command line parameters passed to executable? If you did some configuration already, I'd suggest temporarily removing it. Move these files to some directory: darkmod.cfg and DarkmodKeybinds.cfg. Just to be sure you did not accidentally bind something like "left click" to quick load. I guess you can share some logs, although I'm not sure they will help. First of all, you can attach condump, made after you catch the bad error message: https://wiki.thedarkmod.com/index.php?title=Reporting_Problem#condump Second, you can find tdm_installer_XXXXXX.log files, zip them all and share here (I think I only need the latest one, but better just send all).

So, instead of making a more obvious frob highlight, you suggest making all junk objects static? I think it goes against the spirit of immersive sim even more. I don't like playing such missions: there are many objects on the table, but I cannot pick any of them. Can't even throw the damn candle, because it behaves as "turn off" button.

I don't think tdm_installer is more user-friendly than tdm_update. Aside from a new button for adding icon, I think it gives the same experience. For instance, it does not mess with user permissions (only warns about trying to do something wrong --- just like the very latest incarnations of tdm_update). Yes, Freyk’s Unofficial Installer should be updated to use tdm_installer instead of tdm_update, or it should be removed from website. P.S. @freyk, I'm really sorry to create a name collision between two installers! I tried to take a name different from "tdm_update" to avoid confusion, but collided with your installer as the result

Exactly! That's the main reason why I like the idea of different colors. Although it is quite likely that another change will diminish this problem. I was thinking about candles, which are moveable depending on author's preference. UPDATE: Also, I have seen some maps where bottles and plates were loot, while they did not look like that. Having different highlight color will help with it. As for customization of frob effect, I think anything beyond "change color" or "occlude or not" would require additional code and complexity in the engine.

Normal people don't need to know version. If you want to check for updates, just finish your current playthrough and run tdm_installer without any settings,

Open console: it shows major version + engine revision. When taken together, they allow to fully identify version.

Yes, no way to customize.

If you build engine from source, you have better chances of success. The main idea was to remove version check: it does not reflect our versioning system properly.

New dev build: dev16269-9407

Ok, changed it again, and now it seems to work (at least on appveyor). Not only CheckPIESupported is necessary, but it must also be put after "project" command.

The next dev build will change how surface highlight works. Also, the old code for surface highlight will be completely removed in some near future.

I don't think it is OFF vs FALSE. Cmake has Off/False/0/No. It would be hell if they were not equivalent. I have committed CheckPIESupported to SVN trunk. I hope it works for you out-of-the-box now.