Jump to content
The Dark Mod Forums

Connection to TDM with automation


Recommended Posts

1 hour ago, greebo said:

Re async: I'd launch the long-running task using std::async, storing the returned std::future in the class to be able to await it if really needed. Within your async method and after the actual work is done, queue an event using GlobalUserInterface().dispatch() - this event will be picked by the regular event loop, it will be executed on the UI/main thread.

Isn't it full-blown multithreading?

Link to comment
Share on other sites

Yes. I don't see any other ways to prevent the UI thread from blocking.

Doing it in some sort of custom event loop will not work, I guess. Any part of the process launching code can take long enough to block things at least a bit.

Link to comment
Share on other sites

I'm trying to add wxActivityIndicator to my dialog.

It seems wxFormBuilder does not have it.
So I try to add it in constructor like this:

GameConnectionDialog::GameConnectionDialog() :
    wxutil::TransientWindow(_(GameConnectionDialog_TITLE), GlobalMainFrame().getWxTopLevelWindow(), true)
{
    loadNamedPanel(this, "GameConnectionMainPanel");

    //could not find activity indicator in wxFormBuilder
    wxActivityIndicator* ConnectedActivityIndicator = new wxActivityIndicator(this, wxID_ANY, wxDefaultPosition, wxDefaultSize, 0, wxT("OMG"));
    replaceControl(findNamedObject<wxWindow>(this, "ConnectedActivityIndicator"), ConnectedActivityIndicator);

As the result, layout of the whole window is dropped: I see empty window with all widgets having the same zero position:

image.png.d4839cbd67eb09c65ec6be4046f4edcf.png

Without the additional two lines about progress indicator, the GUI looks perfectly fine.

What kind of magic I need to invoke to recompute window layout?
Just calling Layout() does not help.

Link to comment
Share on other sites

This cramped appearance usually indicates that the the parent-child relationship of widgets is messed up.

Your code in the constructor to load the XRC-defined panel returns a wxPanel* reference, use that as parent to the indicator:

GameConnectionDialog::GameConnectionDialog() :
    wxutil::TransientWindow(_(GameConnectionDialog_TITLE), GlobalMainFrame().getWxTopLevelWindow(), true)
{
    auto* panel = loadNamedPanel(this, "GameConnectionMainPanel");

    //could not find activity indicator in wxFormBuilder
    auto* ConnectedActivityIndicator = new wxActivityIndicator(panel, wxID_ANY, wxDefaultPosition, wxDefaultSize, 0, wxT("OMG"));
    replaceControl(findNamedObject<wxWindow>(this, "ConnectedActivityIndicator"), ConnectedActivityIndicator);
    ...
}

 

  • Thanks 1
Link to comment
Share on other sites

On 7/31/2021 at 2:31 PM, greebo said:

Re async: I'd launch the long-running task using std::async, storing the returned std::future in the class to be able to await it if really needed.

+1 for this approach. I'm a big fan of std::async. You just put your method in a lambda, create the async with the std::launch::async policy, and that's it. The stdlib handles the threading for you, and all you have to do is keep hold of the std::future which other code can block on to make sure that the background work has completed.

On 7/31/2021 at 3:45 PM, stgatilov said:

Isn't it full-blown multithreading?

Under the hood it is, but you don't have to worry about it particularly. The actual thread creation/destruction is hidden from you. All you have to do is specify what code you want to run in the asynchronous thread (you can also use std::async with the std::launch::deferred argument to perform lazy evaluation without creating a new thread, but that wouldn't help with this problem).

The only thing you have to keep in mind is that the std::future returned by std::async will block in its destructor if the task isn't yet finished — you cannot use it for completely "fire and forget" tasks. This means that you have to store the future somewhere if you don't want the main thread to block, i.e.

// Returned std::future is discarded and will block on destruction,
// this line effectively runs synchronously (but still in a new thread)
std::async(std::launch::async, []() { doSomeCode(); });
doSomethingElse() /* <- this doesn't run until doSomeCode() is finished */

// Future is stored, task runs in background thread while main thread
// continues as long as the future is alive
auto future = std::async(std::launch::async. []() { doSomeCode(); });
doSomethingElse() /* other code in main thread */

 

Link to comment
Share on other sites

I was not aware of std::async and it looks exciting

The only concern I have is how the threads are managed under the hood

I.e. if I start an async Load for a thousand texture files, will it launch a thousand threads or queue execution on a %PHYSICAL_CPU_CORES% thread pool?

 

Link to comment
Share on other sites

I'm pretty sure it's one thread per async (although the lambda in that async could load a thousand texture files if you wanted).

I don't know what happens if you actually create a thousand asyncs (it's probably up to the implementation to choose either individual threads or a reusable thread pool), but that would be a lot of std::futures to manage and I suspect all of the synchronisation might eliminate any benefit of using threads in the first place.

Link to comment
Share on other sites

15 minutes ago, OrbWeaver said:

I'm pretty sure it's one thread per async (although the lambda in that async could load a thousand texture files if you wanted).

Unfortunately, it looks like it depends on the implementation: https://ddanilov.me/std-async-implementations/

In short, MSVC appears to use a thread pool, GCC and LLVM do not (at least at the time of that blog post).

  • Like 1
  • Thanks 1
Link to comment
Share on other sites

If a coder chooses to spawn a thousand std::async tasks, I'd really lean towards calling this a design problem, and no one should really rely on the standard library fixing that. :) It's always the programmer's responsibility to do something reasonable here and partition the work into processable chunks.

 

  • Like 1
Link to comment
Share on other sites

19 minutes ago, greebo said:

If a coder chooses to spawn a thousand std::async tasks, I'd really lean towards calling this a design problem, and no one should really rely on the standard library fixing that. :) It's always the programmer's responsibility to do something reasonable here and partition the work into processable chunks.

 

I still think thread pool would be a nice to have as a part of the STL/libstdc++

Link to comment
Share on other sites

Granted, not saying that an STL thread pool wouldn't be nice to have. :) Maybe something like this will be added at some point. C++ has been moving faster the last decade.

Though one can always look around and pick one of the available thread pool implementations, like this one https://github.com/vit-vit/ctpl This specific one doesn't seem to rely on anything other than STL and doesn't require compiling a static lib, so it seems to be light-weight. I'm sure there are more like those.

Link to comment
Share on other sites

A proper thread pool would be nice, but I suspect we won't see it, because it's considered an implementation detail and the C++ standard leans heavily towards specifying only semantics, not implementation.

For example, as I understand it the semantics of std::async do not require that threads are used at all, only that the call returns a std::future object that the calling code can use to extract the value at a later time. Some platforms might not have a concept of threads, and in this case the async code is just a convenient wrapper for lazy evaluation.

For doing lots of things in parallel, std::async isn't the best choice anyway. It would be more appropriate to use the parallel execution version of std::transform or std::for_each which invokes a function in parallel across a sequence of objects, allocating threads as necessary for the implementation. This is available in C++17 on Windows and Linux (although on Linux it requires the TBB library as an additional dependency) but unfortunately isn't supported yet on Mac, so any use of this technique in cross-platform applications would need to be abstracted and fall back to sequential execution on unsupported platforms.

(At work we actually use our own thread pool code to solve the Mac problem. Normally home-grown threading stuff is junk, but it actually works well and has high performance. It really is as simple as partitioning up the work, manually spawning std::threads and assigning chunks to them).

Link to comment
Share on other sites

I really have to get more into the parallelism capabilities of std. I knew about std::transform etc., but so far I never had an application that required multithreading where using this made sense. Also, most of the time, I find using openMP or IPP much easier/quicker and it is also supported by VS2005, which we somehow still have to support (at work) for many of our modules... 😞 

However, std::async somehow completely slipped through my web. Interesting concept.

Link to comment
Share on other sites

11 hours ago, duzenko said:

I was not aware of std::async and it looks exciting

The only concern I have is how the threads are managed under the hood

I.e. if I start an async Load for a thousand texture files, will it launch a thousand threads or queue execution on a %PHYSICAL_CPU_CORES% thread pool?

Hey, you already have jobs system in TDM!
It is most likely superior in many ways, and it won't turn the game into slide show in Debug (unlike STL under MSVC).

Link to comment
Share on other sites

I have implemented restart without multithreading.
Indeed, splitting sequental function into 3-4 blocks is not nice, but I still don't like the idea of threads.

Moving only "Restart Game" to a new thread is not a good idea, because it will share a lot of data with other functionality --- basically, the whole connection and everything surrounding it. If making a new thread, then it should be persistent worker thread for everything in game connection.

However, there are many things going on in GUI thread:

  1. Callbacks for map loading/unloading, map saving, camera changes, entity changes.
  2. Public methods called due to user clicking GUI widgets.
  3. Signals posted for GUI dialog to update itself.
  4. Updating DR camera directly.

And with multithreading, all of this must be handled somehow.

Just putting a mutex on the whole class won't work: it will block GUI anyway as soon as user does at least something (most likely even if he moves the camera). Another alternative is to deactivate all GUI so that user can't click it.

But both these options sound like false responsiveness to me: yes, user can hover mouse over widgets, but cannot do anything. True responsiveness is about allowing things to happen in parallel and thoroughly thinking out what can happen, and writing additional code to support it. I have done it for continuous camera sync. For restart game, I'm eager to do it too. But for the rest of short-lived tasks, blocking sounds way better.

By the way, I don't think I can even stop the user from moving camera or changing entities in a plugin.

Link to comment
Share on other sites

I have a new version to show.
It is in gameconnection branch in my Github fork: https://github.com/stgatilov/DarkRadiant/tree/gameconnection

The main changes:

  1. Added transient GUI window for game connection (removed menu items).
  2. Implemented "Reload Game" button, including dmap.
  3. Major refactoring: basic code for sending requests extracted into AutomationEngine.
  4. Added hack to disable recalculateBrushWindings for hot-reload diffs (sorry).

I don't know where to put the button to toggle GUI window on/off, so I have left it in the Connection menu.

Also, I did not draw icons for the two buttons (respawn selected and pause). I can do it in Inkscape, but I feel that someone who has experience in drawing icons should better to it. If you want me to draw them, just say so.

P.S. With 6/12 cores/threads, unlimited parallel build (12 x 12), and only 16 GB ram, I regularly get out of memory internal errors from compiler when building Release. Hope that's only me.

  • Like 1
Link to comment
Share on other sites

1 hour ago, stgatilov said:

Also, I did not draw icons for the two buttons (respawn selected and pause). I can do it in Inkscape, but I feel that someone who has experience in drawing icons should better to it. If you want me to draw them, just say so.

No problem at all. You can just stick letters on them for now if you want, and I'll do an icon pass once the GUI is finalised.

1 hour ago, stgatilov said:

P.S. With 6/12 cores/threads, unlimited parallel build (12 x 12), and only 16 GB ram, I regularly get out of memory internal errors from compiler when building Release. Hope that's only me.

As a general rule, I tend to allow at least 2 GB per compile process (since my machine is particularly memory-sensitive, and actually completely hard-locks with not even mouse pointer movement if it runs out of RAM), so I wouldn't generally use more than 6 compile processes even though I theoretically have 12 logical cores available.

If it's one or two things using particularly high amounts of RAM, it might be possible to refactor the code to avoid memory spikes. We've done this in the past at work by using the Pimpl idiom for a particularly memory-heavy class.

Link to comment
Share on other sites

6 hours ago, OrbWeaver said:

As a general rule, I tend to allow at least 2 GB per compile process (since my machine is particularly memory-sensitive, and actually completely hard-locks with not even mouse pointer movement if it runs out of RAM), so I wouldn't generally use more than 6 compile processes even though I theoretically have 12 logical cores available.

If it's one or two things using particularly high amounts of RAM, it might be possible to refactor the code to avoid memory spikes. We've done this in the past at work by using the Pimpl idiom for a particularly memory-heavy class.

They eat dozens of megabytes each.
The problem is that Visual Studio has two independent parallelism options, which results in 144 threads instead of 12.

By the way, Pimpl won't help you with templates.

Link to comment
Share on other sites

Linux build on Github CI fails on my fork:

[ 35%] Building CXX object plugins/dm.gameconnection/CMakeFiles/dm_gameconnection.dir/GameConnectionDialog.cpp.o
/home/runner/work/DarkRadiant/DarkRadiant/plugins/dm.gameconnection/GameConnectionDialog.cpp:8:10: fatal error: wx/activityindicator.h: No such file or directory
    8 | #include <wx/activityindicator.h>

It seems that activity indicator is simply missing in the version of WxWidgets installed on CI machine, i.e. WxWidgets is rather old there.

Is it a big problem?

UPDATE: CMake says 3.0.4 is installed, but activity indicator was added in 3.1.0.

Link to comment
Share on other sites

That's a problem I regularly stumble over. wx 3.1 has many nice advancements, and I more than once had to add ugly preprocessor switches to the code to fall back to some ugly workaround to be able to compile against 3.0.x. Nothing we can do about that, I'm afraid, other than to look out for things that are only available in 3.1.x.

It's a rant of course, but I've been using wxwidgets 3.1 in Windows for years and years now and it's stable as a rock, but Linux distros are still stuck with the older 3.0.x because it's not officially realeased as stable (they are all waiting for 3.2.0, I guess).

wxWidgets 3.1.0 has been released in 2016, with the following text

Quote

Please notice that while 3.1.0 is officially a “development” version because it is not fully compatible with 3.0.x, the list of backwards incompatible changes is very short, so you shouldn’t have any problems updating to this version from 3.0.x in practice, and you’re encouraged to try this version out and use it in production in spite of its odd version number.

In 2018 the text changed to "and you’re encouraged to use this release, including in production". But of course, as long as they don't call it final the distros are holding back.

From the looks of it, things will clear up rather soon, because the wxWidgets folks want to release 3.2 in autumn, so keep your fingers crossed that distros catch up on the release until Spring 2022.

  • Like 1
Link to comment
Share on other sites

Yes, but I'd like DR users to have this GUI before Spring 2022 😉

Should I provide some sort of replacement under #ifdef?
For instance, show an image with still activity indicator instead of spinning indicator.

Link to comment
Share on other sites

21 minutes ago, stgatilov said:

Yes, but I'd like DR users to have this GUI before Spring 2022 😉

Oh, in that case... ;) 

Finding a replacement for the 3.0.x scenario is the only way to go, I'm afraid. The #ifdef is totally up to you, if you think the replacement is really inferior to the 3.1 version, then let's do the #if version check. If the workaround is equally good, then we can just use that one alone. If it's a complex widget, we can also introduce a convenience type to wxutil, not sure if that's worthwile here.

I believe there's a wxGauge progress bar widget available in 3.0 that can indicate work without actually showing the state of completion. But it needs to be Pulsed regularly to be animated: https://docs.wxwidgets.org/3.0/classwx_gauge.html

(I just checked the wx roadmap, they plan to release 3.1.6 before 3.2.0, and so far there's no sign of 3.1.6. So I wouldn't bet any money on Spring 2022 anyway...)

Link to comment
Share on other sites

I fixed the build on Linux CI.

Basically, if wxUSE_ACTIVITYINDICATOR is not set to 1, then wxStaticBitmap is used. Its png image was created from screenshot of a working activity indicator on Windows, so it should look similar. But it won't rotate 😀

Link to comment
Share on other sites

Every time I see my annoying non-updating OpenGL previews (texture browser and media browser) I wonder if we should just bite the bullet and integrate our own version of wxWidgets, rather than relying on whatever ancient version the distros are shipping.

It's not a small task I'm sure, and we would probably need a Git submodule (which we decided against using for Eigen), but if wxWidgets version problems are going to get in the way of future development, maybe it's a price worth paying.

Link to comment
Share on other sites

I don't want to push it...
But are there any plans about integrating my latest changes?

I have finished working on game connection for now, at least on DR side.
In my opinion there are several TODOs remaining:

  1. Now "Connection" menu contains only one item, which spawns the new window. Is it OK? Should it be moved somewhere else?
  2. There are no icons for "pause" and "respawn" buttons. I guess I'll try to draw them myself (I use Inkscape a lot myself). It is obvious how to draw "pause", but what to draw on "respawn" icon?
  3. It would be nice to have some verbose words about the new GUI window. Should it be in tooltips or in manual/help? Should I try writing them (judging from feedback over wiki page, I'd probably avoid it) ?
Link to comment
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.

Guest
Reply to this topic...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

 Share


  • Recent Status Updates

    • freyk

      Some years earlier i created some launchers for TDM. uploaded today some new sourcecode and executables on my github repo.
       
      · 0 replies
    • Epifire

      Say, I know it's been a while since the site got overhauled from the crash. But did we ever figure out if/how to get the recent topics & replies list back? It's not a total deal breaker but it was nice for becoming a thread creeper again...


      · 2 replies
    • Epifire

      Some of you who've been on the TDM discord know I was out of work last Winter, just putting in hrs for the developer portfolio. Currently I've been working a seasonal job to pay the bills and now I'm finally in my last week before I get bumped off. Things will be tight but I'm planning a long off period to make as much content as I can. Big plans in Unreal Engine as well as my most ambitious TDM collab yet! Never been so excited to be a stuck at home to pursue my life's work. With a lotta time and maybe some luck, I'm hoping to get enough art work done that I may start applying around to studios.
      · 4 replies
    • STiFU

      I finally got around to play Prey and I truly loved it. It is an incredible homage to System Shock and Deus Ex. While the gameplay is not en par with those two titles, the game makes up for that with its well written lore and story. The whole "world" just feells so authentic and it features a ton of really god environmental story-telling. Recommended for every immersive sim fan, i.e., everyone on this forum.
      · 2 replies
    • jaxa

      Alder Lake has arrived:
      https://forums.thedarkmod.com/index.php?/topic/18055-2016-cpugpu-news/page/15/&tab=comments#comment-466190
       
      · 0 replies
×
×
  • Create New...