[2.09] Hot reload feedback

[stgatilov 1225]

Now that TDM 2.09 has gone beta, it is possible to use "hot reload" for maps, i.e. upload map changes into the game without restarting it.
I'm creating this forum post to gather all the discussion and feedback regarding it, including the DarkRadiant side of things.
 

I have written the wiki article about it, though it is rather technical and probably more suited to people who want to understand the new system well.
The instruction how to use it is with DR is in the DarkRadiant section:

1. Start TheDarkMod game.
2. Select the FM and start the map you want to work with.
3. Open game console and ensure that com_automation cvar is set to 1 (i.e. automation is enabled).
4. Start DarkRadiant.
5. Select the FM and open the map --- that same ones that you opened in TDM.
6. Now you are ready use any features from the "Connection" menu.

I suggest trying the "Enable camera synchronization" mode first.
Among numerous hot reload settings, "Enable map update mode" is surely the most advanced.

Here is one video I made for hot reload:

Spoiler

 

 

The current GUI in DR is pretty bad. Ideas regarding it are welcome.
Right now the idea is to create a persistent window for all "Game Connection" stuff, looking approximately like this (LEDs can cause problems though):

Spoiler

And if you manage to try all the hot-reload modes and think that some of them are unnecessary, please say so.
There is major feeling that exposing both "map update" and "map reload" is probably superfluous.

[peter_spy 1603]

Sorry for double post, writing on the phone here.

Edited December 15, 2020 by peter_spy

[peter_spy 1603]

In addition to what has already been said here, that GUI poposition is also quite bad, UX-wise. These red LEDs look like recording buttons, and they're before the green ones, if anything , they should be after. I'm assuming you want to have 2 or 3 of them, although IMO one with changing states will be more readable. And, while connected and camera sync make sense, LEDs for reload map and update map look confusing, as these seem like some command names. I presume these option names should be more intuitive, or simplified. Also, yellow color for "enabled" is really weird.

Maybe the whole traffic light analogy is bad here. IMO it needs reworking to something simpler, like On/Off button or slider per each option, and maybe fewer options.

And in general, as automation feature, so far this actually offloads quite a bit of work to mappers in order to work properly. And mappers have enough on their plate already, and they're just people, they forget and confuse things. The fewer steps and options to toggle here, the better. The idea as such is great, but so far it's too implementation-driven and not enough user-friendly, I think.

Edited December 15, 2020 by peter_spy

[HMart 334]

I assume this is exclusive for TDM engine because of that game console cvar. Petty, I don't use TDM engine but would certainly love this feature.

[MirceaKitsune 258]

Didn't think mapping could be even further improved, yet this feature sounds like a great way to do it. Thank you for another great idea!

So if the player modifies the map in DR and saves it, they can automatically alt-tab switch to TDM and see the changes? First issue that pops to mind is how adding or removing areas may cause the player to get stuck inside walls or left floating / falling in the void, unless there's a smart way to know how to reposition them. The rest I'm assuming solutions have been found for... like caching newly added textures in realtime, I know the engine wants every asset to be loaded at map start.

Something that would be equally useful, perhaps even more if not as much as this change: Could DarkRadiant offer a button to launch TDM on the map currently being edited? So each time you want to test you don't have to manually open TDM then type the name of the map in the console, just hit one button and it all happens for you.

[stgatilov 1225]

3 hours ago, HMart said:

I assume this is exclusive for TDM engine because of that game console cvar. Petty, I don't use TDM engine but would certainly love this feature.

Yes, it is TDM only. There is quite a piece of "automation" code I had to add.

1 hour ago, MirceaKitsune said:

So if the player modifies the map in DR and saves it, they can automatically alt-tab switch to TDM and see the changes?

Sort of yes, depending on settings. One mode updates on save, the other one just updates constantly.

Quote

First issue that pops to mind is how adding or removing areas may cause the player to get stuck inside walls or left floating / falling in the void, unless there's a smart way to know how to reposition them.

Oh yeah, I forgot to write it: the hot reload feature is for entities only!
If you change something that needs dmap (adding rooms, portals, etc.), it won't work properly. So if you add a wall in editor, it simply won't appear in the game. I think the most that you can do is to delete some func_static stairs, and guards will probably fall and get stuck there.

Quote

Something that would be equally useful, perhaps even more if not as much as this change: Could DarkRadiant offer a button to launch TDM on the map currently being edited? So each time you want to test you don't have to manually open TDM then type the name of the map in the console, just hit one button and it all happens for you.

You are not the first one with this idea
If people keep saying it, it will be implemented.

[MirceaKitsune 258]

Cheers. A little suggestion then if it's possible: When a geometry change is detected... could there be a mode that automatically pauses the game, runs dmap, then unpauses it on the updated version? Just like running dmap while being connected, but instead of ending up in the main menu everything seamlessly resumes once the map has recompiled. Only thinking of this for completions sake... I can't say I see myself using even the live entity updates so this offers more than enough in terms of helping mappers

[stgatilov 1225]

13 minutes ago, MirceaKitsune said:

A little suggestion then if it's possible: When a geometry change is detected... could there be a mode that automatically pauses the game, runs dmap, then unpauses it on the updated version?

Too hard to implement: the world areas, portals, AAS go too deep into the game to update them.

Right now the only viable idea for updating brushes is to display added brushes as models, without changing portals, areas, and AAS.
I was also thinking about making dmap incremental (faster runs after small changes), but I'm not yet sure I really want to attempt this

P.S. Added a video in the first post.

[jonri 50]

15 hours ago, stgatilov said:

Too hard to implement: the world areas, portals, AAS go too deep into the game to update them.

Right now the only viable idea for updating brushes is to display added brushes as models, without changing portals, areas, and AAS.
I was also thinking about making dmap incremental (faster runs after small changes), but I'm not yet sure I really want to attempt this

P.S. Added a video in the first post.

I suppose the other alternative is to save the player's position, dmap and reload the map, and put the player back into the same position.  You'd lose anything else that had progressed over time, but if you're just blocking out your map that's not as important.

Of course, this approach could have side effects if you've drawn a brush over the player's position, but not much worse than what you'd get if you added the brush as a model.  The model approach wouldn't let you delete brushes either.

 

[jonri 50]

@stgatilovWhat do you think about having the hot reload also reload models, materials, skins, scripts, etc when the same is done in DR?

[peter_spy 1603]

I'd go even further and do that as soon as the texture, model or declaration file is updated on the hdd. Unity works like that, and it's great. Right now testing materials means continuous alt tabbing and left hand pain.

Edited December 30, 2020 by peter_spy

[kingsal 591]

45 minutes ago, peter_spy said:

I'd go even further and do that as soon as the texture, model or declaration file is updated on the hdd. Unity works like that, and it's great. Right now testing materials means continuous alt tabbing and left hand pain.

@jonri @peter_spy @stgatilov I could see auto updating models and textures, but I would be cautious about auto updating definition files and scripts without user confirmation. There might be cases where the author needs to touch a few different files before reloading the definition in game or simply wants to save their progress while writing a script. When hotReloading this stuff, the engine might grab incomplete data an cause errors while the user isnt looking. For instance you bind an object to an AI in a def file, but you havent created that object yet- the engine will try to grab a null entity and possibly crash in some cases.

[peter_spy 1603]

I'd be cautious about scripts too, because if there's an error in the script, you'll be thrown out of the map. But decls should be fine. Typically for errors in decls will just result in a console message, and e.g. model or material going black. You'd have to debug it the same way, when all steps are manual.

IIRC, def files need both reloaddecls command and actual reloading the map, so there will be kind of two steps before a change is reloaded in TDM: saving a txt file and issuing a map yourMap command.

Edited December 30, 2020 by peter_spy

[jonri 50]

1 hour ago, peter_spy said:

I'd go even further and do that as soon as the texture, model or declaration file is updated on the hdd. Unity works like that, and it's great. Right now testing materials means continuous alt tabbing and left hand pain.

I've got a proof-of-concept of this working in DarkRadiant right now.  I'm not sure it would be as easy to do directly in TDM due to cross-platform stuff that wxWidgets takes care of in DR.  Having this feature combined with the hot-reload, though, would provide the same end result as long as DR was running to relay the reload command.

[peter_spy 1603]

That would be a fantastic feature for making models and materials for TDM. Imagine having Gimp or text editor on one monitor and TDM on the other, and the surface updating as soon as you save the .mtr file, or export an image. That would speed up the iteration a lot.

[stgatilov 1225]

9 hours ago, peter_spy said:

That would be a fantastic feature for making models and materials for TDM. Imagine having Gimp or text editor on one monitor and TDM on the other, and the surface updating as soon as you save the .mtr file, or export an image. That would speed up the iteration a lot.

It exists in some sense for 2 years already.
But without proper release scheme (neither part of TDM not part of DR) and requiring Python to be installed.

I use it most of the time to edit shaders, and it supports models and materials just as easily.

 

[stgatilov 1225]

I'm looking at various complicated cases when mapper has to manually respawn entity in order to update some spawnargs.
For instance, various light properties depending on lit/extunguished, LOD settings.

Right now the implementation has hardcoded blacklist of spawnargs. If any of them is changed, then the entity is respawned afresh. If none of them are changed, then some fixed updates are performed directly. Currently the following direct updates are supported: "origin", "rotation", "model", "skin", and some light properties.
The funny fact is that there is myriad of spawnargs which are not handled properly and which are not yet in the blacklist; changing these don't work currently. On the other hand, respawning the entity manually solves the problem perfectly in most cases.

I wonder if I should replace blacklist with whitelist.
I.e. respawn any modified entity, unless all the modified spawnargs are the ones which can be handled directly (listed above).

[stgatilov 1225]

Another alternative is to move this tweaking to def/script area.
It can work like this.

1) If entity has spawnarg "hr_respawn_all" "1", then any modification to this entity leads to its respawn.
This setting would be useful for static entities, which don't move around and don't have any sort of memory (func_static, func_emitter, func_speaker, ...). Respawning them is always a good idea, since the fresh entity is indistinguishable from the old one anyway.

2) If entity has spawnarg "hr_respawn__s_shader" "1", then modification of "s_shader" spawnarg leads to entity respawn.
This can be added to base classes for spawnargs which are known to require respawn. The obvious downside is that there would be much more spawnargs like this.
Also, I guess spawnargs like "hr_respawn_prefix__set " "1" are needed in order to respawn when "set XXX on YYY" is modified.

3) If entity is not respawned, then I can call method "on_hot_reload" on entity's script object (if it exists). In addition to direct modification of origin, rotation, and ordinary model and skin.
Hopefully, someone can write this method for TDM lights with their _lit/_extinguished properties, so that they could be updated without respawn. Perhaps it could be added to AIs too.