Xml Registry Upgrades

[OrbWeaver]

With the 0.8.1 release, I have changed the default size of the ModelSelector's preview widget from 40% of screen height to 60% of screen height. Unfortunately, I think users probably will not pick up this change without manually editing their user.xml, because their local copy will cache the 0.4 value from the original installation. This could be a problem for future upgrades.

 

The problem

The local copy of the user.xml always takes precedence over the installation copy, which is correct behaviour since the local copy may contain custom modifications. The problem is that there is no way of distinguishing between values that have been changed by the user, and those that have been copied verbatim out of the installation version. This means that new versions of DarkRadiant cannot change default settings, even if these have never been changed by the user, because the local copy contains a complete cache of the previous version.

 

Possibly solution

Maintain two XML trees in memory. One of them (the "installation" tree) contains the entire set of nodes parsed from the installation XML files, and is readonly. The other (the "user" tree) contains the entire set of nodes parsed from the local XML files (which currently only contains user.xml, but the handling is the same), and is read/writable. The structure of the trees is the same, except that the user tree is initially unpopulated (i.e. the first time DR is run). The two trees are used as follows:

 

1. Any time a node is looked up, the user tree is checked first, then the installation tree if it is not found (similar to curent behaviour).

2. When a node is changed, the change is written into the user tree (and created there if it doesn't exist). This means it will be found first in future lookups, and the changed value will take effect.

3. At shutdown, the user tree is written out to local files.

 

Largely, this behaviour seems similar to what happens at the moment, but with a crucial difference -- the local user.xml will ONLY contain values that have been changed from the defaults (because the user tree starts off empty, and is only populated with changed values). It will therefore be possible to upgrade DR and have new default values take effect if they have not been modified by the user.

 

@greebo: Any thoughts? Does this sound feasible to you? I haven't looked very closely at the XML registry code, so I don't know how similar this is to the existing implementation.

[greebo]

It definitely sounds feasible, with some testing this should be implemented quite quickly.

 

Parts of our problems could be (or could have been) solved via the upgradepaths I implemented a while ago, but admittedly not all of them.

 

I could think of two more approaches additionally to the one you suggested:

 

1) Instead of maintaining two separate trees in the memory the written values could be tagged with a "dirty" flag, which will have them exported at DarkRadiant shutdown. This would require a minimal algorithm change at the point where the files are exported, as we would delete all the nodes without the dirty flag. I haven't thought entirely through this, but it may offer some advantages.

 

2) We think about what values are going into the user.xml (and therefore into the local copy) in the first place (and which are not). Definitely there are some values that have to be kept (like last opened maps, for example, or cubic clipping or the background image), and some others can be categorised as "advanced" DarkRadiant tweaking like changing the size of the ModelPreview window or changing the toolbar arrangements. In short, some values are more "worthy" being saved and kept over DarkRadiant upgrades, and some are not (and are more likely to be overwritten at the next release). This would be a distinction of "DarkRadiant State" or "Preferences" (MRU, Cubic Clipping) and "DarkRadiant Configuration" (sizes, toolbar arrangements, etc.).

 

Users that want to fiddle around with configuration settings can do this as well in the main configuration files, as those values aren't accessible via an interface anyways (so it technically doesn't make large a difference which file they edit). If they are willing to mess around in XML files, they might do it a second time as well after they upgraded their DarkRadiant.

 

These are my first thoughts about this, it can well be that a hybrid approach of all the above could be realised and covers most of our problems.

[OrbWeaver]

Parts of our problems could be (or could have been) solved via the upgradepaths I implemented a while ago, but admittedly not all of them.

 

The upgradepaths solution seems rather complex to me, not only due to the amount of code required to parse and execute the upgrade paths file itself, but also the need to potentially maintain an upgradepath section for EACH previous version (for example somebody might install 0.8 over 0.6, without installing 0.7 first).

 

"Design intuition" suggests that there ought to be a simpler way of doing this than with upgrade paths, although I could be wrong obviously.

 

1) Instead of maintaining two separate trees in the memory the written values could be tagged with a "dirty" flag, which will have them exported at DarkRadiant shutdown. This would require a minimal algorithm change at the point where the files are exported, as we would delete all the nodes without the dirty flag. I haven't thought entirely through this, but it may offer some advantages.

 

You certainly could do this, however I suspect the advantage of two trees is that you don't have to merge the user nodes into the installation tree -- presumably you have to do something like this at the moment with a single tree. The two trees idea addresses both the requirement for user nodes to override installation nodes when loaded from disk, and for changed nodes to be separate from unchanged nodes.

 

There's probably not a lot in it though, whichever is easiest to implement is best.

 

2) We think about what values are going into the user.xml (and therefore into the local copy) in the first place (and which are not). Definitely there are some values that have to be kept (like last opened maps, for example, or cubic clipping or the background image), and some others can be categorised as "advanced" DarkRadiant tweaking like changing the size of the ModelPreview window or changing the toolbar arrangements. In short, some values are more "worthy" being saved and kept over DarkRadiant upgrades, and some are not (and are more likely to be overwritten at the next release). This would be a distinction of "DarkRadiant State" or "Preferences" (MRU, Cubic Clipping) and "DarkRadiant Configuration" (sizes, toolbar arrangements, etc.).

 

That was one of my ideas actually -- have "UI configuration" to hold stuff like the model preview size, which does not change during normal usage but can be edited manually, and "UI state" to hold all of the stuff that changes frequently and needs to be saved between DR versions. I don't really like this solution though, because it involves more "prior knowledge" and thus increased complexity; you've probably gathered now my intention is always to simplify, simplify and simplify

[greebo]

Ok, I will try to implement the dual-tree solution then >> opened an issue on the bugtracker.

[greebo]

I've implemented your suggestion as follows:

 

The XMLRegistry has now two RegistryTree instances, namely the standard tree and the user tree.

 

The Registry interface (iregistry.h) stayed nearly untouched, the commands are passed in such way to the subtrees that only the user tree eats the write accesses and the default tree is treated as read-only.

 

A Tree enum value can be passed to the import() method so that the client can choose which tree the file should be imported into.

 

I've tested it with a few settings and so far it seems to work without complaints. Maybe there will some issues come to the surface during the future testing, but the overall behaviour is ok.

 

edit: oh, and I sent the upgradepath code to sleep into a separate file, so it's inactive at the moment. I can reimplement it should we need it in the future. I will leave the existing upgradePaths for the moment being.

[OrbWeaver]

The Registry interface (iregistry.h) stayed nearly untouched, the commands are passed in such way to the subtrees that only the user tree eats the write accesses and the default tree is treated as read-only.

 

Cool, that's exactly how I imagined it would work.

 

A Tree enum value can be passed to the import() method so that the client can choose which tree the file should be imported into.

 

Ah, OK. I didn't think of that -- the registry does not load the files itself, so some means of distinguishing between them is needed from the application code.

 

I've tested it with a few settings and so far it seems to work without complaints. Maybe there will some issues come to the surface during the future testing, but the overall behaviour is ok.

 

That's good to know -- presumably it displays the planned behaviour of writing out a user.xml with only the changed settings?

 

edit: oh, and I sent the upgradepath code to sleep into a separate file, so it's inactive at the moment. I can reimplement it should we need it in the future. I will leave the existing upgradePaths for the moment being.

 

No problem with that.

[greebo]

That's good to know -- presumably it displays the planned behaviour of writing out a user.xml with only the changed settings?

Yes, but don't expect the user.xml to total around 100 Bytes; as soon as you press OK in the Preference dialog all the settings get saved by calling set() and this means they are instantly written into the user tree.

 

I could make one more step and implement a comparison into the set() method that checks if the value to be written is any different from the default value (if that one even exists) and only writes changed values into the user tree. For this to work effectively it would mean that keyvalues matching the default values should be deleted from the usertree - but this could have some "Unforeseen Consequences™" (couldn't resist ) if the deleted node contained any subnodes. The workload for the registry would increase and I don't know if it's worth the hassle...

[OrbWeaver]

Yes, but don't expect the user.xml to total around 100 Bytes; as soon as you press OK in the Preference dialog all the settings get saved by calling set() and this means they are instantly written into the user tree.

 

That's fine, I would expect that to happen with preferences. At least there is now the required distinction between those and the stuff that users probably never change, like the UI layout options.

 

but this could have some "Unforeseen Consequences" (couldn't resist )

 

I think that's going to be the "Macbeth" of the Dark Mod -- if you mention it you have to recite some charm in order to ward off bad luck.