Jump to content
The Dark Mod Forums
Sign in to follow this  
grayman

DR and its shifting planes

Recommended Posts

Today I moved back to a part of my map that I hadn't visited in a month or so, and I found this:

 

post-3633-0-66769600-1355273053_thumb.jpg

 

Is anyone working on DR these days?

 

If so, can we get a discussion started here on how to correct this nagging bug? The *.map format needs to stay the same, but I want to investigate if there's some interim format we can move DR to that will eliminate plane shifting. Just tell DR to output a *.map file when you're ready to build and test, but keep the working data in a separate file that doesn't cause this problem.

 

I'm not looking forward to spending hours having to re-address all the planes in my map that DR has shifted on me. I started creating models for some of the more problematic architecture that DR kept tearing apart every couple days, but I can't make--nor should I have to make--all problematic areas into models.

 

Thanks.

  • Like 2

Share this post


Link to post
Share on other sites

I haven't actually seen this before. Is this shifting a whole brush a few units in a direction? Does it happen when it saves the .map and burps on the vertex locations or something like that? And is it usually complex brushwork?


What do you see when you turn out the light? I can't tell you but I know that it's mine.

Share this post


Link to post
Share on other sites

I'm sure this creeps up quite often but doesnt get noticed all that often, or overlooked as a stray keystroke. BD might run into it, all that fiddly work :)

 

Did you by any chance edit the map in the engine and then save it out? (i.e when working with ragdolls or something).

 

Other than that, was it aligned to grid, or is it perhaps a rounding bug?

Share this post


Link to post
Share on other sites

When DR writes the *.map file, its internal representation of a plane is converted to a text string that describes the plane using the plane's normal and the shortest distance from the plane to the origin. (Or, in the case of func_statics, the shortest distance from the plane to the func_static's origin.) A brush is made up of several planes.

 

Converting from the internal representation to the text string and back doesn't necessarily yield the same plane, because of rounding errors. For example, here's one plane definition from the brush that's showing the most caulk in the picture above:

 

( 1 0 0 -91.93686676025391 ) // the normal is in the +X direction and the origin is 91.93686676025391 units behind the plane's face

 

If I correct the brush and put it back where I first drew it many moons ago, that plane definition reverts back to what it was originally:

 

( 1 0 0 -92 )

 

So, over many writes and reads of this *.map file, that original "-92" (the shortest distance from that plan to the func_static's origin) migrated to "-91.93686676025391" because of the read/write rounding error. (Now, one would think that it would be easy to hang onto an integer like "-92", but DR works with floating point numbers, which can suffer rounding/precision errors that build up over time.)

 

Since the points of a brush are defined by the intersections of the brush's planes, when a plan shifts like this, the points shift. So points of adjoining brushes that are supposed to have the same XYZ values no longer do, giving results similar to my picture.

 

Since all mappers are advised to "keep points on the grid", one would think that point data is more important than plane data. But that's not how the *.map file keeps things, at least for some brushes.

 

So my reason for this thread is to discuss possibilities for avoiding this read/write problem, to maintain the integrity of what we originally designed so that we don't have to be constantly putting things back where they belong.

 

When I was working on In the North, I spent many hours correcting these problems, and did so along the way. I went so far as to replace all the detail in the prefab cabinet poles with simple cylinders so my wife wouldn't have to listen to me grumbling about it all the time.

 

For Home Again, I decided I would wait until the very end before going back and making corrections, to reduce the amount of time wasted on this. But then I spotted what you see in the picture above, and decided we really shouldn't have to live with this if some solution could be reached.

Share this post


Link to post
Share on other sites

This is a REAL problem and it needs urgent response! I started making detail stuff into .ase models to protect them from corruption. This is clumsy and not suitable solution for everything.

 

And this is not paranoia: I see the stuff degradating: every now and then I see func_static objects no longer fitting snugly into the place they sat in. Selecting the object in DR, reverting it into worldspawn and snapping it back into grid causes a visible but subtle change in the object even when it was originally built flawless.

 

Please, DR wizards, can you have a look at the problem? Mapping is a lot of work and I genuinely fear that budding mappers would cast mapping aside due to frustration of steady degradation of their work.

 

I personally find this one of the biggest bug-formed threats against TDMs success at the moment. From a mapper point of view, as an artist, there is a huge negative emotional impact of seeing your hard work crumbling in front of your eyes. We don't want to associate that negativity to TDM mapping.


Clipper

-The mapper's best friend.

Share this post


Link to post
Share on other sites

I addressed the problem over a year ago but I think it's gotten worse...It feels like my caulking of unseen faces over the years is useless as the pink caulc keeps creeping into view and results in strange artefacts in game...

Share this post


Link to post
Share on other sites

Could you have a parallel file that just holds values, and on saving, if a brush wasn't tagged as changed (& maybe a changed value is within a certain radius, but that maybe redundant), then it restores the saved values, and if the brush is changed it updates the values -- Something in that neighborhood?


What do you see when you turn out the light? I can't tell you but I know that it's mine.

Share this post


Link to post
Share on other sites

Judging from the commit history on github, I'd assume we can't expect a fix very soon. The only contributions were by angua and they were months apart. Let's hope Greebo or Orbweaver drops in... I am absolutely not familiar with the map code, so I'd probably mess things more up than they currently are. Plus, all my programming time currently goes into work, so I am out of the equation anyway.

Share this post


Link to post
Share on other sites

Some thoughts on things that have come up thus far:

 

It seems that certain type of objects are prone to the rounding errors, usually those containing small detail brush/patchwork features or 'strange' planes as mentioned by grayman. Normal blocky sealing geometry is never affected, at least in my experience. My qualitative feel is that some detail objects were quickly corrupted and some were not corrupted at all.

 

In order to avoid the corruption, use the .ase exporter to make detailed objects and other objects that are easily corrupted into models. That way you will never have to redo them. Also reusing those pieces is easier as you can always insert a new model. It saves memory too if the object is a recurring one.

 

Thus my personal recommendation is that, while we wait for the fix, the mappers should make .ase models in order to avoid the need to readjust some objects again. Making an .ase model takes only a few seconds, so it is a worthy change in your workflow if it liberates you from the need to repair the object later. Just make sure you design the object properly so once you 'permanently glue' it into a model, you do not need to change it anymore. It is probably a good idea to store the func_static source versions of the model in a separate mapfile that you access seldom.

 

Looks like the object corruption bug will gently lead mappers to take a small step towards the modular building principle, eh?


Clipper

-The mapper's best friend.

Share this post


Link to post
Share on other sites

Looks like the object corruption bug will gently lead mappers to take a small step towards the modular building principle, eh?

I just thought the same when reading this thread :smile:

 

IMO this isn't really a rounding error problem, cause numbers like 92 mentioned by grayman can be described exactly by floating point numbers

as far as I can conclude from what was written here there seem to be two major problem sources:

 

- the conversion floating-point number -> text -> floating-point number on save/load definitely causes errors, I had this in a couple of C++ programmes I had written in the past (but that's a conversion, not an rounding problem); this could get avoided by saving the specified informations in a number format (binary)

 

-

the points of a brush are defined by the intersections of the brush's planes

this is a very, very terrible idea; as Sotha mentioned this whole problem mainly occurs on "non-blocky" geometries, so geos where the planes normal may not all be orthogonal to each other or to the coordinate system; especially if the way those points are evaluated is not fixed but can change if you e.g. delete a brush or so this may also cause lot's of trouble here; the intersecting points / lines should be saved seperately, but this will of course enlargen the map file (and maybe the proc, don't know what kind of information goes in there)

 

I'm not a very good programmer and don't have the time to look at the code anyway

 

I convert complex geometries into func-statics, independent if they'll stay this in the end,

if I can see shifting problems, I just convert them to worldspawn, choose an appropiate grid-size and snap it to grid, then I convert it back to FS

(and if that doesn't work I plug my fingers into my ears and start humming)


FM's: Builder Roads, Old Habits, Old Habits Rebuild

WIP's: Several. Although after playing Thief 4 I really wanna make a city mission.

Mapping and Scripting: Apples and Peaches

Sculptris Models and Tutorials: Obsttortes Models

My wiki articles: Obstipedia

Texture Blending in DR: DR ASE Blend Exporter

Share this post


Link to post
Share on other sites
  • Is anyone working on DR these days?
  • I'm not looking forward to spending hours having to re-address all the planes in my map that DR has shifted on me.

  • Yup.
  • I thought this only effected FS models made by the mapper or groups of brushes that were being rotated or moved..?

Share this post


Link to post
Share on other sites

Just in case it got lost in getting ready for 1.08:

 

http://forums.thedar...post__p__291091

 

 

I can push these changes into the actual one on github once I get a confirmation that my build helps to solve the floating point problems

 

Did this not solve the issue?

Share this post


Link to post
Share on other sites

I know this isn't ideal but if you need to use models yet also need to seal the void with

the same geometry you can "inline" models:

 

http://www.doom3world.org/phpbb2/viewtopic.php?f=1&t=25941

 

which will convert the models to brushes at map load.

 

Many reservations about how robust that is but it might offer a lesser of two evils...


Please visit TDM's IndieDB site and help promote the mod:

 

http://www.indiedb.com/mods/the-dark-mod

 

(Yeah, shameless promotion... but traffic is traffic folks...)

Share this post


Link to post
Share on other sites

Biker, this has nothing to do with moving objects. The objects get corrupted without the mapper touching them at all.


Clipper

-The mapper's best friend.

Share this post


Link to post
Share on other sites

Sotha: You mean it fixates the position of the individual brushwork? How do one get about the .ase exporting?

 

Yeah, it exports the selection you have into a model. The model cannot be changed anymore so it is DR object corruption bug immune. Downside is you need to export it again if you want to change it later.

 

Choose the whole object. Check the "scripts/export .ase" or similar in the top menu. Check the option "0,0,0 center origin <or something like that>". You do not need to export caulk.

Then select a folder you save the model. darkmod/models/darkmod/Fieldmedic or something. The first "darkmod" folder is the one where you have thedarkmod.exe. Save the .ase model in your own folder so it is easily found. Then you can either restart DR so that the model list is rebuilt or choose "file/reload models" (or something like that).

 

Then in DR normally right click -> "create model" and you should see your newly created model in the models list: models/darkmod/Fieldmedic. When you release the mission, you of course need to package the models/darkmod/Fieldmedic folders into the .pk4 too otherwise your models will not be found.

 

Easy, fun, useful!

  • Like 2

Clipper

-The mapper's best friend.

Share this post


Link to post
Share on other sites

Nice! If I'd known that when A Night To Remember was produced it would have saved me lots of headaches and frustration when creating the grand piano...

Share this post


Link to post
Share on other sites
Biker, this has nothing to do with moving objects. The objects get corrupted without the mapper touching them at all.

Oh dear, that is going to be a right fricken pain in the bloody arse that is..

Share this post


Link to post
Share on other sites

Today I moved back to a part of my map that I hadn't visited in a month or so, and I found this:

 

post-3633-0-66769600-1355273053_thumb.jpg

 

Is anyone working on DR these days?

 

If so, can we get a discussion started here on how to correct this nagging bug? The *.map format needs to stay the same, but I want to investigate if there's some interim format we can move DR to that will eliminate plane shifting. Just tell DR to output a *.map file when you're ready to build and test, but keep the working data in a separate file that doesn't cause this problem.

 

I'm not looking forward to spending hours having to re-address all the planes in my map that DR has shifted on me. I started creating models for some of the more problematic architecture that DR kept tearing apart every couple days, but I can't make--nor should I have to make--all problematic areas into models.

 

Thanks.

 

The entire problem stems from this fact:

 

When you load a .map file, the "plane equations" are converted to vertices for the brush (by intersecting all planes with each other). When the map is written back, the vertices are converted back to plane equations again.

 

Since DR uses floating point numbers, this is a lossy operation. Each load/save cycle will slightly alter the equations and thus the brush. Even if DR used doubles, this would not change the underlying issue (the error is smaller, but it is still there). Rounding the number upon save will also not really help, except in some cases.

 

https://en.wikipedia...on_and_rounding

 

The only real solution is to save the original plane equations (as written in the map file) upon loading, and when saving a map, write them back unmodified for brushes that where not touched. (for brushes that have been touched have and can of course be regenerated).

 

(Note: Swift Mazes does not have this probem because it always compiles the map anew, but if you open/save the map with the prefabs alot, you run into the same problem :(


"The reasonable man adapts himself to the world; the unreasonable one persists in trying to adapt the world to himself. Therefore, all progress depends on the unreasonable man." -- George Bernard Shaw (1856 - 1950)

 

"Remember: If the game lets you do it, it's not cheating." -- Xarax

Share this post


Link to post
Share on other sites

No response from the folks who've most recently worked on DR.

 

So, if I can get DR to build, I'm going to investigate the problem.

Share this post


Link to post
Share on other sites

I don't know if greebo or angua frequent the public forums very much (and Orbweaver rarely visits the forums at all anymore) so if you want their input you may need to pm them.

Share this post


Link to post
Share on other sites

I've been able to change DR to use a brush definition where faces are represented by 3 points (vertices) rather than the normal/distance plane equation.

 

Back in idTech3, this is how brushes were represented in map files. For idTech4, id changed the brush definition to use the normal/distance plane equation.

 

I've done no testing yet to show that plane shifting no longer occurs, so I don't know if this is a solution to the problem.

 

I have a PM in to greebo to discuss the merits of using the older format in interim DR map files and exporting the newer format only when it's time to build. No word yet.

 

If anyone knows why id changed the format from the brushDef syntax (3 points) to the brushDef3 syntax (normal/distance), I'd love to hear it. I can't find anything on the net about it.

Share this post


Link to post
Share on other sites

Maybe it's for saving space


FM's: Builder Roads, Old Habits, Old Habits Rebuild

WIP's: Several. Although after playing Thief 4 I really wanna make a city mission.

Mapping and Scripting: Apples and Peaches

Sculptris Models and Tutorials: Obsttortes Models

My wiki articles: Obstipedia

Texture Blending in DR: DR ASE Blend Exporter

Share this post


Link to post
Share on other sites

No idea why they changed it, or it saves space or is easier to handle. Who knows. There isn't even any real documantion on the new or old formats except what some people figured out on the net.

 

The problem is, if you are still using "plane equation => 3 points => plane equation", you will still have floating point errors and the problem will still occur. You have to think of the common situation of "on morning, load map, on evening save" and multiply this a few hundred times.

 

Having the map stored in an interim format and then have to "export" it for the game would just make it more cumbersome for the mapper, and still not really solve the problem.


"The reasonable man adapts himself to the world; the unreasonable one persists in trying to adapt the world to himself. Therefore, all progress depends on the unreasonable man." -- George Bernard Shaw (1856 - 1950)

 

"Remember: If the game lets you do it, it's not cheating." -- Xarax

Share this post


Link to post
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.

Guest
Reply to this topic...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

Sign in to follow this  

×
×
  • Create New...