Itches, Glitches & Anything Else...

[OrbWeaver]

I wonder - when I define this external reference, how can Linux users suppress checking out the Win32 dependencies? If this is not possible, everything would be checked out to a Linux working copy too.

 

There is an "ignore externals" option when you checkout. It may be worth setting it up without externals for the time being, since using ignore externals to avoid bringing down Windows dependencies would also prevent the future use of externals for anything that was needed across platforms.

[greebo]

Another thought: Assume a user has checked out the trunk/darkradiant/ folder to C:\DarkRadiant. Were you thinking that the user performs another checkout of the trunk/dependencies/ folder in the same C:\DarkRadiant folder?

 

I seem to recall TortoiseSVN emitting a warning when the user tries to checkout something into an existing working copy.

 

On the other hand, if the user checks out the trunk/dependencies/ to C:\DarkRadiant\dependencies, this would be circumvented.

[OrbWeaver]

Yes, I imagine that the repository layout would be something like:

 

w32deps/gtk2

w32deps/openal

w32deps/boost

 

The user would check out the main repository wherever they liked, and then checkout the w32deps directory into a w32deps subdirectory of their main working copy. The SConscript would then be set up to look for Windows includes/libs in this w32deps local directory.

 

This means that the top-level .svn directory would contain the main repository metadata, and have no knowledge about the w32deps subdirectory which would contain its own .svn directory pointing to the respective URL. I guess an svn:ignore property would be needed on the top-level (main) repository, to avoid it seeing the unknown w32deps directory as a new object and suggest adding it.

[greebo]

Sounds like a good plan, I'll upgrade the boost headers and will look into this afterwards.

[greebo]

Ok, all the dependencies have been moved to trunk/w32deps/.

 

It's a minor hassle that TortoiseSVN doesn't provide the SVN>Checkout option when right-clicking into a folder that is already under version control, so users will have to choose an unversioned parent folder to perform the w32deps checkout. I'll include that information in the compilation guide on the wiki, once it's back up.

[OrbWeaver]

Excellent, that's a much more compact working copy now (on Linux at least).

[greebo]

While refactored the Winding class I remembered what namespace told us about the winding rendering code. I experimented a bit and removed the stack-allocated normals array from the function and made it a class member. In a given scene in the bonehoard the renderer took ~120-140 msec. each frame before my change, now it's 80-90 msec. which is significant.

 

I will commit that change, but it's only a preliminary one. Issues still remain, because

- the Winding presumably eats much more (heap) memory now

- the normals are still copied into that mutable Vector3 array each render pass, which is usually four copy operations per winding = brush face.

- the render routine still uses vertex arrays. Namespace suggested moving this to VBOs, which I'd like to check out.

 

An open question would be how many VBOs I'm allowed to create in a given OpenGL implementation. If the number is limited to 16 bits (65536), for instance, we might get into trouble, as the bonehoard has about 9000 brushes with mostly 6 windings and more.

[greebo]

I managed to setup a proper testcase for the brush rendering. I created a scene with ~10,000 brushes and issued a render call 50 times. This is a profile build (automated test run), 1 orthoview, 1 camera, the camera view frustum includes all brushes.

 

The original winding render code with the stack-allocated normals array has

total time: 137 sec, each frame: 2000 msec

After my change it's this:

total time: 66 sec, each frame: 600 msec

 

Also, the Winding::draw() method crops up at the top of the profile, which is good for testing.

 

edit: Another thing that might be interesting is this:

  %   cumulative   self			  self	 total		   
time   seconds   seconds	calls   s/call   s/call  name	
24.95	  6.29	 6.29 1110656232	 0.00	 0.00  BasicVector3<double>::BasicVector3()

The Vector3 default constructor takes a massload of calls and eats 6 seconds in the old code. So it's official that allocating a strong number of Vector3s is indeed expensive.

[OrbWeaver]

- the normals are still copied into that mutable Vector3 array each render pass, which is usually four copy operations per winding = brush face.

 

A good first step, but going forward we should try to eliminate copying altogether. I don't remember the exact bit of code where this was happening though, or why,

 

An open question would be how many VBOs I'm allowed to create in a given OpenGL implementation. If the number is limited to 16 bits (65536), for instance, we might get into trouble, as the bonehoard has about 9000 brushes with mostly 6 windings and more.

 

A single VBO per brush would not be ideal, what we probably want to look for is one VBO per material per light. I.E. for each light, group the rendered objects by material and give each one a VBO. The entire group can then be rendered at once.

[OrbWeaver]

edit: Another thing that might be interesting is this:

  %   cumulative   self			  self	 total		   
time   seconds   seconds	calls   s/call   s/call  name	
24.95	  6.29	 6.29 1110656232	 0.00	 0.00  BasicVector3<double>::BasicVector3()

The Vector3 default constructor takes a massload of calls and eats 6 seconds in the old code. So it's official that allocating a strong number of Vector3s is indeed expensive.

 

To be fair, allocating 1.1 billion of anything would be expensive. What this line tells us is that there are too many copy operations, not that there is much point in trying to optimise the copy constructor itself (the seconds-per-call value is zero, since the total time of 6 seconds is divided by the 1.1 billion calls).

[greebo]

Both issues are eliminated now: the copying is only performed when the winding plane actually changes, and the normal array is constructed only once along with the Winding.

 

I still have to change the Winding array to std::vector, because currently, it's a hardcoded 256-element array, which is way too memory-expensive.

[greebo]

And done: the Array structure has been removed in favour of std::vector which works just as fine. I also incorporated the normals into the WindingVertex structure, so the ugly array is gone now too.

[greebo]

I seem to have introduced a crash due to my latest changes. First I thought that only the changes in the referencecache branch would be unstable, but unfortunately the trunk is affected too.

 

I'm investigating it and I'll fix that soon, apologies for that - I should have branched off earlier. I hope you didn't plan to release within the next two or three days? If so, we probably need to take an earlier revision for release.

[OrbWeaver]

No problem, I hadn't noticed yet. I may do a release next weekend if everything is working, but I wasn't planning to in the next couple of days.

[greebo]

Ok, fine. I get inconsistent crashes in a (completely legit) map insertion operation and I don't know why. Also, the weak_ptr::lock() method seems to fail inconsistently, causing new Resource insertions. I'm digging into this and I think it will get resolved, but I need a bit more time.

[greebo]

Got it sorted out now. I should've investigated the Call Stack more deeply to see the problem, which was rooted in a concurrent clear() and insert() operation in the ModelCache. The trunk should be stable again, at least not less stable than it was before, of course.

[greebo]

During the last few days I spent some time in separating the Model/Map resource handling. Before I started, map files were considered as "models" in a general way, but there were a lot of IFs and special cases handled by the code, which was kind of difficult to see through.

 

Now we have ModelResource as well as MapResource, both deriving from Resource. I stripped the unneeded stuff from the specialised Map/ModelResource classes, so that the code is a bit clearer.

 

Both Resources are acquired through the GlobalReferenceCache(), as before. The ReferenceCache recognises requests for maps because of their extension and the correct Resource type is allocated.

 

I'm still unsure whether we want the maps be handled through the ReferenceCache, though.

[greebo]

Hm, I'm running into problems in MinGW/Win32 now after upgrading the boost headers to 1.34.1. When trying to link against the "old" libboost_filesystem-gcc-1_33_1.lib, the linker complains about a destructor being defined more than once. Obviously the old lib is not compatible to the new headers, which is not very surprising after all (strangely, it works with the boost regex library).

 

@OrbWeaver: How did you compile the libboost_regex.lib back then? I tried building the static library using bjam, but this doesn't produce a .lib file, only an .a file. When I try to use that one in the Sconstruct script, the linker complains about not being able to find that lib.

 

Any ideas what I might be doing wrong?

[OrbWeaver]

@OrbWeaver: How did you compile the libboost_regex.lib back then? I tried building the static library using bjam, but this doesn't produce a .lib file, only an .a file. When I try to use that one in the Sconstruct script, the linker complains about not being able to find that lib.

 

I can't remember how I compiled it, but I imagine it was through BJam as you did. MinGW should certainly be able to use a .a file rather than a .lib, although I guess there may be search order or naming issues involved.

[greebo]

Gah, I can't get it to work properly. I found a static library in a backup of my previous attempt at migrating to boost 1.34.1 (back in May), this links fine, but DarkRadiant still asks for a DLL at startup. I resignated and included that DLL in the w32 dependencies, and it seems to work. I'll leave it at that, I guess MinGW is a dying platform the way it looks.

 

On a different note: I noticed that I broke the trunk repository several times in the past and I wondered whether we should consider branching off after each release, like this?

 

trunk/

branches/0.9.6/

 

This way we are guaranteed to have a stable, compiling version available. What do you think, OrbWeaver? Is this worth the hassle or should we rather continue with our current approach (trunk + tagging before release)?

[OrbWeaver]

This way we are guaranteed to have a stable, compiling version available. What do you think, OrbWeaver? Is this worth the hassle or should we rather continue with our current approach (trunk + tagging before release)?

 

That's "the other" way of using SVN -- rather than unstable trunk + stable release branches/tags, you get stable trunk + unstable development branches. There probably isn't a lot in it either way, but I think the existing system works fine: people should always expect the trunk to be unstable, and if they want a stable version they should use one of the release tags (which admittedly I often forget to create, but they can always use the revision log to find a stable version).

 

We should obviously continue to use development branches for large pieces of work which are likely to result in long periods of instability, but I don't think it is necessary to have a development branch for each release.

[greebo]

That's fine with me.

 

Speaking of large pieces of work: My life is changing once again, as I'm going to start working in 2008, (more precisely on the 2nd of January).

 

Stating the obvious, I'll have less time than before for working on the mod, but I'd like to help changing the scenegraph when time comes.

 

My suggestion would be that we try to get 0.9.5 released soon and then start to work on the scenegraph.

[OrbWeaver]

Speaking of large pieces of work: My life is changing once again, as I'm going to start working in 2008, (more precisely on the 2nd of January).

Stating the obvious, I'll have less time than before for working on the mod, but I'd like to help changing the scenegraph when time comes.

 

That's fair enough; you've been pretty unusual so far in having such a lot of time available for the mod.

 

My suggestion would be that we try to get 0.9.5 released soon and then start to work on the scenegraph.

 

Sounds good to me, although I do want to get the Objectives Editor done in 0.9.6 as well now that we have the documentation from Ishtvan.

 

Speaking of releases, how do you feel about doing the Windows builds in future and uploading it somewhere for me to QA test? This might make more sense since you use Windows most of the time whereas I almost never use it, plus it adheres to "best practices" for QA by having a different person perform the final testing than create the actual build.

 

I guess the Linux builds could then be the other way round, with me producing the DEB and you or Tels doing the final QA.

[greebo]

Sounds good to me, although I do want to get the Objectives Editor done in 0.9.6 as well now that we have the documentation from Ishtvan.

Yep, that also needs doing. Two major points for the next release plus bigfixing is probably enough on the plate.

 

Speaking of releases, how do you feel about doing the Windows builds in future and uploading it somewhere for me to QA test? This might make more sense since you use Windows most of the time whereas I almost never use it, plus it adheres to "best practices" for QA by having a different person perform the final testing than create the actual build.

Incidentally, I was developing mainly in Linux the past few weeks, but anyways I'm fine with that suggestion.

 

I also considered creating the Win32 build with VC++ this time. The binary sizes are smaller by far (can't say why), but it should be thoroughly tested by somebody else before we actually package it.

 

I guess the Linux builds could then be the other way round, with me producing the DEB and you or Tels doing the final QA.

No problem with that.

[OrbWeaver]

Incidentally, I was developing mainly in Linux the past few weeks, but anyways I'm fine with that suggestion.

 

OK, although part of the basis for the suggestion was the assumption it would be easier for you because you are already using Windows, but the QA issue is valid in any case.

 

I also considered creating the Win32 build with VC++ this time. The binary sizes are smaller by far (can't say why), but it should be thoroughly tested by somebody else before we actually package it.

 

Fine with me, it will also be a good test of the process because I don't have VC++ or any related libraries installed at the moment (haven't got round to the 500 MB download as yet).