New tdm_installer and dev builds

[stgatilov 998]

Some time ago I started investigation for a better differential update algorithm. The first attempt was known as time-travelling tdmupdate (also "tdmsync"). The second attempt has been much more cumbersome, but I think it has reached some usable state.

 

Here is the new tdm_installer:

• Windows (32-bit)
• Linux 64-bit

Note that it is not a finished application yet: it has some important features missing, probably bugs also.
If your TDM installation is important to you, better copy it before running the installer on it.
Please read the comments below to know how to workaround some issues.

 

If you select "Get custom version" check, then you will be able to choose which version to install.
Notice that it includes the very first dev build of TDM (with manbeast changes and the new experimental rendering backend).

Needless to say, development builds of TDM come unsupported: they are very likely to contain bugs.
They are not intended for normal playing.
We would be happy if you report bugs for recent dev builds of course, but won't mind if you wait until next beta phase.

Obviously, old versions of TDM won't get any fixes

 

Some known tdm_installer problems:

1) Installs/updates cannot be canceled/interrupted properly yet.
If you click on "START" button, there is no turning back except for killing the process in task manager.

2) It is strongly recommended to always enable "Force scan" check on the first page.
It will spend some time, but will save you from error at the end of installation procedure.
Normally, you don't need to scan if you did not modify your installation after previous install. But it's better to be safe yet.

3) The installer does not unpack zips yet.
So after install you will have to unpack everything from tdm_shared_stuff.zip manually.

4) As of now, the installer downloads data from the main TDM server.
It is rather slow, so think twice before running it in empty directory.
Instead of clean install, better copy all tdm_****.pk4 files from any existing installation into a new directory and run updater there.
Note that you can see how much data is going to be downloaded on the page with "START" button.

 

 

[stgatilov 998]

For now, I'll post changelog for dev builds here.

dev15976-8815
Includes all the changes from trunk which did not get into 2.08:
 * RGTC compression is now used by default.
 * More work on X-ray view.
 * Cull objects behind mirrors (4946).
 * AAS limit hit by Painter's Wife removed.
 * Warnings when approaching event limit (5270).
 * Dragofer's massive manbeast fixes.
 * Cabalistic's FBO refactoring, parallel jobs fixes, and new rendering backend.
Known problems:
 * HeatHaze shaders don't work (i.e. no heat/glass/water distortion).
 * Changing texture anisotropy leaves screen black.
 * Lights with stencil shadows don't illuminate properly until you reloadModels or switch shadows (after loading save).

dev15981-8875
* Fixed known problems of previous build.
* Revised OpenGL loading, version and extentions.
* Fixed warning about "bad internalFormat".
* Optimizations in new backend: individual shadow scissors.
* OpenAL: show active device name properly in console.
* Various guards against NaN (5296).
* Refactored lightgem rendering with triple buffering.
* Disable "Omit Frame Pointers" on Linux build.
* Fixed occasional crashes due to inventory cursors (5302).
* Fixed "Render Scale" slider in settings menu.
* Fixed rendering of several parchment fonts (link).

dev15987-8910
* Rotation hack fixed: bad rotation is now embedded into model for all entities (4970).
* Hot reload: added command reloadMap (5316).
* Fixed bad highlight on lockpicking. Also hardened GUI scripts against such errors (5311).
* Fixed black preview screenshots in in-game downloader (4488).
* Enabled r_useBumpmapLightTogglingFix by default: bumpy surfaces will be darkened when light falls under low angle.
* New rendering backend linkage fix on AMD.
* Better window resize on Windows.
* Cleaning hacks: r_testSpecularFix is always on (as in 2.08), r_testStencilSelfShadowFix is always off (as in 2.08).
* EditAFs: spawned entities are now frobable/draggable by default (5297).
* Fixed crash when unlocking door in "Hidden Hands: The Lost Citadel" (5318).
* savegame_admin and target_savegame: fixed missing spawnclass.
* Removed player_sounds_doublevision and similar spawnargs, removing warnings (5170).
Known problems:
* Several FMs do not boot due to rotation-hacked entity cm or md5 model: "A Night to Remember", "The Elixir", "Rightful Property", "Siege Shop", "William Steele 4: The Warrens".

dev16010-8948
* decl_stack: Every warning about missing image/sound is followed by detailed explanation how it was referenced (5170).
* Fixed occasional bug in concatenation of GUI commands (5323).
* Skin files: process hyphens in unquoted model paths correctly (5322).
* Automation system: implemented on Linux, fixed file leak.
* Supported reloadmap-diff automation command for fast hot-reload integrated into DarkRadiant in future (5316).
* Fixed doors occasionally getting stuck after hours of gameplay (5331).
* Creeping now works in "always run" mode (5320).
* Fixes for new rendering backend: ambientGamma and lightgem, bloom scissoring.
* Updated comments in mainmenu_briefing_video.gui for FFmpeg cinematics (5323).
* Fixed printing deleted pointer on startup (5170).
* Disabled "Door %s is not within a valid AAS area" warning (5170).
* Fixed warning spam when TDM is minimized (introduced after 2.08).
* Fixed parsing error when precaching "play XXX" GUI command, fixing a warning (5170).
* If ASE material is not referenced by its meshes, don't try resolving this material in TDM, don't post warning (5170).
* Fixed a bunch of assets referencing nonexistent files (5170).
* Added respawn console command for debugging (5316).
* Added setGameTime/getGameTime console commands for debugging (5331).
* mainmenu_briefing_video.gui no longer resets cinematics on startup (5324).
* Changed when mapStart GUI command is set up, removed MainMenuWatchDog (5323).
 

[freyk 464]

Nice job.

Here are my mentions, during the first time i run this build:
===
This in comparison with my unofficial tdm installer (at the tdm download page)

• Installer window very large in screensize.
• Doesnt use the standard winforms buttons, interface, etc.
  (can scare people. i expected an installer like DR, using the winforms interface)
  But if this a multiplatform, then i Prefer an automatically gui switcher with winforms when the use uses windows)
• Canot find any installation selection list for components (shortcuts, uninstaller, docs, etc)
• Installer selects automatically the location where the installer is placed.
  (so you want to install the game in your downloads folder??)
  When changing another location, the next-button grays out.
• Cant create folders in standard application locations (like program files > permission denied.)
• i love the version selector!
• Dont like The total progress bar (and his yellow progress colour)
  Howabout dividing in two progressbars (total and file),
  Download Status-line field would be nice. (people can then copy error lines to forum
• Dont see the tdm binaries in the folder, only the updater.

About linux installer: how about promoting the existing repo's & packagemanagers? 

Is there sourcecode available?
==

Please look at other installers out there and my unofficial tdm installer (at the tdm download page) as an example.
(and sourcecode of the nsis installer script at my github repo)

Edited July 13 by freyk

[stgatilov 998]

3 minutes ago, freyk said:

Window very large in screensize.

Yes, that's because of version selector.
If I keep the size of tdm_update, then scrolling to find the version you need would be painful (keep in mind that dev builds can come weekly).

Don't know what to do with it. I also don't like font size.

Quote

Doesnt use the standard winforms buttons, interface, etc.

That's the cost for cross-platform and light-weight GUI.

Quote

Installer selects automatically the location where the installer is placed.
(so you want to install the game in your downloads folder??)

Just like tdm_update, the installer always works in the directory where you placed it.
At least it shows you the path and allows to change that.

Quote

When changing another location, the next-button grays out.

I think there a restart button must appear --- it will auto-copy installer to selected location are restart from there.
As I explained, the installer should be where TDM is going to be installed.

Quote

Cant create folders in standard program files > permission denied.

As usual.
I hope it shows something remotely meaningful when you try. Warnings can be improved later.

Quote

Dont like The total progress bar (and his yellow progress colour),..
please devide in two progressbars (total and file)

I have no idea why FLTK progress bars are yellow by default.
It has caught my eye too. I think I can return the default greenish color

Quote

Canot find any installation selection list for components (shortcuts, uninstaller)

Maybe I would add shortcuts some time later. If I find how to do it both for Windows and Linux in C++

Uninstaller? Just delete the whole directory...

Quote

Is there sourcecode available?

Yes, it is all in tdm_installer directory in darkmod_src trunk.

Quote

Will there a ports for linux & mac?

I have posted the link for Linux version, right near the link for Windows version.

As for Mac: no, there won't, since TDM does not support Mac. And given that TDM does not support ARM, I think TDM won't support Mac anytime soon.
Although I think building tdm_installer for Mac would be quite easy.

[stgatilov 998]

Small update: green progress bars and tweaked image.

Perhaps time to check if self-update works

[stgatilov 998]

New version unpacks tdm_shared_stuff.zip automatically.

[stgatilov 998]

With new version, it is no longer necessary to check "Force scan".
The installer will check local zips by size and modification data, and will automatically perform scan if anything is dirty.

It also means that most likely you won't have to wait for this "analysis" any more, except for the first run.

UPDATE: Also I have added remaining time estimate to the main "downloading" progress bar.
So that user can see at least approximately how much time it would take to download a fresh 2.08 release

[New Horizon 496]

Nice!

So far I haven't run into any issues using it. 

I have to echo freyk's concerns with the appearance though.  The UI lacks the cohesive look and appearance I would expect from TDM.  Can the standard windows appearance be used with this installer?  

Functionally the installer is great but the appearance lacks a professional quality.  The lightweight UI is a bit sloppy looking.

Thanks for all the great work.

[stgatilov 998]

36 minutes ago, New Horizon said:

Can the standard windows appearance be used with this installer?

No.

Quote

I have to echo freyk's concerns with the appearance though.  The UI lacks the cohesive look and appearance I would expect from TDM. 

There are some schemes in FLTK, like "GTK+" and "plastic" scheme. They can be easily enabled in the code.

You can also take source code SVN, start ThirdParty\artefacts\fltk\bin\win64_s_vc15_rel_mt\fluid.exe, and open tdm_installer\GuiFluidAutoGen.fl in it. You can change layout and appearance, also can regenerate code and rebuild tdm_installer.

I have been looking at it for long enough time, and it does not look strange to me any more.
Can anyone bring some specific suggestions (aside from "make them look natively") ?

[stgatilov 998]

Ok, let me start with what I find bad:

1. Large window size and increased font size. I think it's better to return default size of font everywhere and reduce window size.

2. Buttons. They are awful! Despite my tweaks, they are bad for 3 reasons: a) thin border, b) same color as background, c) no hover action. I think I can fix (b) and (c), so it would become much better.

Anything else is less disturbing. Or did I forgot something?

[cabalistic 658]

I just tried the installer on Linux, and it has some problems with folder permissions: it created the "fms" folder with permissions 644 instead of the required 755, and consequently it errored out on not being able to create anything inside the fms folder due to missing permissions.

[stgatilov 998]

1 hour ago, cabalistic said:

I just tried the installer on Linux, and it has some problems with folder permissions: it created the "fms" folder with permissions 644 instead of the required 755, and consequently it errored out on not being able to create anything inside the fms folder due to missing permissions.

Damn, I always forget that the shared NTFS directory in Linux VM is very different from native FS.

[stgatilov 998]

Published new version with several improvements:

• The main download can now be canceled. Very useful if you thought it would be fast but remaining estimate shows you an hour or so
• Fixed directory permission problem on Linux. Also TDM executables are now marked as executable automatically.
• Now installer checks if writing is allowed, and checks for free space available, showing errors and warnings.
• Target manifest is always downloaded from the main mirror --- so setting untrusted servers as mirrors is OK now.

UPDATE: And one more:

• (Windows-only) Check if installer has elevated rights and post warning if it has.

[freyk 464]

How about building a gui switcher for the windows version, so can its use winforms.

I took a look at the available old versions.
I think/presume that we dont need the old beta versions of released versions.
Keep the release old versions.
For know, i would like only use this installer for downloading old released versions and  those dev-builds.
(Are those build newer than the release version?)

And about the permissions.
I will repeat every time to make installations work in proper/standard applications folders.
You see in other topics that windows users want to install their games in the those locations.
As would other developers also do.
For windows is it the "c:\program files" and in linux in the "/usr/share".
If you compare your existing folders in program files, you will see that all users can write in those subfolders, set up by the installers. 
In the common linuxdistro's, users in the game-group can also write in the /usr/share.

(but as an normal user in my beloved opensuse,the updater still cant write in the Tdm folder that the packagers set up) 

Edited July 12 by freyk

[stgatilov 998]

9 hours ago, freyk said:

How about building a gui switcher for the windows version, so can its use winforms.

Not possible, except if someone wants to rewrite the whole GUI part from scratch.
I will try to improve the look, but it will never look exactly as native Windows GUI.

Quote

I think/presume that we dont need the old beta versions of released versions.

They are needed for beta testing a new release.
After the release is out, they are left for historical purposes. They don't eat much space anyway.

Quote

(Are those build newer than the release version?)

The only dev build currently available is newer than release208.
I have just renamed its folder from 208 to 209.
By the way, dev build name is composed from two svn revision numbers.

Quote

You see in other topics that windows users want to install their games in the those locations.

They don't want, they just have a habit.

[freyk 464]

Tested Build v.015 (12th july build) on linux opensuse kde.
The things that i mentioned in my first post for the windows version, are the same for this one.

Edited July 13 by freyk

[New Horizon 496]

I haven't had a chance to download Fluid and the installer code yet but is it the choice of some type of skin for fluid that prevents it from looking like a standard Windows installer?  I was watching a Fluid tutorial the other day and the installer they were making looked like a standard windows installer in their video.  Not sure what settings they had used to obtain it though.  Might have time to investigate later.  It just looked much more cohesive.

[stgatilov 998]

I have published a new version with one major new feature: custom manifest URL is working.
However, we don't have any custom versions to test it yet... maybe we'll try it on VR build

1 hour ago, New Horizon said:

I haven't had a chance to download Fluid and the installer code yet but is it the choice of some type of skin for fluid that prevents it from looking like a standard Windows installer?  I was watching a Fluid tutorial the other day and the installer they were making looked like a standard windows installer in their video.  Not sure what settings they had used to obtain it though.  Might have time to investigate later.  It just looked much more cohesive.

I can say that:

1. I used some nonstandard buttons (as I described why above)
2. I used font larger than default
3. Perhaps some widgets are larger than they usually are, which also looks unfamiliar.

I'm going to rework GUI style.
Share a link of the video if you liked it, I'll look through it too.

[New Horizon 496]

I'll see if I can find the video again.  I don't really think it goes into how they got that look, it just seemed to be default and they were teaching how to make an installer in general.

 

[stgatilov 998]

New version is published.

Now when you install some version, installer saves the set of files this version contains in a file. When you install different version later, it deletes all those files of the old version.
This won't change much if you install official versions, aside from minor details like alsoft.ini being removed when downgrading to 2.06 or earlier. But it changes things if you install unofficial versions (like VR version) which add their own files. Such custom files will now be deleted when you decide to install back an official version.

By the way, the VR version can be now installed in tdm_installer like this:

• Check "Get Custom Version" on first page and click next.
• Copy/paste this to "Custom manifest URL":  http://darkmod.taaaki.za.net/zipsync/VR/209a1/manifest.iniz
• Choose release208 in GUI tree --- that is the base version.
• Continue as usual.

Also, I will post some sort of changelog for dev. versions in this thread.
New dev build will appear soon.

Next I will look into GUI style issues (make it look more familiar) and documentation of the whole system on wiki.

 

[stgatilov 998]

Published a new version of tdm_installer.

The GUI is reworked completely. Everything has less size, sizes are more or less like my OS windows have. Reverted to default font size. Colors tweaked a bit to remind how my Windows 10 looks. That's not the default of FLTK, but looks more like modern GUIs look. Unfortunately, alerts and file browser now look a bit different than everything else

I did not add any hover actions for now.

@freyk, @New Horizon, please say what you think about new GUI style.

[stgatilov 998]

New dev build of TDM published in tdm_installer.
Changelog provided here.

[freyk 464]

Is it possible to separate this topic, in a topic about the tdm installer and the tdm devbuilds?

About the installer .

• Nice work on the checkboxes, buttons, etc. And please dont forget the filebrowser.
• The tdm installer windows on windows is smaller, but is, to me, still to large.
• After the downloads, the installer waits some seconds before it repacks files.
  Impatient new users doesnt now how to react on it.
  Maybe adding a new label to the "repacking" progressbar? Or adding a total progress bar?
   

 

Edited July 20 by freyk

[stgatilov 998]

8 hours ago, freyk said:

Is it possible to separate this topic, in a topic about the tdm installer and the tdm devbuilds?

It does not make sense until tdm_installer becomes official and tdm_update bites dust.
Because dev builds are only available in tdm_installer.

Quote

And please dont forget the filebrowser.

Filebrowser and alert message are going to be pain. Because right now I spawn them all with simple C functions.

Quote

After the downloads, the installer waits some seconds before it repacks files.
Impatient new users doesnt now how to react on it.
Maybe adding a new label to the "repacking" progressbar? Or adding a total progress bar?

Yes, I have noticed it too. It is especially bad on fresh install.
Downloading is already over, files are verified to match checksums. Architecturally, it is part of "downloading" task, but progress ratio is measured by downloaded bytes, so it stays 100%.


 

[stgatilov 998]

Meanwhile, a new version of installer is ready.

Now user can optionally delete darkmod.cfg and create desktop shortcut after install.
It is astonishing how a simple problem of creating shortcut takes half a day to implement