Tool for analyzing mission links from the Mission List

[gnartsch]

Added an option to check a specific mission only.

 

e.g.:

 

check all missions

 

C:\> TDMMissionDbCheck.bat

 

check only mission 68

 

C:\> TDMMissionDbCheck.bat 68

 

It will still fetch all mission links (in order to detect mixed up mission files), but will perform the actual checking much faster.

V 0.43 : TDMMissionDbCheck_0.43.zip

Edited December 4, 2013 by gnartsch

[AluminumHaste]

Very cool work man! <3

[gnartsch]

Thanks, AluminiumHaste !

 

Version 0.44 is out.

I added some tracing to the console window as well, so that you get a rough idea about what sort of problem to expect for which mission.

The log-file still has the details.

Also I corrected the parsing for the localisation packs for XML/PHP.

Obviously a typo had slipped in at some point on my side... No idea when or how it happened.

[Bikerdude]

@Gnartsch, sorry I haven't gotten back to you. I will get around to using this tool eventually.

[gnartsch]

@Bikerdude: no problem. I don't give up that easily and will keep posting the entire next year if I have to.

 

 

EDIT :

 

Oh, can you please update Solar Escape 1 by Tr00perty ?

 

On august 21 he posted an updated version of the mission with various bugfixes and also with TDM 2.0 compatibility.

http://forums.thedar...post__p__317979

The version on the servers is still the original july 22 version.

 

 

EDIT2:

 

Lord Dufford by stumpy has been updated for TDM 2.0 as well on october 28.

So, it should be added back to the download repository.

http://forums.thedar...post__p__324925

 

 

IMHO, someone should really open an official stickyfied 'New Mission & Update Announcement Thread'.

Authors expecting that the admins will read each and every post is simply unrealistic.

With such a thread, players could 'announce' them in case the authors forgot to do so.

And the admins would have pretty much a single thread to follow.

Would be nice for players as well, in case they return after a long time without having to read all the threads and posts.

Just my 2 cents.

Edited December 13, 2013 by gnartsch

[gnartsch]

Someone added a mirror for 'In Remembrance of Him' just recently.

Now the tool complains about detected mismatching filesizes.

The offending mirror is on Southquarter. The link on Fical is v2.0 as it should be.

 

Adding mirrors that point to outdated versions does not help anybody at all.

 

Merry X-mas anyway

 

 

EDIT: in case it helps, the tool also provides a mode to test any mirrors you might intend to add upfront before really adding them on the server.

Download the XML file from the server and save it as available_missions.xml in the directory where you run the tool from.

Then you can manually add any potential mirrors to it.... and see if any complaints show up running the tool.

In that mode there is no crosschecking done against the PHP, XML or the Mission Details webpages, though.

Don't forget to delete available_missions.xml afterwards!

Edited December 21, 2013 by gnartsch

[Bikerdude]

Afternoon, will get around to doing this at some point over the holidays..

[gnartsch]

v0.45 is out with a few minor corrections.

[Bikerdude]

Downloading...

 

Can I make a request..? do you have any plans to create a basic GUI for this app, rather than having it present a command prompt..?

[gnartsch]

Adding a UI was not actually on my todo list.

Maybe I find some time for that sometime next year, or maybe not at all. Don't know yet.

What exactly would you like to see there, which the command-line version can not show you?

Or, to put it different: which part would you consider to be rather tedious in it's current form besides having to open the results in a text editor?

 

I understand that a good UI may help to pinpoint some issues in a better way, however, I wonder if it wouldn't be better to request the PHP page (or whatever you are required to use in order to maintain the mission DB) to be improved.

Such as cutting off trailing whitespace from any link that gets added - which makes up the majority of the current issues, although these are not the most important ones.

I would be curious to see what that page looks like and how helpful it really is - considering that in the past a few missions were even pointing to totally different missions.

Could you add a screenie?

[Bikerdude]

Sorry, I was only thinking of a basic gui which would show the results inside its own window rather than generating a text file. And my results maybe have a simple filter which would show which missions need looking at.

[Bikerdude]

Ok ran the latest version of the tool and did some housekeeping and here is some feedback -

• The output text file lists the Fm non-alphabetically, a bit of a pain as it is.
  
• Please add some leeway on files sizes, getting an error for being 0.1, 0.5 out is overkill.
  
• What does trailing whitespaces mean - how do i fix this and sorry if you have explained this previously
  

[gnartsch]

Sounds like a text-editor would be sufficient.

Check for the words PROBLEM or WARNING.

 

This will have to do for the moment until I find the mood to go some steps further.

If I decide to implement a UI it will end up being more complicated (but more user friendly) then just filtering for troublesome missions and fill a text field with the raw details as they are printed to the file.

That would take some time. We will see if and when I get to that.

 

 

Ok, just saw your last response.

Feed back is welcome!

 

- If the sorting is an issue, then I can try to address that during the next few days.

- Allowing a difference of ~0.5 MB sounds reasonable. Will do that as well.

 

- Regarding the whitespaces:

I have trouble fully understanding that myself as well.

But obviously there is some code in the DarkMod website already which is trying to handle that at least for the mission detail webpages.

See this small excerpt from the details for Business as Usual :

 

<td><a href="http://www.fidcal.com/darkuser/missions/TDM2testmissions/businessv3.pk4">http://www.fidcal.com/darkuser/mis... spaces</a></td>

 

The website obvisously retrieves the raw links from the database and at least corrects the actual LINK on the left by removing the trailing whitespaces.

But the 'display text' on the right still proofs that the link in the DB does have trailing blanks.

 

Now, turning to the code that feeds the XML & PHP pages:

I do not know what that code looks like, but it behaves different.

Any Link containing trailing blanks will never be send to the player-machine via the XML, because the server obviously does some checking (like whether the url ends with '.pk4' or not).

As a result, all those links ending with blanks are non-existant for the internal downloader.

 

<mission id="14" title="Business as Usual" releaseDate="2010-01-14" size="16" version="3" internalName="business3" type="single" author="Bikerdude">

<downloadLocation language="English" url="http://darkmod.taaaki.za.net/fms/business3.pk4"/>

<downloadLocation language="English" url="http://www.southquarter.com/tdm/fms/TDM2testmissions/business3.pk4"/>

<localisationPack url="http://darkmod.taaaki.za.net/fms/business_l10n.pk4"/>

<localisationPack url="http://www.southquarter.com/tdm/fms/business_l10n.pk4"/>

<localisationPack url="http://www.fidcal.com/darkuser/missions/business_l10n.pk4"/>

</mission>

 

As you can see, there is no link to the Fidcal mirror. It got discarded by the server before the XML was sent to my local PC.

Edited December 30, 2013 by gnartsch

[Bikerdude]

Regarding the whitespaces:

 

There is some code in the DarkMod website already which is trying to handle that at least for the mission detail webpages.

See this small excerpt from the details for Business as Usual :

 

<td><a href="http://www.fidcal.com/darkuser/missions/TDM2testmissions/businessv3.pk4">http://www.fidcal.com/darkuser/mis... spaces</a></td>

 

The website obvisously retrieves the raw links from the database and at least corrects the actual LINK on the left by removing the trailing whitespaces.

But the 'display text' on the right still proofs that the link in the DB does have trailing blanks.

 

As you can see, there is no link to the Fidcal mirror. It got discarded by the server before the XML was sent to my local PC.

 

Ah I see now what's happening!

 

Its when the description of the download link to the .PK4 file is too long to fit the formatting of the html frame on the mission's page, the website is then truncating the description of the download link. If I'm wrong someone please correct me, but I believe the in-game downloader I dosen't talk directly to the database rather than the website.

[gnartsch]

Ah I see now what's happening!

 

Its when the description of the download link to the .PK4 file is too long to fit the formatting of the html frame on the mission's page, the website is then truncating the description of the download link. If I'm wrong someone please correct me, but I believe the in-game downloader I dosen't talk directly to the database rather than the website.

 

I do not think that this is the issue. But it's hard to tell without seeing what you see when editing the details pages.

But here is another example.

 

<td><a href="http://darkmod.taaaki.za.net/fms/sound_alert_trainer.pk4">http://darkmod.taaaki.za.net/fms/sound_alert_trainer.pk4 </a></td>

</tr>

 

Actually the trailing whitespaces consist of a blank, followed by a tab.

(Save that webpage to disk and open it in an editor.)

 

The internal downloader itself of course can not access the database.

It calls a PHP script on the server, which will returns an XML file.

So, it's the server that crawls through the database and fetches the latest info from there before it assembles the XML.

[Bikerdude]

The internal downloader itself of course can not access the database. It calls a PHP script on the server, which will returns an XML file.

So, it's the server that crawls through the database and fetches the latest info from there before it assembles the XML.

Hmm, the only thing I can do is delete and re-create the download links on the mission pages then.

[gnartsch]

Can't you just delete the individual offending links and recreate them?

The analysis.txt file tells you exactly which these would be.

 

Opening a bugtracker against the webpage would be advisable as well. The code should strip off any trailing whitespaces.

 

I wonder if those blanks are introduced because of the way someone copy/pastes them from another source.

 

 

Oh, and I would like to draw your attention to post #30 as well.

Solar Escape & Lord Dufford got updated for 2.0 already. But the links in the database are older versions.

Edited December 30, 2013 by gnartsch

[Bikerdude]

• Can't you just delete the individual offending links and recreate them?
  
• Opening a bugtracker against the webpage would be advisable as well. The code should strip off any trailing whitespaces.
  
• I wonder if those blanks are introduced because of the way someone copy/pastes them from another source.
  
• Solar Escape & Lord Dufford got updated for 2.0 already. But the links in the database are older versions.
  

• Read my post above, :-P
  
• Can you do the honours as you understand more of whats going on.
  
• When I delete and recreate the download links there are no spaces, so f*** knows how they are still getting in there.
  
• I have replied in the lord Dufford thread, as Stumpy should fix his own mission. And for Solar Escape, version 1 is the latest and thats links on the FM page for that mission.
  

[gnartsch]

I opened a bugtracker for that issue. But there is not much more that I can do.

Since I have no access to the admin pages (off course) all I can do is guessing.

 

Regarding Solar Escape:

Tr00perty announced a TDM 2.0 compatibility update on August 21 in this post: http://forums.thedarkmod.com/topic/14944-fan-mission-solar-escape-1/page__view__findpost__p__317979

If you download the PK4 and open it, you will notice that the maps and various folder got updated on that day.

If you download the mission via any link from the database, you will see a date like July 22 instead (and the PK4 has a different size as well).

 

Unfortunately he failed to update the version in the readme or the darkmod.txt.

Nevertheless, I would consider that to be an official update that should be caught up.

And maybe increase the version number along with that.

[Bikerdude]

Nevertheless, I would consider that to be an official update that should be caught up.And maybe increase the version number along with that.

I've just over written the file on all three mirrors, if trooper does any more work on it we can change the version number then.

[gnartsch]

Thanks Bikerdude !

 

With that there are only 3 issues left, aside from the ones affected by whitespaces.

Two of them are still a pain in the ass.

 

Sneak & Destroy

'http://darkmod.taaaki.za.net/fms/sneak_destroy.pk4'

This link is much smaller then it should be. And if you download it, you can not even open it. --> So, it is corrupted, obviously!

Please drop the link or replace the pk4.

 

In Remembrance of Him

'http://www.southquarter.com/tdm/fms/remembrance.pk4'

This link is older and smaller then the other links. --> Outdated version

Please replace it with this one:

'http://www.southquarter.com/tdm/fms/TDM2testmissions/remembrance.pk4'

 

Crystal Grave

'http://www.southquarter.com/tdm/fms/TDM2testmissions/crystalgrave2_l10n.pk4'

This localisation link is dead. Please replace it with the one below.

'http://www.southquarter.com/tdm/fms/TDM2testmissions/crystalgravev2_l10n.pk4'

 

 

Wish you a Happy New Year !

[Bikerdude]

Sneak & Destroy - just uploading now

In Remembrance of Him

Crystal Grave

Done, happy new year!

[gnartsch]

Congrats on the release of TDM and The Gatehouse !

 

Just a minor update (FYI)

 

Thanks to taaaki, the issue with the Admin webpage producing whitespaces on links should be gone.

(at least I guess so, or it should be gone as soon as the servers are moved)

http://bugs.thedarkm...iew.php?id=3640

 

With that most missions (except for one) appear green.

Thanks!

 

Currently there is only one bad link :

httttp://www.fidcal.com/darkuser/missions/gatehouse.pk4

 

 

 

Btw: v0.46 is up - with a couple of minor changes and improvements.

The most important thing is that it will ignore the xml file on the server, in case it is not present,

thus all the errors you would see with the older versions will be suppressed.

(Don't know yet if that file has moved or whether it is no longer used at all and gone for good.

If I am not mistaken it was only a fallback URL for the downloader, in case PHP would not respond.)

 

Thanks again!

[Bikerdude]

• Currently there is only one bad link : - httttp://www.fidcal.com/darkuser/missions/gatehouse.pk4
  
• Btw: v0.46 is up - with a couple of minor changes and improvements.
  

• Fixed
  
• Downloading...
  

[taaaki]

Thanks to taaaki, the issue with the Admin webpage producing whitespaces on links should be gone.

(at least I guess so, or it should be gone as soon as the servers are moved)

http://bugs.thedarkm...iew.php?id=3640

 

Btw: v0.46 is up - with a couple of minor changes and improvements.

The most important thing is that it will ignore the xml file on the server, in case it is not present,

thus all the errors you would see with the older versions will be suppressed.

(Don't know yet if that file has moved or whether it is no longer used at all and gone for good.

If I am not mistaken it was only a fallback URL for the downloader, in case PHP would not respond.)

Mission details are already being served off the new server. I've also moved the missiondb info to the missions.thedarkmod.com sub-domain (requests for www.thedarkmod.com/missiondb/ get redirected here automatically). The static xml is still supposed to be there, but I just forgot to update the folder permissions to allow the webserver to write out the new one. That has been fixed. I also added some newlines to the XML generated by the PHP page (and the the available_missions.xml file) to make it more human-readable in case you need to inspect it manually.

 

 

• Fixed
  
• Downloading...
  

Biker: I think you added the fixed link to the Translations section, not the normal download links. I've gone and fixed this since I needed to check if the xml was created sucessfully.

 

Also, I'm looking into a more convenient way to add / edit mirrors.