Jump to content


Photo

Tool for analyzing mission links from the Mission List


  • Please log in to reply
118 replies to this topic

#1 gnartsch

gnartsch

    Member

  • Member
  • PipPip
  • 479 posts

Posted 22 October 2013 - 06:21 PM

I spent some time today creating a little Java application which analyzes the consistency of the mission list.
I understand that keeping the mission database up to date is a tedious task, especially because only very few mirrors are under direct control of the TDM team itself.
The goal is to provide a little tool which searches for dead links, HTTPS-links, outdated versions of a mission & corresponding localization links (if any).

How it works:
- the Java application retrieves the mission list from the TDM server and extracts the URLs for the various missions
- then it analyzes the HTTP headers for each link and extracts the size of the referenced URL from it
- in case multiple mirrors were found for a mission, a comparison of the sizes is made in order to detect potential mismatches
- HTTPS links are indicated since they do not work in the internal mission downloader anyway
- dead links are reported as well
- the same is done for links to the Localization packs (if any)

Here is a first list of problems that were detected.
Question: would you find such a tool useful?
If so, I would be glad to donate it to TDM team after fine tuning it a bit more.

At any rate, please report this link to the people responsible for the downloads, because some things are amiss right now,
which causes people to download outdated versions for some missions.

Analyzing mission download list
Last update: Mon Oct 21 20:32:23 CEST 2013

Mission # 67:crystalgrave, "Crystal Grave" by ERH+ and Bikerdude
Date : 2013-02-09
Version : 2
Nominal Size : 12.4 MB

3 download links
detected download filesize: 12,0 MB (12586312)
3 localization links
detected localization filesize: 0,0 MB (3585)
[HTTP/1.1 404 Not Found] - -0,0 KB (-1) - http://www.southquar...fms/crystal.pk4
Last update: ?

--> dead link for localization pack

Mission # 60:remembrance, "In Remembrance of Him" by RPGista
Date : 2012-04-23
Version : 2
Nominal Size : 40.1 MB

2 download links
PROBLEM: Differing filesizes detected!
OK - 38,3 MB (40157064) - http://www.fidcal.co...remembrance.pk4
Last update: Sun Aug 04 15:27:42 CEST 2013

OK - 27,0 MB (28346001) - http://www.southquar...remembrance.pk4
Last update: Fri May 31 19:48:20 CEST 2013

--> outdated mirror

Mission # 39:lockdown1_2_1, "Lockdown" by GameDevGoro/b1k3rdude
Date : 2010-12-25
Version : 1
Nominal Size : 3 MB

3 download links
detected download filesize: 3,0 MB (3177294)
[HTTP/1.1 404 Not Found] - -0,0 MB (-1) - http://www.fidcal.co...ns/lockdown.pk4
Last update: ?

OK - 3,0 MB (3177294) - http://darkmod.taaak...ckdown1_2_1.pk4
Last update: Tue Oct 08 23:29:23 CEST 2013
[HTTP/1.1 404 Not Found] - -0,0 MB (-1) - http://www.southquar...ms/lockdown.pk4
Last update: ?

0 localization links
--> 2 dead download links

Mission # 64:oldhabits1, "Old Habits 1" by Obsttorte
Date : 2010-10-19
Version : 1
Nominal Size : 20 MB

3 download links
PROBLEM: Differing filesizes detected!
OK - 28,6 MB (30027252) - http://www.southquar.../oldhabits2.pk4
Last update: Sun Oct 20 23:13:55 CEST 2013

OK - 19,7 MB (20680862) - http://www.southquar.../oldhabits1.pk4
Last update: Sun Oct 20 12:29:37 CEST 2013
[HTTP/1.1 404 Not Found] - -0,0 MB (-1) - http://darkmod.taaak.../oldhabits1.pk4
Last update: ?

0 localization links
--> The first red link points to wrong version ! This should be oldhabits1.
--> And the second download link is dead.

Mission # 13:returntothecity, "Return to the City" by Melan & Bikerdude
Date : 2010-10-01
Version : 3
Nominal Size : 35 MB

2 download links
detected download filesize: 35,8 MB (37557302)
1 localization links
[HTTP/1.1 404 Not Found] - -0,0 KB (-1) - http://www.southquar...rntothecity.pk4
Last update: ?

--> dead download link

Mission # 48:samhain, "Samhain Night on Bone Hill" by PranQster
Date : 2011-10-09
Version : 1
Nominal Size : 10 MB

0 download links
PROBLEM: No download links detected!
1 localization links
detected localization filesize: 9,9 MB (10427780)
PROBLEM: Fairly big localization file! Maybe this is a link to the mission instead?
OK - 10.183,4 KB (10427780) - http://www.southquar...fms/samhain.pk4
Last update: Fri May 31 19:55:02 CEST 2013

--> This link shows up in the 'Localization' section. But it is the mission itself.
Since it isn't ready for 2.0 anyway, it should be taken down. But clearly it is no 'Localization pack'.

Mission # 58:kneipe24, "Sneak and Destroy" by SeriousToni
Date : 2012-04-12
Version : 1
Nominal Size : 158 MB

3 download links
PROBLEM: Differing filesizes detected!
OK - 158,3 MB (165951602) - http://www.southquar...eak_destroy.pk4
Last update: Fri May 31 18:26:39 CEST 2013

OK - 10,5 MB (11042816) - http://darkmod.taaak...eak_destroy.pk4
Last update: Tue Oct 08 23:53:30 CEST 2013

OK - 158,3 MB (165951602) - http://www.fidcal.co...eak_destroy.pk4
Last update: Thu Apr 12 07:58:46 CEST 2012

0 localization links
--> this link appears to be corrupted. After downloading the file from that URL, my ZIP program refuses to open it.

Mission # 30:somewhere, "Somewhere Above the City" by Grayman
Date : 2010-08-20
Version : 3
Nominal Size : 11 MB

3 download links
detected download filesize: 11,1 MB (11629480)
[HTTP/1.1 404 Not Found] - -0,0 MB (-1) - http://www.fidcal.co...omewhere1_1.pk4
Last update: ?

OK - 11,1 MB (11629480) - http://www.southquar...omewhere1_1.pk4
Last update: Fri May 31 20:24:26 CEST 2013
OK - 11,1 MB (11629480) - http://darkmod.taaak...omewhere1_1.pk4
Last update: Tue Oct 08 23:42:02 CEST 2013
1 localization links
[HTTP/1.1 404 Not Found] - -0,0 KB (-1) - http://www.southquar...omewhere1.1.pk4
Last update: ?

--> 2 dead links, 1 for the mission itself and 1 for the localization pack

Mission # 22:delivery, "Special Delivery" by Silencium18
Date : 2010-06-09
Version : 1
Nominal Size : 2 MB

3 download links
detected download filesize: 1,7 MB (1766091)
1 localization links
detected localization filesize: 1,7 MB (1766091)
PROBLEM: Fairly big localization file! Maybe this is a link to the mission instead?
OK - 1.724,7 KB (1766091) - http://www.southquar...ms/delivery.pk4
Last update: Fri May 31 17:42:21 CEST 2013

--> The 'localization pack' is as big as the mission itself. Actually it IS the mission itself and can be taken down.

Mission # 68:builders_blocks, "The Builder's Blocks" by Jesps
Date : 2013-03-18
Version : 1
Nominal Size : 2.85 MB

3 download links
detected download filesize: 2,9 MB (2994086)
OK - 2,9 MB (2994086) - http://darkmod.taaak...ders_blocks.pk4
Last update: Tue Oct 08 23:27:06 CEST 2013
OK - 2,9 MB (2994086) - http://www.southquar...ders_blocks.pk4
Last update: Fri May 31 17:12:27 CEST 2013
[HTTP/1.1 404 Not Found] - -0,0 MB (-1) - http://www.fidcal.co...ders_blocks.pk4
Last update: ?

0 localization links
--> dead download link

Mission # 9:too_late, "Too Late" by Nielsen74
Date : 2009-12-24
Version : 1
Nominal Size : 3.8 MB

3 download links
detected download filesize: 3,6 MB (3815636)
1 localization links
detected localization filesize: 5,5 MB (5742311)
OK - 5.607,7 KB (5742311) - https://dl.dropbox.c...o_late_l10n.pk4
Last update: ?

--> Localization Pack on HTTPS. InGame download can't download it.

Mission # 10:trapped, "Trapped!" by RailGun
Date : 2009-12-30
Version : 1
Nominal Size : 6.3 MB

3 download links
1 localization links
detected localization filesize: 0,0 MB (10432)
OK - 10,2 KB (10432) - https://dl.dropbox.c...rapped_l10n.pk4
Last update: ?

--> Localization Pack on HTTPS. InGame download can't download it.


  • Obsttorte likes this

#2 Springheel

Springheel

    Creative Director (retired)

  • Admin
  • 36238 posts

Posted 22 October 2013 - 07:14 PM

Sounds pretty neat...I'm sure Biker will be interested.

#3 Bikerdude

Bikerdude

    Mod hero

  • Member
  • PipPipPipPipPip
  • 18454 posts

Posted 22 October 2013 - 07:31 PM

I'm sure Biker will be interested.

Indeed and I am more than happy to hellp you beta test it as I think this might help with the having multiple mirrors issue I have touched upon in another thread!

#4 gnartsch

gnartsch

    Member

  • Member
  • PipPip
  • 479 posts

Posted 23 October 2013 - 04:44 AM

Great!
So, I will work on completing this tool and improving the output, before sending you the first version probably by end of the week.
Let me know if some special requirements come to your mind.
The charme of the current solution is that the entire analysis completes within 1 minute, because the tool does not even require to actually download the missions.
If you want, you can start correcting the current server entries. I will use a local copy of the current XML mission list for further development.

#5 Serpentine

Serpentine

    Uber member

  • Active Developer
  • PipPipPipPip
  • 2903 posts

Posted 23 October 2013 - 05:11 AM

Sounds really handy (and reminds me to go fix the taaaki mirror now :)).

Edit: Nevermind, turns out I hadnt read the OP. How unusual... :(

#6 Bikerdude

Bikerdude

    Mod hero

  • Member
  • PipPipPipPipPip
  • 18454 posts

Posted 23 October 2013 - 12:08 PM

If you want, you can start correcting the current server entries. I will use a local copy of the current XML mission list for further development.

Will do.

#7 gnartsch

gnartsch

    Member

  • Member
  • PipPip
  • 479 posts

Posted 23 October 2013 - 05:33 PM

Thanks, Bikerdude!
I noticed you did some changes and along with that I have a question.
When I had a look into the source for the internal downloader, I noticed that the code can either attempt to get the mission list from a php script or from an XML file on the server.
http://www.thedarkmo...le_missions.php
http://www.thedarkmo...le_missions.xml

I just checked both of them and the XML seems still outdated since it hasn't your latest changes.
Now the question (out of pure curiosity) would be : under which conditions does the XML get updated?

#8 Serpentine

Serpentine

    Uber member

  • Active Developer
  • PipPipPipPip
  • 2903 posts

Posted 23 October 2013 - 10:55 PM

I fixed the problems with fms hosted on taaaki.za.net.

I'll maaaybe write a similar python tool later on, just to hook it into my general health-check package. Having you figure out the kinks is pretty nifty :) Very handy work!

#9 Bikerdude

Bikerdude

    Mod hero

  • Member
  • PipPipPipPipPip
  • 18454 posts

Posted 24 October 2013 - 04:57 AM

Now the question (out of pure curiosity) would be : under which conditions does the XML get updated?

I have no idea, perhaps Serps can help with this query..?

#10 Serpentine

Serpentine

    Uber member

  • Active Developer
  • PipPipPipPip
  • 2903 posts

Posted 24 October 2013 - 05:02 AM

Nope I have no access to the server etc, also it's PHP afaik -- something I will live happily avoiding at all costs.

#11 gnartsch

gnartsch

    Member

  • Member
  • PipPip
  • 479 posts

Posted 24 October 2013 - 06:17 PM

OK, here is a first version. (Source code included, feel free to do with it whatever you want)
EDIT: here is the real download link instead of the attachment.
http://www.gnartsch....DbCheck_0.1.zip

After unzipping, you will find a batch file inside.
Assuming you have a JRE installed already (at least 5.0 is required) on your Windows system, the batch file should detect your JRE and launch the analysis.

The output to the console is only a progress indicator.
For the actual analysis results, check the file analysis.txt which should be created in your current working directory.
Hopefully it is pretty much self-explanatory.

In case you have suggestions on improvement, let me know.
I intend to do some changes on it in order to improve it anyway.
Currently it uses the PHP url by default.

Edited by gnartsch, 26 October 2013 - 11:44 AM.


#12 gnartsch

gnartsch

    Member

  • Member
  • PipPip
  • 479 posts

Posted 27 October 2013 - 06:07 AM

I replaced the attachment for TDMMissionDbCheck_0.1.zip in my previous post by a proper download link.
Please note that identifying the correct download link for a mission is not as simple as checking which URL is reported to be newer.
That is because the date that is being reported is the date it got uploaded to the server.
So, one needs to look into the actual ZIPs in order to avoid any mistake.

#13 Bikerdude

Bikerdude

    Mod hero

  • Member
  • PipPipPipPipPip
  • 18454 posts

Posted 27 October 2013 - 07:00 AM

both links dont work, just right click on a line of text or a word and select the add link button.

#14 gnartsch

gnartsch

    Member

  • Member
  • PipPip
  • 479 posts

Posted 27 October 2013 - 11:44 AM

Ups. The link is now functional.
Sorry!

#15 Bikerdude

Bikerdude

    Mod hero

  • Member
  • PipPipPipPipPip
  • 18454 posts

Posted 27 October 2013 - 04:17 PM

Ok will have a look at this tomoz.

#16 gnartsch

gnartsch

    Member

  • Member
  • PipPip
  • 479 posts

Posted 27 October 2013 - 05:19 PM

Thanks!

Actually I just finished v0.2. TDMMissionDbCheck_0.2..zip
Now it does not only check the links from the PHP, but also the ones from the XML plus the links from the Mission Downloads WebPage by traversing down to the actual pages with the mission details.
The corresponding sources where the links were found are now indicated with PHP, XML and/or WEB in the output file.
(The links are printed out only in case there was some issue identified, though)

And a question on the Webpage: how comes that some links appear on the website only?
Does it mean that the web page is updated separately, apart from whatever needs to be edited for the internal downloader?

For example this link exists on the webpage only, and is actually dead.
http://www.southquar...ntothecityv3.pk

Edited by gnartsch, 27 October 2013 - 05:26 PM.


#17 rich_is_bored

rich_is_bored

    Advanced Member

  • Member
  • PipPipPip
  • 854 posts

Posted 27 October 2013 - 10:12 PM

Actually that link works if you correct it to ".pk4" instead of ".pk".

#18 Bikerdude

Bikerdude

    Mod hero

  • Member
  • PipPipPipPipPip
  • 18454 posts

Posted 28 October 2013 - 07:11 AM

I will correct all those links today..

#19 gnartsch

gnartsch

    Member

  • Member
  • PipPip
  • 479 posts

Posted 02 November 2013 - 06:47 PM

Version 0.3 is up. (binaries and source code included)

Besides adding some logic to detect the problems outlined in this thread (wrong pk4-files for download on the website for Rightful Property and Builders Blocks),
it also detects and explains why some download links appear only on the website, but not in the list that is sent to the internal downloader (trailing blanks in the URL!).
And a couple of more improvements.

Please check the outputfile (analysis.txt) after running the tool.
Currently there are issues for 10 reported.

Edited by gnartsch, 02 November 2013 - 06:55 PM.


#20 Bikerdude

Bikerdude

    Mod hero

  • Member
  • PipPipPipPipPip
  • 18454 posts

Posted 02 November 2013 - 08:43 PM

Ok, will take a look at this when I have the time.

#21 gnartsch

gnartsch

    Member

  • Member
  • PipPip
  • 479 posts

Posted 03 November 2013 - 01:48 PM

Version 0.4 is up.

And for your convience also the current result file listing the issues.
http://www.gnartsch....ta/analysis.txt

#22 gnartsch

gnartsch

    Member

  • Member
  • PipPip
  • 479 posts

Posted 12 November 2013 - 06:02 PM

Currently 13 missions with issues are reported (console output below along with details from the logs and my own comments).
Let me point out the most urgent issues and explain what exactly is wrong about them.
You can go to the corresponding download pages in order to find those findings confirmed there.


Wrong mission up for download :

(These two issues got reported two weeks ago already from some player and I had included detection for that sort of issue in version 0.4 already.
http://forums.thedar...ownloader-bugs/.
There, you can also find the original correct links.)

Checking mission # 59 : Rightful Property
Problems detected!


PROBLEM: Conflicting filenames detected ('remembrance.pk4') between this mission (# 59 : Rightful Property) and mission # 60 : In Remembrance of Him
'http://www.fidcal.co...remembrance.pk4'
'http://www.fidcal.co...remembrance.pk4'

'http://www.fidcal.co...remembrance.pk4'
'http://darkmod.taaak...remembrance.pk4'
'http://www.southquar...remembrance.pk4'

ALL download links refer to 'In Remembrance of Him' !

Checking mission # 68 : The Builder's Blocks
Problems detected!


PROBLEM: Conflicting filenames detected ('crystalgrave2_1.pk4') between this mission (# 68 : The Builder's Blocks) and mission # 67 : Crystal Grave
'http://darkmod.taaak...talgrave2_1.pk4'
'http://darkmod.taaak...talgrave2_1.pk4'

'http://darkmod.taaak...talgrave2_1.pk4'
'http://www.fidcal.co...talgrave2_1.pk4'
'http://www.southquar...talgrave2_1.pk4'

ALL download links refer to 'Crystal Grave' !


Bad Localization Packs :

Checking mission # 48 : Samhain Night
Problems detected!


INFO: 1 localization links
'http://darkmod.taaak...ms/alberic3.pk4'

The localization pack download link refer to 'Alberics Curse' !
And the internal downloader will actually download it and rename it to 'samhain_l10n.pk4'!

EDIT:
I just tried out what happens on a fresh download of the entire FM:
With that bad L10N file in place you actually end up playing Alberics Curse, since the L10N overrides the entire mission.


Wrong/outdated download links :

Checking mission # 58 : Sneak and Destroy
Problems detected!


INFO: 3 download links
PROBLEM: Differing filesizes detected!

OK - 'http://darkmod.taaak...eak_destroy.pk4'
10,5 MB (11042816)

Last update: Tue Oct 08 23:53:30 CEST 2013
Language: English
Sources: [PHP, XML, WEB]
OK - 'http://www.fidcal.co...eak_destroy.pk4'
158,3 MB (165951602)
Last update: Thu Apr 12 07:58:46 CEST 2012
Language: English
Sources: [PHP, XML, WEB]
OK - 'http://www.southquar...eak_destroy.pk4'
158,3 MB (165951602)
Last update: Fri May 31 18:26:39 CEST 2013
Language: English
Sources: [PHP, XML, WEB]

The mirror on "http://darkmod.taaaki.za.net" still refers to a corrupted pk4.


The full logs can be found here.

Edited by gnartsch, 12 November 2013 - 06:22 PM.


#23 Bikerdude

Bikerdude

    Mod hero

  • Member
  • PipPipPipPipPip
  • 18454 posts

Posted 12 November 2013 - 06:39 PM

Will look at this tomoz.

#24 gnartsch

gnartsch

    Member

  • Member
  • PipPip
  • 479 posts

Posted 27 November 2013 - 05:12 PM

'Tomoz' +14.
And even worse:
'The Builder's Blocks' & 'Rightful Property' not even availlable for download for almost 4 weeks.

#25 Bikerdude

Bikerdude

    Mod hero

  • Member
  • PipPipPipPipPip
  • 18454 posts

Posted 27 November 2013 - 07:03 PM

Apologies, RL and then a holiday happened, will try and fix all the above now Im back etc.




0 user(s) are reading this topic

0 members, 0 guests, 0 anonymous users