Jump to content
The Dark Mod Forums

Tool for analyzing mission links from the Mission List


gnartsch

Recommended Posts

I spent some time today creating a little Java application which analyzes the consistency of the mission list.

I understand that keeping the mission database up to date is a tedious task, especially because only very few mirrors are under direct control of the TDM team itself.

The goal is to provide a little tool which searches for dead links, HTTPS-links, outdated versions of a mission & corresponding localization links (if any).

 

How it works:

- the Java application retrieves the mission list from the TDM server and extracts the URLs for the various missions

- then it analyzes the HTTP headers for each link and extracts the size of the referenced URL from it

- in case multiple mirrors were found for a mission, a comparison of the sizes is made in order to detect potential mismatches

- HTTPS links are indicated since they do not work in the internal mission downloader anyway

- dead links are reported as well

- the same is done for links to the Localization packs (if any)

 

Here is a first list of problems that were detected.

Question: would you find such a tool useful?

If so, I would be glad to donate it to TDM team after fine tuning it a bit more.

 

At any rate, please report this link to the people responsible for the downloads, because some things are amiss right now,

which causes people to download outdated versions for some missions.

 

Analyzing mission download list

Last update: Mon Oct 21 20:32:23 CEST 2013

 

Mission # 67:crystalgrave, "Crystal Grave" by ERH+ and Bikerdude

Date : 2013-02-09

Version : 2

Nominal Size : 12.4 MB

3 download links

detected download filesize: 12,0 MB (12586312)

3 localization links

detected localization filesize: 0,0 MB (3585)

[HTTP/1.1 404 Not Found] - -0,0 KB (-1) - http://www.southquar...fms/crystal.pk4

Last update: ?

--> dead link for localization pack

 

Mission # 60:remembrance, "In Remembrance of Him" by RPGista

Date : 2012-04-23

Version : 2

Nominal Size : 40.1 MB

2 download links

PROBLEM: Differing filesizes detected!

OK - 38,3 MB (40157064) - http://www.fidcal.co...remembrance.pk4

Last update: Sun Aug 04 15:27:42 CEST 2013

OK - 27,0 MB (28346001) - http://www.southquar...remembrance.pk4

Last update: Fri May 31 19:48:20 CEST 2013

--> outdated mirror

 

Mission # 39:lockdown1_2_1, "Lockdown" by GameDevGoro/b1k3rdude

Date : 2010-12-25

Version : 1

Nominal Size : 3 MB

3 download links

detected download filesize: 3,0 MB (3177294)

[HTTP/1.1 404 Not Found] - -0,0 MB (-1) - http://www.fidcal.co...ns/lockdown.pk4

Last update: ?

OK - 3,0 MB (3177294) - http://darkmod.taaak...ckdown1_2_1.pk4

Last update: Tue Oct 08 23:29:23 CEST 2013

[HTTP/1.1 404 Not Found] - -0,0 MB (-1) - http://www.southquar...ms/lockdown.pk4

Last update: ?

0 localization links

--> 2 dead download links

 

Mission # 64:oldhabits1, "Old Habits 1" by Obsttorte

Date : 2010-10-19

Version : 1

Nominal Size : 20 MB

3 download links

PROBLEM: Differing filesizes detected!

OK - 28,6 MB (30027252) - http://www.southquar.../oldhabits2.pk4

Last update: Sun Oct 20 23:13:55 CEST 2013

OK - 19,7 MB (20680862) - http://www.southquar.../oldhabits1.pk4

Last update: Sun Oct 20 12:29:37 CEST 2013

[HTTP/1.1 404 Not Found] - -0,0 MB (-1) - http://darkmod.taaak.../oldhabits1.pk4

Last update: ?

0 localization links

--> The first red link points to wrong version ! This should be oldhabits1.

--> And the second download link is dead.

 

Mission # 13:returntothecity, "Return to the City" by Melan & Bikerdude

Date : 2010-10-01

Version : 3

Nominal Size : 35 MB

2 download links

detected download filesize: 35,8 MB (37557302)

1 localization links

[HTTP/1.1 404 Not Found] - -0,0 KB (-1) - http://www.southquar...rntothecity.pk4

Last update: ?

--> dead download link

 

Mission # 48:samhain, "Samhain Night on Bone Hill" by PranQster

Date : 2011-10-09

Version : 1

Nominal Size : 10 MB

0 download links

PROBLEM: No download links detected!

1 localization links

detected localization filesize: 9,9 MB (10427780)

PROBLEM: Fairly big localization file! Maybe this is a link to the mission instead?

OK - 10.183,4 KB (10427780) - http://www.southquar...fms/samhain.pk4

Last update: Fri May 31 19:55:02 CEST 2013

--> This link shows up in the 'Localization' section. But it is the mission itself.

Since it isn't ready for 2.0 anyway, it should be taken down. But clearly it is no 'Localization pack'.

 

Mission # 58:kneipe24, "Sneak and Destroy" by SeriousToni

Date : 2012-04-12

Version : 1

Nominal Size : 158 MB

3 download links

PROBLEM: Differing filesizes detected!

OK - 158,3 MB (165951602) - http://www.southquar...eak_destroy.pk4

Last update: Fri May 31 18:26:39 CEST 2013

OK - 10,5 MB (11042816) - http://darkmod.taaak...eak_destroy.pk4

Last update: Tue Oct 08 23:53:30 CEST 2013

OK - 158,3 MB (165951602) - http://www.fidcal.co...eak_destroy.pk4

Last update: Thu Apr 12 07:58:46 CEST 2012

0 localization links

--> this link appears to be corrupted. After downloading the file from that URL, my ZIP program refuses to open it.

 

Mission # 30:somewhere, "Somewhere Above the City" by Grayman

Date : 2010-08-20

Version : 3

Nominal Size : 11 MB

3 download links

detected download filesize: 11,1 MB (11629480)

[HTTP/1.1 404 Not Found] - -0,0 MB (-1) - http://www.fidcal.co...omewhere1_1.pk4

Last update: ?

OK - 11,1 MB (11629480) - http://www.southquar...omewhere1_1.pk4

Last update: Fri May 31 20:24:26 CEST 2013

OK - 11,1 MB (11629480) - http://darkmod.taaak...omewhere1_1.pk4

Last update: Tue Oct 08 23:42:02 CEST 2013

1 localization links

[HTTP/1.1 404 Not Found] - -0,0 KB (-1) - http://www.southquar...omewhere1.1.pk4

Last update: ?

--> 2 dead links, 1 for the mission itself and 1 for the localization pack

 

Mission # 22:delivery, "Special Delivery" by Silencium18

Date : 2010-06-09

Version : 1

Nominal Size : 2 MB

3 download links

detected download filesize: 1,7 MB (1766091)

1 localization links

detected localization filesize: 1,7 MB (1766091)

PROBLEM: Fairly big localization file! Maybe this is a link to the mission instead?

OK - 1.724,7 KB (1766091) - http://www.southquar...ms/delivery.pk4

Last update: Fri May 31 17:42:21 CEST 2013

--> The 'localization pack' is as big as the mission itself. Actually it IS the mission itself and can be taken down.

 

Mission # 68:builders_blocks, "The Builder's Blocks" by Jesps

Date : 2013-03-18

Version : 1

Nominal Size : 2.85 MB

3 download links

detected download filesize: 2,9 MB (2994086)

OK - 2,9 MB (2994086) - http://darkmod.taaak...ders_blocks.pk4

Last update: Tue Oct 08 23:27:06 CEST 2013

OK - 2,9 MB (2994086) - http://www.southquar...ders_blocks.pk4

Last update: Fri May 31 17:12:27 CEST 2013

[HTTP/1.1 404 Not Found] - -0,0 MB (-1) - http://www.fidcal.co...ders_blocks.pk4

Last update: ?

0 localization links

--> dead download link

 

Mission # 9:too_late, "Too Late" by Nielsen74

Date : 2009-12-24

Version : 1

Nominal Size : 3.8 MB

3 download links

detected download filesize: 3,6 MB (3815636)

1 localization links

detected localization filesize: 5,5 MB (5742311)

OK - 5.607,7 KB (5742311) - https://dl.dropbox.c...o_late_l10n.pk4

Last update: ?

--> Localization Pack on HTTPS. InGame download can't download it.

 

Mission # 10:trapped, "Trapped!" by RailGun

Date : 2009-12-30

Version : 1

Nominal Size : 6.3 MB

3 download links

1 localization links

detected localization filesize: 0,0 MB (10432)

OK - 10,2 KB (10432) - https://dl.dropbox.c...rapped_l10n.pk4

Last update: ?

--> Localization Pack on HTTPS. InGame download can't download it.

  • Like 1
Link to comment
Share on other sites

Great!

So, I will work on completing this tool and improving the output, before sending you the first version probably by end of the week.

Let me know if some special requirements come to your mind.

The charme of the current solution is that the entire analysis completes within 1 minute, because the tool does not even require to actually download the missions.

If you want, you can start correcting the current server entries. I will use a local copy of the current XML mission list for further development.

Link to comment
Share on other sites

Thanks, Bikerdude!

I noticed you did some changes and along with that I have a question.

When I had a look into the source for the internal downloader, I noticed that the code can either attempt to get the mission list from a php script or from an XML file on the server.

http://www.thedarkmod.com/missiondb/get_available_missions.php

http://www.thedarkmod.com/missiondb/available_missions.xml

 

I just checked both of them and the XML seems still outdated since it hasn't your latest changes.

Now the question (out of pure curiosity) would be : under which conditions does the XML get updated?

Link to comment
Share on other sites

OK, here is a first version. (Source code included, feel free to do with it whatever you want)

EDIT: here is the real download link instead of the attachment.

http://www.gnartsch.de/Download/TDMMissionDbCheck_0.1.zip

 

After unzipping, you will find a batch file inside.

Assuming you have a JRE installed already (at least 5.0 is required) on your Windows system, the batch file should detect your JRE and launch the analysis.

 

The output to the console is only a progress indicator.

For the actual analysis results, check the file analysis.txt which should be created in your current working directory.

Hopefully it is pretty much self-explanatory.

 

In case you have suggestions on improvement, let me know.

I intend to do some changes on it in order to improve it anyway.

Currently it uses the PHP url by default.

Edited by gnartsch
Link to comment
Share on other sites

I replaced the attachment for TDMMissionDbCheck_0.1.zip in my previous post by a proper download link.

Please note that identifying the correct download link for a mission is not as simple as checking which URL is reported to be newer.

That is because the date that is being reported is the date it got uploaded to the server.

So, one needs to look into the actual ZIPs in order to avoid any mistake.

Link to comment
Share on other sites

Thanks!

 

Actually I just finished v0.2. TDMMissionDbCheck_0.2..zip

Now it does not only check the links from the PHP, but also the ones from the XML plus the links from the Mission Downloads WebPage by traversing down to the actual pages with the mission details.

The corresponding sources where the links were found are now indicated with PHP, XML and/or WEB in the output file.

(The links are printed out only in case there was some issue identified, though)

 

And a question on the Webpage: how comes that some links appear on the website only?

Does it mean that the web page is updated separately, apart from whatever needs to be edited for the internal downloader?

 

For example this link exists on the webpage only, and is actually dead.

http://www.southquar...ntothecityv3.pk

Edited by gnartsch
Link to comment
Share on other sites

Version 0.3 is up. (binaries and source code included)

 

Besides adding some logic to detect the problems outlined in this thread (wrong pk4-files for download on the website for Rightful Property and Builders Blocks),

it also detects and explains why some download links appear only on the website, but not in the list that is sent to the internal downloader (trailing blanks in the URL!).

And a couple of more improvements.

 

Please check the outputfile (analysis.txt) after running the tool.

Currently there are issues for 10 reported.

Edited by gnartsch
Link to comment
Share on other sites

  • 2 weeks later...

Currently 13 missions with issues are reported (console output below along with details from the logs and my own comments).

Let me point out the most urgent issues and explain what exactly is wrong about them.

You can go to the corresponding download pages in order to find those findings confirmed there.

 

 

Wrong mission up for download :

 

(These two issues got reported two weeks ago already from some player and I had included detection for that sort of issue in version 0.4 already.

http://forums.thedar...ownloader-bugs/.

There, you can also find the original correct links.)

 

Checking mission # 59 : Rightful Property

Problems detected!

 

PROBLEM: Conflicting filenames detected ('remembrance.pk4') between this mission (# 59 : Rightful Property) and mission # 60 : In Remembrance of Him

'http://www.fidcal.com/darkuser/missions/TDM2testmissions/remembrance.pk4'

'http://www.fidcal.com/darkuser/missions/remembrance.pk4'

 

'http://www.fidcal.com/darkuser/missions/TDM2testmissions/remembrance.pk4'

'http://darkmod.taaaki.za.net/fms/remembrance.pk4'

'http://www.southquarter.com/tdm/fms/TDM2testmissions/remembrance.pk4'

 

ALL download links refer to 'In Remembrance of Him' !

 

Checking mission # 68 : The Builder's Blocks

Problems detected!

 

PROBLEM: Conflicting filenames detected ('crystalgrave2_1.pk4') between this mission (# 68 : The Builder's Blocks) and mission # 67 : Crystal Grave

'http://darkmod.taaaki.za.net/fms/crystalgrave2_1.pk4'

'http://darkmod.taaaki.za.net/fms/crystalgrave2_1.pk4'

 

'http://darkmod.taaaki.za.net/fms/crystalgrave2_1.pk4'

'http://www.fidcal.com/darkuser/missions/TDM2testmissions/crystalgrave2_1.pk4'

'http://www.southquarter.com/tdm/fms/TDM2testmissions/crystalgrave2_1.pk4'

 

ALL download links refer to 'Crystal Grave' !

 

 

Bad Localization Packs :

 

Checking mission # 48 : Samhain Night

Problems detected!

 

INFO: 1 localization links

'http://darkmod.taaaki.za.net/fms/alberic3.pk4'

 

The localization pack download link refer to 'Alberics Curse' !

And the internal downloader will actually download it and rename it to 'samhain_l10n.pk4'!

EDIT:

I just tried out what happens on a fresh download of the entire FM:

With that bad L10N file in place you actually end up playing Alberics Curse, since the L10N overrides the entire mission.

 

 

Wrong/outdated download links :

 

Checking mission # 58 : Sneak and Destroy

Problems detected!

 

INFO: 3 download links

PROBLEM: Differing filesizes detected!

 

OK - 'http://darkmod.taaaki.za.net/fms/sneak_destroy.pk4'

10,5 MB (11042816)

Last update: Tue Oct 08 23:53:30 CEST 2013

Language: English

Sources: [php, XML, WEB]

OK - 'http://www.fidcal.com/darkuser/missions/sneak_destroy.pk4'

158,3 MB (165951602)

Last update: Thu Apr 12 07:58:46 CEST 2012

Language: English

Sources: [php, XML, WEB]

OK - 'http://www.southquarter.com/tdm/fms/sneak_destroy.pk4'

158,3 MB (165951602)

Last update: Fri May 31 18:26:39 CEST 2013

Language: English

Sources: [php, XML, WEB]

 

The mirror on "http://darkmod.taaaki.za.net" still refers to a corrupted pk4.

 

 

The full logs can be found here.

Edited by gnartsch
Link to comment
Share on other sites

  • 2 weeks later...

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.

Guest
Reply to this topic...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.


×
×
  • Create New...