gnartsch Posted October 22, 2013 Report Share Posted October 22, 2013 I spent some time today creating a little Java application which analyzes the consistency of the mission list.I understand that keeping the mission database up to date is a tedious task, especially because only very few mirrors are under direct control of the TDM team itself.The goal is to provide a little tool which searches for dead links, HTTPS-links, outdated versions of a mission & corresponding localization links (if any). How it works:- the Java application retrieves the mission list from the TDM server and extracts the URLs for the various missions- then it analyzes the HTTP headers for each link and extracts the size of the referenced URL from it- in case multiple mirrors were found for a mission, a comparison of the sizes is made in order to detect potential mismatches- HTTPS links are indicated since they do not work in the internal mission downloader anyway- dead links are reported as well- the same is done for links to the Localization packs (if any) Here is a first list of problems that were detected.Question: would you find such a tool useful?If so, I would be glad to donate it to TDM team after fine tuning it a bit more. At any rate, please report this link to the people responsible for the downloads, because some things are amiss right now,which causes people to download outdated versions for some missions. Analyzing mission download listLast update: Mon Oct 21 20:32:23 CEST 2013 Mission # 67:crystalgrave, "Crystal Grave" by ERH+ and BikerdudeDate : 2013-02-09Version : 2Nominal Size : 12.4 MB3 download linksdetected download filesize: 12,0 MB (12586312)3 localization linksdetected localization filesize: 0,0 MB (3585)[HTTP/1.1 404 Not Found] - -0,0 KB (-1) - http://www.southquar...fms/crystal.pk4 Last update: ?--> dead link for localization pack Mission # 60:remembrance, "In Remembrance of Him" by RPGistaDate : 2012-04-23Version : 2Nominal Size : 40.1 MB2 download linksPROBLEM: Differing filesizes detected!OK - 38,3 MB (40157064) - http://www.fidcal.co...remembrance.pk4 Last update: Sun Aug 04 15:27:42 CEST 2013OK - 27,0 MB (28346001) - http://www.southquar...remembrance.pk4 Last update: Fri May 31 19:48:20 CEST 2013--> outdated mirror Mission # 39:lockdown1_2_1, "Lockdown" by GameDevGoro/b1k3rdudeDate : 2010-12-25Version : 1Nominal Size : 3 MB3 download linksdetected download filesize: 3,0 MB (3177294)[HTTP/1.1 404 Not Found] - -0,0 MB (-1) - http://www.fidcal.co...ns/lockdown.pk4 Last update: ?OK - 3,0 MB (3177294) - http://darkmod.taaak...ckdown1_2_1.pk4 Last update: Tue Oct 08 23:29:23 CEST 2013[HTTP/1.1 404 Not Found] - -0,0 MB (-1) - http://www.southquar...ms/lockdown.pk4 Last update: ?0 localization links--> 2 dead download links Mission # 64:oldhabits1, "Old Habits 1" by ObsttorteDate : 2010-10-19Version : 1Nominal Size : 20 MB3 download linksPROBLEM: Differing filesizes detected!OK - 28,6 MB (30027252) - http://www.southquar.../oldhabits2.pk4 Last update: Sun Oct 20 23:13:55 CEST 2013OK - 19,7 MB (20680862) - http://www.southquar.../oldhabits1.pk4 Last update: Sun Oct 20 12:29:37 CEST 2013[HTTP/1.1 404 Not Found] - -0,0 MB (-1) - http://darkmod.taaak.../oldhabits1.pk4 Last update: ?0 localization links--> The first red link points to wrong version ! This should be oldhabits1.--> And the second download link is dead. Mission # 13:returntothecity, "Return to the City" by Melan & BikerdudeDate : 2010-10-01Version : 3Nominal Size : 35 MB2 download linksdetected download filesize: 35,8 MB (37557302)1 localization links[HTTP/1.1 404 Not Found] - -0,0 KB (-1) - http://www.southquar...rntothecity.pk4 Last update: ?--> dead download link Mission # 48:samhain, "Samhain Night on Bone Hill" by PranQsterDate : 2011-10-09Version : 1Nominal Size : 10 MB0 download linksPROBLEM: No download links detected!1 localization linksdetected localization filesize: 9,9 MB (10427780)PROBLEM: Fairly big localization file! Maybe this is a link to the mission instead?OK - 10.183,4 KB (10427780) - http://www.southquar...fms/samhain.pk4 Last update: Fri May 31 19:55:02 CEST 2013--> This link shows up in the 'Localization' section. But it is the mission itself.Since it isn't ready for 2.0 anyway, it should be taken down. But clearly it is no 'Localization pack'. Mission # 58:kneipe24, "Sneak and Destroy" by SeriousToniDate : 2012-04-12Version : 1Nominal Size : 158 MB3 download linksPROBLEM: Differing filesizes detected!OK - 158,3 MB (165951602) - http://www.southquar...eak_destroy.pk4 Last update: Fri May 31 18:26:39 CEST 2013OK - 10,5 MB (11042816) - http://darkmod.taaak...eak_destroy.pk4 Last update: Tue Oct 08 23:53:30 CEST 2013OK - 158,3 MB (165951602) - http://www.fidcal.co...eak_destroy.pk4 Last update: Thu Apr 12 07:58:46 CEST 20120 localization links--> this link appears to be corrupted. After downloading the file from that URL, my ZIP program refuses to open it. Mission # 30:somewhere, "Somewhere Above the City" by GraymanDate : 2010-08-20Version : 3Nominal Size : 11 MB3 download linksdetected download filesize: 11,1 MB (11629480)[HTTP/1.1 404 Not Found] - -0,0 MB (-1) - http://www.fidcal.co...omewhere1_1.pk4 Last update: ?OK - 11,1 MB (11629480) - http://www.southquar...omewhere1_1.pk4 Last update: Fri May 31 20:24:26 CEST 2013OK - 11,1 MB (11629480) - http://darkmod.taaak...omewhere1_1.pk4 Last update: Tue Oct 08 23:42:02 CEST 20131 localization links[HTTP/1.1 404 Not Found] - -0,0 KB (-1) - http://www.southquar...omewhere1.1.pk4 Last update: ?--> 2 dead links, 1 for the mission itself and 1 for the localization pack Mission # 22:delivery, "Special Delivery" by Silencium18Date : 2010-06-09Version : 1Nominal Size : 2 MB3 download linksdetected download filesize: 1,7 MB (1766091)1 localization linksdetected localization filesize: 1,7 MB (1766091)PROBLEM: Fairly big localization file! Maybe this is a link to the mission instead?OK - 1.724,7 KB (1766091) - http://www.southquar...ms/delivery.pk4 Last update: Fri May 31 17:42:21 CEST 2013--> The 'localization pack' is as big as the mission itself. Actually it IS the mission itself and can be taken down. Mission # 68:builders_blocks, "The Builder's Blocks" by JespsDate : 2013-03-18Version : 1Nominal Size : 2.85 MB3 download linksdetected download filesize: 2,9 MB (2994086)OK - 2,9 MB (2994086) - http://darkmod.taaak...ders_blocks.pk4 Last update: Tue Oct 08 23:27:06 CEST 2013OK - 2,9 MB (2994086) - http://www.southquar...ders_blocks.pk4 Last update: Fri May 31 17:12:27 CEST 2013[HTTP/1.1 404 Not Found] - -0,0 MB (-1) - http://www.fidcal.co...ders_blocks.pk4 Last update: ?0 localization links--> dead download link Mission # 9:too_late, "Too Late" by Nielsen74Date : 2009-12-24Version : 1Nominal Size : 3.8 MB3 download linksdetected download filesize: 3,6 MB (3815636)1 localization linksdetected localization filesize: 5,5 MB (5742311)OK - 5.607,7 KB (5742311) - https://dl.dropbox.c...o_late_l10n.pk4 Last update: ?--> Localization Pack on HTTPS. InGame download can't download it. Mission # 10:trapped, "Trapped!" by RailGunDate : 2009-12-30Version : 1Nominal Size : 6.3 MB3 download links1 localization linksdetected localization filesize: 0,0 MB (10432)OK - 10,2 KB (10432) - https://dl.dropbox.c...rapped_l10n.pk4 Last update: ?--> Localization Pack on HTTPS. InGame download can't download it. 1 Quote Link to comment Share on other sites More sharing options...
Springheel Posted October 23, 2013 Report Share Posted October 23, 2013 Sounds pretty neat...I'm sure Biker will be interested. Quote TDM Missions: A Score to Settle * A Reputation to Uphold * A New Job * A Matter of Hours Video Series: Springheel's Modules * Speedbuild Challenge * New Mappers Workshop * Building Traps Link to comment Share on other sites More sharing options...
Bikerdude Posted October 23, 2013 Report Share Posted October 23, 2013 I'm sure Biker will be interested.Indeed and I am more than happy to hellp you beta test it as I think this might help with the having multiple mirrors issue I have touched upon in another thread! Quote Link to comment Share on other sites More sharing options...
gnartsch Posted October 23, 2013 Author Report Share Posted October 23, 2013 Great!So, I will work on completing this tool and improving the output, before sending you the first version probably by end of the week.Let me know if some special requirements come to your mind.The charme of the current solution is that the entire analysis completes within 1 minute, because the tool does not even require to actually download the missions.If you want, you can start correcting the current server entries. I will use a local copy of the current XML mission list for further development. Quote Link to comment Share on other sites More sharing options...
Serpentine Posted October 23, 2013 Report Share Posted October 23, 2013 Sounds really handy (and reminds me to go fix the taaaki mirror now ). Edit: Nevermind, turns out I hadnt read the OP. How unusual... Quote Link to comment Share on other sites More sharing options...
Bikerdude Posted October 23, 2013 Report Share Posted October 23, 2013 If you want, you can start correcting the current server entries. I will use a local copy of the current XML mission list for further development.Will do. Quote Link to comment Share on other sites More sharing options...
gnartsch Posted October 23, 2013 Author Report Share Posted October 23, 2013 Thanks, Bikerdude!I noticed you did some changes and along with that I have a question.When I had a look into the source for the internal downloader, I noticed that the code can either attempt to get the mission list from a php script or from an XML file on the server.http://www.thedarkmod.com/missiondb/get_available_missions.phphttp://www.thedarkmod.com/missiondb/available_missions.xml I just checked both of them and the XML seems still outdated since it hasn't your latest changes.Now the question (out of pure curiosity) would be : under which conditions does the XML get updated? Quote Link to comment Share on other sites More sharing options...
Serpentine Posted October 24, 2013 Report Share Posted October 24, 2013 I fixed the problems with fms hosted on taaaki.za.net. I'll maaaybe write a similar python tool later on, just to hook it into my general health-check package. Having you figure out the kinks is pretty nifty Very handy work! Quote Link to comment Share on other sites More sharing options...
Bikerdude Posted October 24, 2013 Report Share Posted October 24, 2013 Now the question (out of pure curiosity) would be : under which conditions does the XML get updated?I have no idea, perhaps Serps can help with this query..? Quote Link to comment Share on other sites More sharing options...
Serpentine Posted October 24, 2013 Report Share Posted October 24, 2013 Nope I have no access to the server etc, also it's PHP afaik -- something I will live happily avoiding at all costs. Quote Link to comment Share on other sites More sharing options...
gnartsch Posted October 24, 2013 Author Report Share Posted October 24, 2013 (edited) OK, here is a first version. (Source code included, feel free to do with it whatever you want)EDIT: here is the real download link instead of the attachment.http://www.gnartsch.de/Download/TDMMissionDbCheck_0.1.zip After unzipping, you will find a batch file inside.Assuming you have a JRE installed already (at least 5.0 is required) on your Windows system, the batch file should detect your JRE and launch the analysis. The output to the console is only a progress indicator.For the actual analysis results, check the file analysis.txt which should be created in your current working directory.Hopefully it is pretty much self-explanatory. In case you have suggestions on improvement, let me know.I intend to do some changes on it in order to improve it anyway.Currently it uses the PHP url by default. Edited October 26, 2013 by gnartsch Quote Link to comment Share on other sites More sharing options...
gnartsch Posted October 27, 2013 Author Report Share Posted October 27, 2013 I replaced the attachment for TDMMissionDbCheck_0.1.zip in my previous post by a proper download link.Please note that identifying the correct download link for a mission is not as simple as checking which URL is reported to be newer.That is because the date that is being reported is the date it got uploaded to the server.So, one needs to look into the actual ZIPs in order to avoid any mistake. Quote Link to comment Share on other sites More sharing options...
Bikerdude Posted October 27, 2013 Report Share Posted October 27, 2013 both links dont work, just right click on a line of text or a word and select the add link button. Quote Link to comment Share on other sites More sharing options...
gnartsch Posted October 27, 2013 Author Report Share Posted October 27, 2013 Ups. The link is now functional.Sorry! Quote Link to comment Share on other sites More sharing options...
Bikerdude Posted October 27, 2013 Report Share Posted October 27, 2013 Ok will have a look at this tomoz. Quote Link to comment Share on other sites More sharing options...
gnartsch Posted October 27, 2013 Author Report Share Posted October 27, 2013 (edited) Thanks! Actually I just finished v0.2. TDMMissionDbCheck_0.2..zipNow it does not only check the links from the PHP, but also the ones from the XML plus the links from the Mission Downloads WebPage by traversing down to the actual pages with the mission details.The corresponding sources where the links were found are now indicated with PHP, XML and/or WEB in the output file.(The links are printed out only in case there was some issue identified, though) And a question on the Webpage: how comes that some links appear on the website only?Does it mean that the web page is updated separately, apart from whatever needs to be edited for the internal downloader? For example this link exists on the webpage only, and is actually dead.http://www.southquar...ntothecityv3.pk Edited October 27, 2013 by gnartsch Quote Link to comment Share on other sites More sharing options...
rich_is_bored Posted October 28, 2013 Report Share Posted October 28, 2013 Actually that link works if you correct it to ".pk4" instead of ".pk". Quote ModWiki Link to comment Share on other sites More sharing options...
Bikerdude Posted October 28, 2013 Report Share Posted October 28, 2013 I will correct all those links today.. Quote Link to comment Share on other sites More sharing options...
gnartsch Posted November 2, 2013 Author Report Share Posted November 2, 2013 (edited) Version 0.3 is up. (binaries and source code included) Besides adding some logic to detect the problems outlined in this thread (wrong pk4-files for download on the website for Rightful Property and Builders Blocks),it also detects and explains why some download links appear only on the website, but not in the list that is sent to the internal downloader (trailing blanks in the URL!).And a couple of more improvements. Please check the outputfile (analysis.txt) after running the tool.Currently there are issues for 10 reported. Edited November 2, 2013 by gnartsch Quote Link to comment Share on other sites More sharing options...
Bikerdude Posted November 3, 2013 Report Share Posted November 3, 2013 Ok, will take a look at this when I have the time. Quote Link to comment Share on other sites More sharing options...
gnartsch Posted November 3, 2013 Author Report Share Posted November 3, 2013 Version 0.4 is up. And for your convience also the current result file listing the issues.http://www.gnartsch.de/Thief-FM/beta/analysis.txt Quote Link to comment Share on other sites More sharing options...
gnartsch Posted November 12, 2013 Author Report Share Posted November 12, 2013 (edited) Currently 13 missions with issues are reported (console output below along with details from the logs and my own comments).Let me point out the most urgent issues and explain what exactly is wrong about them.You can go to the corresponding download pages in order to find those findings confirmed there. Wrong mission up for download : (These two issues got reported two weeks ago already from some player and I had included detection for that sort of issue in version 0.4 already.http://forums.thedar...ownloader-bugs/.There, you can also find the original correct links.) Checking mission # 59 : Rightful Property Problems detected! PROBLEM: Conflicting filenames detected ('remembrance.pk4') between this mission (# 59 : Rightful Property) and mission # 60 : In Remembrance of Him 'http://www.fidcal.com/darkuser/missions/TDM2testmissions/remembrance.pk4' 'http://www.fidcal.com/darkuser/missions/remembrance.pk4' 'http://www.fidcal.com/darkuser/missions/TDM2testmissions/remembrance.pk4''http://darkmod.taaaki.za.net/fms/remembrance.pk4''http://www.southquarter.com/tdm/fms/TDM2testmissions/remembrance.pk4' ALL download links refer to 'In Remembrance of Him' ! Checking mission # 68 : The Builder's Blocks Problems detected! PROBLEM: Conflicting filenames detected ('crystalgrave2_1.pk4') between this mission (# 68 : The Builder's Blocks) and mission # 67 : Crystal Grave 'http://darkmod.taaaki.za.net/fms/crystalgrave2_1.pk4' 'http://darkmod.taaaki.za.net/fms/crystalgrave2_1.pk4' 'http://darkmod.taaaki.za.net/fms/crystalgrave2_1.pk4''http://www.fidcal.com/darkuser/missions/TDM2testmissions/crystalgrave2_1.pk4''http://www.southquarter.com/tdm/fms/TDM2testmissions/crystalgrave2_1.pk4' ALL download links refer to 'Crystal Grave' ! Bad Localization Packs : Checking mission # 48 : Samhain Night Problems detected! INFO: 1 localization links'http://darkmod.taaaki.za.net/fms/alberic3.pk4' The localization pack download link refer to 'Alberics Curse' !And the internal downloader will actually download it and rename it to 'samhain_l10n.pk4'!EDIT:I just tried out what happens on a fresh download of the entire FM:With that bad L10N file in place you actually end up playing Alberics Curse, since the L10N overrides the entire mission. Wrong/outdated download links : Checking mission # 58 : Sneak and Destroy Problems detected! INFO: 3 download linksPROBLEM: Differing filesizes detected! OK - 'http://darkmod.taaaki.za.net/fms/sneak_destroy.pk4'10,5 MB (11042816) Last update: Tue Oct 08 23:53:30 CEST 2013 Language: English Sources: [php, XML, WEB]OK - 'http://www.fidcal.com/darkuser/missions/sneak_destroy.pk4'158,3 MB (165951602) Last update: Thu Apr 12 07:58:46 CEST 2012 Language: English Sources: [php, XML, WEB]OK - 'http://www.southquarter.com/tdm/fms/sneak_destroy.pk4'158,3 MB (165951602) Last update: Fri May 31 18:26:39 CEST 2013 Language: English Sources: [php, XML, WEB] The mirror on "http://darkmod.taaaki.za.net" still refers to a corrupted pk4. The full logs can be found here. Edited November 12, 2013 by gnartsch Quote Link to comment Share on other sites More sharing options...
Bikerdude Posted November 12, 2013 Report Share Posted November 12, 2013 Will look at this tomoz. Quote Link to comment Share on other sites More sharing options...
gnartsch Posted November 27, 2013 Author Report Share Posted November 27, 2013 'Tomoz' +14.And even worse:'The Builder's Blocks' & 'Rightful Property' not even availlable for download for almost 4 weeks. Quote Link to comment Share on other sites More sharing options...
Bikerdude Posted November 28, 2013 Report Share Posted November 28, 2013 Apologies, RL and then a holiday happened, will try and fix all the above now Im back etc. Quote Link to comment Share on other sites More sharing options...
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.