Tool for analyzing mission links from the Mission List

[gnartsch]

 

$ curl --head http://swift-mazes.com/fms/antr.pk4

...

Content-Length: 33057589

Content-Type: application/octet-stream

Weird. I wonder what curl is doing different than the Java code I use.

swiftmazes is the only server which won't report the Content Length for a mission when doing

   URLConnection tmpConnection = new URL(url).openConnection();
   tmpConnection.getContentLength();

That's why that tool needs to ignore swiftmaze with respect to the checking file sizes... unless I figure out what I am missing.

All I get is the following, and digging deeper into possible options and alternatives to modify the behaviour of the URLConnection did not reveal any insights so far either.

Transfer-Encoding=[chunked], Keep-Alive=[timeout=5, max=100], null=[HTTP/1.1 200 OK], Server=[Apache/2.2.29], Connection=[Keep-Alive], Date=[Tue, 28 Apr 2015 20:39:33 GMT], Content-Type=[text/html]}

The official internal name for "Thief's Den" is now 'thiefsden', as should be the name of the pk4.

See http://missions.thedarkmod.com/get_available_missions.php

Edited April 28, 2015 by gnartsch

[Tels]

Weird. I wonder what curl is doing different than the Java code I use.

swiftmazes is the only server which won't report the Content Length for a mission when doing

   URLConnection tmpConnection = new URL(url).openConnection();
   tmpConnection.getContentLength();

That's why that tool needs to ignore swiftmaze with respect to the checking file sizes... unless I figure out what I am missing.

All I get is the following, and digging deeper into possible options and alternatives to modify the behaviour of the URLConnection did not reveal any insights so far either.

Transfer-Encoding=[chunked], Keep-Alive=[timeout=5, max=100], null=[HTTP/1.1 200 OK], Server=[Apache/2.2.29], Connection=[Keep-Alive], Date=[Tue, 28 Apr 2015 20:39:33 GMT], Content-Type=[text/html]}

The official internal name for "Thief's Den" is now 'thiefsden', as should be the name of the pk4.

See http://missions.thedarkmod.com/get_available_missions.php

 

Thanx for the info, if taaki or Biker delete the "thiefs_den" file, I'll nuke it, too.

 

As for the size: I'm not familiar with what the Java code does underneath, but may guess is it doesn't query it with HEAD; but a regualar GET request and then aborts? And this seems to confirm it:

https://stackoverflow.com/questions/263013/java-urlconnection-how-can-i-find-out-the-size-of-a-web-file

 

Basically, a "GET" will do a "chunked-transfer", where the lenght can't be sent, while a "HEAD" will work properly (and is actually the right way to get only the header info). With a GET, the server is prepared to send you the entire file, wether you want it or not. Sending, then aborting such a request will just lead to unec. work on the server side.

 

There is a bit of code on that page which shows how to use a HEAD request. I would program your tool so it first tries a HEAD request, and if that doesn't work, falls back to the regular method.

 

Hope this helps!

[Tels]

Stashing the L10N files into some backup-'archive' isn't my primary intention either.

But from what I know far more mission received a translation, than the number of mission got actually enabled for L10N.

And it still seems like quite a few authors don't want their missions to be touched by anyone else.

That's the issue here, I guess.

So, moving the unused ones away is just a means to keep the primary directory clean.

 

Yeah, but only for the case of "Author doesn't want to" && "it wasn't enabled".

 

If the author gave permission, it can't be hard to "enable" the mission and just use the files? We do have more or less automated tools for that. Would be much better than to just delete the work of the translators...

 

If someone needs help with that, let's just do it.

[Bikerdude]

Thanx for the info, if taaki or Biker delete the "thiefs_den" file, I'll nuke it, too.

Already done.

[gnartsch]

Thanks for the tip, Tels!

Unfortunately the resulting effect is terrible!

Instead of simply providing the data in a 'cleaner' way, the HEAD requests causes the entire PK4 to be downloaded as soon as I try to get the header info.

Seems like there must be some more details to this issue.

Oh, ... and even after the PK4 was fully downloaded, the file-size was still not to be found anywhere in the response or header. A mystery.

[Tels]

Already done.

 

Strange:

 

 

$ ls -la thief*

112908 Jan 24 2014 thiefs_den_l10n.pk4

3459871 Apr 12 2014 thiefsden.pk4

112908 Jan 24 2014 thiefsden_l10n.pk4

 

Is the "new" version now without l10n? That would be a step back

[Tels]

Thanks for the tip, Tels!

Unfortunately the resulting effect is terrible!

Instead of simply providing the data in a 'cleaner' way, the HEAD requests causes the entire PK4 to be downloaded as soon as I try to get the header info.

Seems like there must be some more details to this issue.

Oh, ... and even after the PK4 was fully downloaded, the file-size was still not to be found anywhere in the response or header. A mystery.

 

Uh, that is exactly backwards!

 

Maybe it is that problem:

 

https://community.oracle.com/thread/2112393

[gnartsch]

An issue in the mission database:

 

seems like Requiem received a silent update?

 

The version on taaki's server is slightly bigger and newer than the other two mirrors.

Once again I see that the other two references are pointing to the TDM2testmissions folder.

I guess the updated version got pushed to the regular missions folder instead.

I would recommend to clean up those folders and move the current versions to the regular folder.

I don't think that all that stuiff is still under test.

 

'http://darkmod.taaaki.za.net/fms/requiem.pk4'
OK
Size: 102.5 MB (107437077)
Last update: Sat May 30 09:35:37 CEST 2015

 

'http://www.fidcal.com/darkuser/missions/TDM2testmissions/requiem.pk4'
OK
Size: 102.4 MB (107330042)
Last update: Sat Oct 05 11:43:42 CEST 2013
Language: English
Sources: [php, XML]

'http://www.southquarter.com/tdm/fms/TDM2testmissions/requiem.pk4'
OK
Size: 102.4 MB (107330042)
Last update: Sat Oct 05 11:46:52 CEST 2013

[Bikerdude]

And regarding where they are getting uploaded to, all mission are going to the regular folder on all mirrors.

 

I beleive the issue is that the FM page is using old links, which I need to change. I will also have a look at all the other FMs and make sure none of them are using old links.

[gnartsch]

A few minor issues in the list of download URLs:

 

Penny Dreadful 1: The grail of regrets

bad URL : protocol reads 'ttp' instead of 'http'

ttp://www.fidcal.com/darkuser/missions/penny1_1.pk4

Patently Dangerous

dead link

http://www.southquarter.com/tdm/fms/patently_dangerous_v2.pk4

Thomas Porter 3: Glenham Tower

dead link

http://www.southquarter.com/tdm/fms/glenham.pk4

[Bikerdude]

Fixing now.

[skina]

http://www.thedarkmod.com/missiondetails/?id=111

ttp://darkmod.taaaki.za.net/fms/volta1_1.pk4

[Bikerdude]

Fixing now.

[nbohr1more]

A new scan attached.

 

Is there a way to remove the swift-maze mirror from the scan?

 

analysis.txt

[Bikerdude]

Fyi, those mirrors are I believe Springheels.

[gnartsch]

A new scan attached.

analysis.txt

Whoops!

Someone actually using that little tool?

I am baffled and surprised!

[Springheel]

A new scan attached.

 

Is there a way to remove the swift-maze mirror from the scan?

 

analysis.txt

 

Swift-mazes belonged to Tels IIRC.

[SeriousToni]

Indeed, that is correct, but he has not been around lately and sadly.

[Bikerdude]

i will get around to sorting that out at some point.

[nbohr1more]

I've been slowly cleaning-up the issues in my latest scan.

 

For some reason, I thought that stgatilov had updated the mirrors with zips where media was unpacked to match the local copies

when I uploaded to southquarter after they recovered from the hack.

 

Any chance for a linux version so I can do some of these on my break at work?

[nbohr1more]

Fixed all the mismatches.

 

A few false positive errors about translation packs in there.

 

analysis.txt

analysis.txt

[grayman]

Thanks for doing this, @NB.

[gnartsch]

Any chance for a linux version so I can do some of these on my break at work?

 

A few false positive errors about translation packs in there.

Thanks for looking into this, nbohr1more !

 

 

A Linux version shouldn't be a problem at all.

I just had a look at the code again and found nothing that might be Windows specific in any way.

So, I would actually expect the code to run on Linux just fine as well.

But I don't have a Linux system at my disposal right now.

 

I assume you are just asking for some launch script?

What happnes if you try to run it as follows:

java -jar TDMMissionDbCheck.jar

With respect to the 1 false positive:

The localizations pack for 'Special Delivery' show up only in the XML version of the mission list, but not in the PHP version.

http://missions.thedarkmod.com/available_missions.xml

shows

<localisationPack url="http://www.fidcal.com/darkuser/missions/delivery_l10n.pk4" />
<localisationPack url="http://darkmod.taaaki.za.net/fms/delivery_l10n.pk4" />
<localisationPack url="http://www.southquarter.com/tdm/fms/delivery_l10n.pk4" />

whereas this one doesn't :

http://missions.thedarkmod.com/get_available_missions.php

This means that you will see no Translation seems available when browsing this mission on the website (which uses the PHP version) http://www.thedarkmod.com/missiondetails/?id=22,

whereas the internal downloader (which uses the XML version) is well aware of these 'localization packs'.

 

That's why the latest scan indicates an inconsistency there.

 

Anyway: These specific 'localization 'files should be probably dropped entirely, as they are 100% identical to the actual mission file delivery.pk4.

So, they aren't a localized version at all. Or are they?

I see some risk that any future update to that mission might mean that people would be playing the new version, but with a bunch of stuff superseeded by an outdated version.

Maybe the files on the server should be replaced with an empty pk4, to make sure everyones local copy of the L10N files get overwritten if there should ever be an update to the mission itself.

Edited February 3, 2019 by gnartsch

[gnartsch]

A small update :

 

I just updated to TDM 2.07 - and to my surprise that L10N package for 'Special Delivery' no longer shows up.

Does that mean that the internal Downloader is now using the PHP version of the mission list - instead of the XML version?

 

In that case an update to my tool would be required sooner or later, as it would complain about the XML and PHP version running out of sync over time.

 

Still, anyone already having that delivery_l10n.pk4 would run into trouble if that mission would ever be updated.

I made a dirty little experiment - just for fun:

I created a copy of newjob.pk4, renamed it to training_l10n.pk4 and moved it along side the training.pk4.

Guess what mission I ended up playing when installing the "Training Mission"?

Right, "A New Job" !

So, any L10N package takes 100% precedence over the actual mission file.