Jump to content


Photo

Tool for analyzing mission links from the Mission List


  • Please log in to reply
123 replies to this topic

#101 gnartsch

gnartsch

    Member

  • Member
  • PipPip
  • 489 posts

Posted 28 April 2015 - 03:50 PM

 

$ curl --head http://swift-mazes.com/fms/antr.pk4    

...
Content-Length: 33057589
Content-Type: application/octet-stream

Weird. I wonder what curl is doing different than the Java code I use.

swiftmazes is the only server which won't report the Content Length for a mission when doing

   URLConnection tmpConnection = new URL(url).openConnection();
   tmpConnection.getContentLength();

That's why that tool needs to ignore swiftmaze with respect to the checking file sizes... unless I figure out what I am missing.

All I get is the following, and digging deeper into possible options and alternatives to modify the behaviour of the URLConnection did not reveal any insights so far either.

Transfer-Encoding=[chunked], Keep-Alive=[timeout=5, max=100], null=[HTTP/1.1 200 OK], Server=[Apache/2.2.29], Connection=[Keep-Alive], Date=[Tue, 28 Apr 2015 20:39:33 GMT], Content-Type=[text/html]}

The official internal name for "Thief's Den" is now 'thiefsden', as should be the name of the pk4.

See http://missions.thed...le_missions.php


Edited by gnartsch, 28 April 2015 - 03:52 PM.


#102 Tels

Tels

    Mod hero

  • Member
  • PipPipPipPipPip
  • 15024 posts

Posted 30 April 2015 - 10:04 AM

Weird. I wonder what curl is doing different than the Java code I use.

swiftmazes is the only server which won't report the Content Length for a mission when doing

   URLConnection tmpConnection = new URL(url).openConnection();
   tmpConnection.getContentLength();

That's why that tool needs to ignore swiftmaze with respect to the checking file sizes... unless I figure out what I am missing.

All I get is the following, and digging deeper into possible options and alternatives to modify the behaviour of the URLConnection did not reveal any insights so far either.

Transfer-Encoding=[chunked], Keep-Alive=[timeout=5, max=100], null=[HTTP/1.1 200 OK], Server=[Apache/2.2.29], Connection=[Keep-Alive], Date=[Tue, 28 Apr 2015 20:39:33 GMT], Content-Type=[text/html]}

The official internal name for "Thief's Den" is now 'thiefsden', as should be the name of the pk4.

See http://missions.thed...le_missions.php

 

Thanx for the info, if taaki or Biker delete the "thiefs_den" file, I'll nuke it, too.

 

As for the size: I'm not familiar with what the Java code does underneath, but may guess is it doesn't query it with HEAD; but a regualar GET request and then aborts? And this seems to confirm it:

https://stackoverflo...e-of-a-web-file

 

Basically, a "GET" will do a "chunked-transfer", where the lenght can't be sent, while a "HEAD" will work properly (and is actually the right way to get only the header info). With a GET, the server is prepared to send you the entire file, wether you want it or not. Sending, then aborting such a request will just lead to unec. work on the server side.

 

There is a bit of code on that page which shows how to use a HEAD request. I would program your tool so it first tries a HEAD request, and if that doesn't work, falls back to the regular method.

 

Hope this helps!


"The reasonable man adapts himself to the world; the unreasonable one persists in trying to adapt the world to himself. Therefore, all progress depends on the unreasonable man." -- George Bernard Shaw (1856 - 1950)

"Remember: If the game lets you do it, it's not cheating." -- Xarax

#103 Tels

Tels

    Mod hero

  • Member
  • PipPipPipPipPip
  • 15024 posts

Posted 30 April 2015 - 10:12 AM

Stashing the L10N files into some backup-'archive' isn't my primary intention either.

But from what I know far more mission received a translation, than the number of mission got actually enabled for L10N.

And it still seems like quite a few authors don't want their missions to be touched by anyone else.

That's the issue here, I guess.

So, moving the unused ones away is just a means to keep the primary directory clean.

 

Yeah, but only for the case of "Author doesn't want to" && "it wasn't enabled".

 

If the author gave permission, it can't be hard to "enable" the mission and just use the files? We do have more or less automated tools for that. Would be much better than to just delete the work of the translators...

 

If someone needs help with that, let's just do it.


"The reasonable man adapts himself to the world; the unreasonable one persists in trying to adapt the world to himself. Therefore, all progress depends on the unreasonable man." -- George Bernard Shaw (1856 - 1950)

"Remember: If the game lets you do it, it's not cheating." -- Xarax

#104 Bikerdude

Bikerdude

    Member

  • Banned
  • PipPipPipPipPip
  • 20320 posts

Posted 30 April 2015 - 02:50 PM

Thanx for the info, if taaki or Biker delete the "thiefs_den" file, I'll nuke it, too.

Already done.



#105 gnartsch

gnartsch

    Member

  • Member
  • PipPip
  • 489 posts

Posted 30 April 2015 - 03:47 PM

Thanks for the tip, Tels!

Unfortunately the resulting effect is terrible!

Instead of simply providing the data in a 'cleaner' way, the HEAD requests causes the entire PK4 to be downloaded as soon as I try to get the header info.

Seems like there must be some more details to this issue.

Oh, ... and even after the PK4 was fully downloaded, the file-size was still not to be found anywhere in the response or header. A mystery.



#106 Tels

Tels

    Mod hero

  • Member
  • PipPipPipPipPip
  • 15024 posts

Posted 01 May 2015 - 06:02 AM

Already done.

 

Strange:

 


$ ls -la thief*    
   112908 Jan 24  2014 thiefs_den_l10n.pk4
  3459871 Apr 12  2014 thiefsden.pk4
   112908 Jan 24  2014 thiefsden_l10n.pk4

 

Is the "new" version now without l10n? That would be a step back :(


"The reasonable man adapts himself to the world; the unreasonable one persists in trying to adapt the world to himself. Therefore, all progress depends on the unreasonable man." -- George Bernard Shaw (1856 - 1950)

"Remember: If the game lets you do it, it's not cheating." -- Xarax

#107 Tels

Tels

    Mod hero

  • Member
  • PipPipPipPipPip
  • 15024 posts

Posted 01 May 2015 - 06:03 AM

Thanks for the tip, Tels!

Unfortunately the resulting effect is terrible!

Instead of simply providing the data in a 'cleaner' way, the HEAD requests causes the entire PK4 to be downloaded as soon as I try to get the header info.

Seems like there must be some more details to this issue.

Oh, ... and even after the PK4 was fully downloaded, the file-size was still not to be found anywhere in the response or header. A mystery.

 

Uh, that is exactly backwards!

 

Maybe it is that problem:

 

https://community.or.../thread/2112393


"The reasonable man adapts himself to the world; the unreasonable one persists in trying to adapt the world to himself. Therefore, all progress depends on the unreasonable man." -- George Bernard Shaw (1856 - 1950)

"Remember: If the game lets you do it, it's not cheating." -- Xarax

#108 gnartsch

gnartsch

    Member

  • Member
  • PipPip
  • 489 posts

Posted 09 June 2015 - 02:25 AM

An issue in the mission database:

 

seems like Requiem received a silent update?

 

The version on taaki's server is slightly bigger and newer than the other two mirrors.

Once again I see that the other two references are pointing to the TDM2testmissions folder.

I guess the updated version got pushed to the regular missions folder instead.

I would recommend to clean up those folders and move the current versions to the regular folder.

I don't think that all that stuiff is still under test.

 

'http://darkmod.taaak...ms/requiem.pk4'
OK
   Size: 102.5 MB (107437077)
   Last update: Sat May 30 09:35:37 CEST 2015

 

'http://www.fidcal.co...ns/requiem.pk4'
OK
   Size: 102.4 MB (107330042)
   Last update: Sat Oct 05 11:43:42 CEST 2013
   Language: English
   Sources: [PHP, XML]

'http://www.southquar...ns/requiem.pk4'
OK
   Size: 102.4 MB (107330042)
   Last update: Sat Oct 05 11:46:52 CEST 2013



#109 Bikerdude

Bikerdude

    Member

  • Banned
  • PipPipPipPipPip
  • 20320 posts

Posted 09 June 2015 - 03:56 AM

And regarding where they are getting uploaded to, all mission are going to the regular folder on all mirrors.

 

I beleive the issue is that the FM page is using old links, which I need to change. I will also have a look at all the other FMs and make sure none of them are using old links.



#110 gnartsch

gnartsch

    Member

  • Member
  • PipPip
  • 489 posts

Posted 15 November 2015 - 04:57 AM

A few minor issues in the list of download URLs:

 

Penny Dreadful 1: The grail of regrets

bad URL : protocol reads 'ttp' instead of 'http'

ttp://www.fidcal.com/darkuser/missions/penny1_1.pk4

Patently Dangerous

dead link

http://www.southquarter.com/tdm/fms/patently_dangerous_v2.pk4

Thomas Porter 3: Glenham Tower

dead link

http://www.southquarter.com/tdm/fms/glenham.pk4


#111 Bikerdude

Bikerdude

    Member

  • Banned
  • PipPipPipPipPip
  • 20320 posts

Posted 15 November 2015 - 08:40 AM

Fixing now.



#112 skina

skina

    Member

  • Member
  • PipPip
  • 17 posts

Posted 07 June 2016 - 04:59 PM

http://www.thedarkmo...details/?id=111

ttp://darkmod.taaaki.za.net/fms/volta1_1.pk4


#113 Bikerdude

Bikerdude

    Member

  • Banned
  • PipPipPipPipPip
  • 20320 posts

Posted 07 June 2016 - 05:31 PM

Fixing now.



#114 nbohr1more

nbohr1more

    Darkmod PR, Wordsmith

  • Development Role
  • PipPipPipPipPip
  • 9504 posts

Posted 24 August 2016 - 12:24 PM

A new scan attached.

 

Is there a way to remove the swift-maze mirror from the scan?

 

Attached File  analysis.txt   54.49KB   4 downloads


Please visit TDM's IndieDB site and help promote the mod:

http://www.indiedb.c...ds/the-dark-mod

(Yeah, shameless promotion... but traffic is traffic folks...)

#115 Bikerdude

Bikerdude

    Member

  • Banned
  • PipPipPipPipPip
  • 20320 posts

Posted 24 August 2016 - 01:28 PM

Fyi, those mirrors are I believe Springheels.



#116 gnartsch

gnartsch

    Member

  • Member
  • PipPip
  • 489 posts

Posted 26 August 2016 - 01:54 PM

A new scan attached.


attachicon.gif analysis.txt

Whoops!

Someone actually using that little tool?

I am baffled and surprised!  ;) 



#117 Springheel

Springheel

    Creative Director (retired)

  • Admin
  • 37755 posts

Posted 26 August 2016 - 02:51 PM

A new scan attached.

 

Is there a way to remove the swift-maze mirror from the scan?

 

attachicon.gifanalysis.txt

 

Swift-mazes belonged to Tels IIRC.


TDM Missions:   A Score to Settle   *   A Reputation to Uphold   *   A New Job   *    A Matter of Hours
 
Video Series:   Springheel's Modules   *   Speedbuild Challenge   *   New Mappers Workshop  *   Building Traps

#118 SeriousToni

SeriousToni

    Beginner Contest 2012 Winner

  • Member
  • PipPipPipPip
  • 2207 posts

Posted 26 August 2016 - 06:02 PM

Indeed, that is correct, but he has not been around lately and sadly.


"Einen giftigen Trank aus Kräutern und Wurzeln für die närrischen Städter wollen wir brauen." - Text aus einem verlassenen Heidenlager

#119 Bikerdude

Bikerdude

    Member

  • Banned
  • PipPipPipPipPip
  • 20320 posts

Posted 26 August 2016 - 08:41 PM

i will get around to sorting that out at some point.



#120 nbohr1more

nbohr1more

    Darkmod PR, Wordsmith

  • Development Role
  • PipPipPipPipPip
  • 9504 posts

Posted 29 January 2019 - 11:22 AM

I've been slowly cleaning-up the issues in my latest scan.

 

For some reason, I thought that stgatilov had updated the mirrors with zips where media was unpacked to match the local copies

when I uploaded to southquarter after they recovered from the hack.

 

Any chance for a linux version so I can do some of these on my break at work?


Please visit TDM's IndieDB site and help promote the mod:

http://www.indiedb.c...ds/the-dark-mod

(Yeah, shameless promotion... but traffic is traffic folks...)

#121 nbohr1more

nbohr1more

    Darkmod PR, Wordsmith

  • Development Role
  • PipPipPipPipPip
  • 9504 posts

Posted 29 January 2019 - 11:19 PM

Fixed all the mismatches.

A few false positive errors about translation packs in there.

Attached File  analysis.txt   115.58KB   5 downloads
  • Anderson likes this
Please visit TDM's IndieDB site and help promote the mod:

http://www.indiedb.c...ds/the-dark-mod

(Yeah, shameless promotion... but traffic is traffic folks...)

#122 grayman

grayman

    Master Builder, Coder

  • Active Developer
  • PipPipPipPipPip
  • 12924 posts

Posted 30 January 2019 - 05:23 PM

Thanks for doing this, @NB.



#123 gnartsch

gnartsch

    Member

  • Member
  • PipPip
  • 489 posts

Posted 03 February 2019 - 05:04 AM

Any chance for a linux version so I can do some of these on my break at work?

 

A few false positive errors about translation packs in there.

Thanks for looking into this, nbohr1more !

 

 

A Linux version shouldn't be a problem at all.

I just had a look at the code again and found nothing that might be Windows specific in any way.

So, I would actually expect the code to run on Linux just fine as well.

But I don't have a Linux system at my disposal right now.

 

I assume you are just asking for some launch script?

What happnes if you try to run it as follows:

java -jar TDMMissionDbCheck.jar

With respect to the 1 false positive:

The localizations pack for 'Special Delivery' show up only in the XML version of the mission list, but not in the PHP version.

http://missions.thedarkmod.com/available_missions.xml

shows

<localisationPack url="http://www.fidcal.com/darkuser/missions/delivery_l10n.pk4" />
<localisationPack url="http://darkmod.taaaki.za.net/fms/delivery_l10n.pk4" />
<localisationPack url="http://www.southquarter.com/tdm/fms/delivery_l10n.pk4" />

whereas this one doesn't :

http://missions.thedarkmod.com/get_available_missions.php

This means that you will see no Translation seems available when browsing this mission on the website (which uses the PHP version) http://www.thedarkmo...ndetails/?id=22,

whereas the internal downloader (which uses the XML version) is well aware of these 'localization packs'.

 

That's why the latest scan indicates an inconsistency there.

 

Anyway: These specific 'localization 'files should be probably dropped entirely, as they are 100% identical to the actual mission file delivery.pk4.

So, they aren't a localized version at all. Or are they?

I see some risk that any future update to that mission might mean that people would be playing the new version, but with a bunch of stuff superseeded by an outdated version.

Maybe the files on the server should be replaced with an empty pk4, to make sure everyones local copy of the L10N files get overwritten if there should ever be an update to the mission itself.


Edited by gnartsch, 03 February 2019 - 05:20 AM.

  • Anderson likes this

#124 gnartsch

gnartsch

    Member

  • Member
  • PipPip
  • 489 posts

Posted 03 February 2019 - 06:09 AM

A small update :

 

I just updated to TDM 2.07 - and to my surprise that L10N package for 'Special Delivery' no longer shows up.

Does that mean that the internal Downloader is now using the PHP version of the mission list - instead of the XML version?

 

In that case an update to my tool would be required sooner or later, as it would complain about the XML and PHP version running out of sync over time.

 

Still, anyone already having that delivery_l10n.pk4 would run into trouble if that mission would ever be updated.

I made a dirty little experiment - just for fun:

I created a copy of newjob.pk4, renamed it to training_l10n.pk4 and moved it along side the training.pk4.

Guess what mission I ended up playing when installing the "Training Mission"?

Right, "A New Job" !

So, any L10N package takes 100% precedence over the actual mission file.


  • Anderson likes this




0 user(s) are reading this topic

0 members, 0 guests, 0 anonymous users