Download hosting

[Dimand]

Hi everyone.

 

Just downloading this again for a bit of a play and one thing that is bugging the hell out of me is, DAM this download server is slow.

The fact that we can only grab a file at a time probably doesn't help, possibly you could edit the downloader to grab multiple files of multiple hosts?

 

Anyway, I have a server that has minimal use throughout most of the year. It gets traffic about twice a semester when i run a game of assassins at the uni. Would you like me to mirror up a copy of the mod, I'm not promising blazing fast speeds but the more the merrier hey?

[stumpy]

the updater downloads a zipped file of all the updated files then patches the old files with the files in that zip, it only downloads one file.

[Dimand]

the updater downloads a zipped file of all the updated files then patches the old files with the files in that zip, it only downloads one file.

 

Well I'm downloading from scratch and believe me, it is downloading each file individually.

 

Checking download for 'is-a-zipfile'.
Checking CRC of downloaded file, expecting 4a0a662b
CRC calculated for zip file C:\Program Files\Doom 3\darkmod\__tdm_textures_stone_brick01.pk4 = 4a0a662b
Downloaded file passed the integrity checks.
Moved C:\Program Files\Doom 3\darkmod\__tdm_textures_stone_brick01.pk4 to C:\Program Files\Doom 3\darkmod\tdm_textures_stone_brick01.pk4
Downloading to temporary file C:/Program Files/Doom 3/darkmod/__tdm_textures_stone_cobblestones01.pk4
Initiating Download from http://thedarkmod.robertrueger.de/tdm_textures_stone_cobblestones01.pk4
Download successful: http://thedarkmod.robertrueger.de/tdm_textures_stone_cobblestones01.pk4
Checking filesize of downloaded file, expecting 85089416
Checking download for 'is-a-zipfile'.
Checking CRC of downloaded file, expecting f9c7862f
CRC calculated for zip file C:\Program Files\Doom 3\darkmod\__tdm_textures_stone_cobblestones01.pk4 = f9c7862f
Downloaded file passed the integrity checks.
Moved C:\Program Files\Doom 3\darkmod\__tdm_textures_stone_cobblestones01.pk4 to C:\Program Files\Doom 3\darkmod\tdm_textures_stone_cobblestones01.pk4
Downloading to temporary file C:/Program Files/Doom 3/darkmod/__tdm_textures_stone_flat01.pk4
Initiating Download from http://www.southquarter.com/tdm/tdm_textures_stone_flat01.pk4

 

Perhaps when it is just updating something it grabs a compressed file. It's not an issue, only my connection means i could be downloading about 7 files at once with the upload speed of these servers.

 

It is far from a big issue, I'm just not use to having to wait over three hours to get 2GB

[Serpentine]

Hey there, there are many reasons for doing it the way we currently do - while we understand that using concurrency would speed it up, it's more of a matter of looking after our servers and making sure there's a shared quality of service. That said, if it was under 200k/s or something, then I think it's worth looking into. I know that one of our major ISP's in South Africa limit http threads to 50k/s, so single threaded downloads can be extremely annoying on a good connection.

 

So, can you remember any speed? do you remember which server it was pulling from?

 

As for additional servers, I think we're ok for now, I should be adding two new rather quick mirrors (Germany and Australia) in the next few days, just need to configure and test some stuff. Maybe we can look at relegating some of the slower/high load ones to try keep things quick

[Dimand]

From a simplicity of programming point of view I understand entirely why only one file is requested at a time.

 

 

From memory my speed was around 150-250 kBytes/sec depending on which server the file was grabbed from. the updater seems to select a random (or possibly based on server load) server to get the file on, I wasn't watching which servers were faster than others.

 

I'm in aus (not that proximity makes a large difference for this), so news of a faster server is good. I'm curious as to how the updater picks the download server for each file as some files are much larger. You don't want to get stuck with the 50kb/s server with one of the larger files .

 

 

As a side point, excellent mode and in all reality, this is not a huge issue. I still remember the days of a shared dial up line where obtaining something like this would be literally impossible, this is a dream compared to that.

[Serpentine]

I'm in aus (not that proximity makes a large difference for this), so news of a faster server is good. I'm curious as to how the updater picks the download server for each file as some files are much larger. You don't want to get stuck with the 50kb/s server with one of the larger files .

Well, it's very simple - as most of the updater is meant to be as simple as possible and not rely on fancy yet over complex stuff. Even the differential zip handling is very elegant in the way it works (and compared to commercial patching methods, far less problematic!). The server choice is essentially a weighted random chance, each server that the downloader knows about is allocated a probability of being selected, something like : P(use_server x) = 0.3; this server would then have a 30% chance of being used for your download, the download client doesn't actually know anything of the location/speed or load of the server.

 

As you know, this isn't always optimal for people downloading, however it means that just about any host can be added and it's simple enough to not run into a lot of problems. The largest reason for this method using used is that while speed is important, a lot of shared/vps hosting is limited on IO/s from disk as well as web-server worker threads, since we can weight the probability, it stops any logic from overloading a weak server, hopefully.

 

If only there were some nice cheap dedicated sales on with gbit egress, 100mbit is really feeling quite old now ;/