How to setup the ingame downloader AKA 30 second timeout fix

[ShrewdTactician]

---------

3-3-17 Added notes that state these settings are now included in the release.

---------

28-2-17 Added basic powershell script for syncing files to webserver.

---------

27-2-17 Changed notes section that refers to pure connections to point to 10KBps fix.

---------

19-2-17 Added cache section.

---------

 

This setup/fixes the builtin downloader to let players download maps from servers own repository. This solves the 30 second reset problem that currently happens.

 

=== TESTING ===

As a player/client, you can test this by connecting to the shrewd gaming server.

As a server, make sure that you apply the client steps to any clients you connect with.

 

Video: (I forgot to mention in the video that it does take more then 30 seconds, which is what the current issue causes it to reset to 10kbps)

 

Copy the sections inbetween the lines (don't copy the lines themselves).

-----------------------------------------------------------

(section to copy)

-----------------------------------------------------------

 

====== ON Server =====

Note: These settings are included by default in public releases of Renegade X.
1. In Renegade X\UDKGame\Config\UDKEngine.ini below the [IpDrv.UdpBeacon] section create this section block:

----------------------------------------------------------------------------------------------
[IpDrv.HTTPDownload]
RedirectToURL=http://www.edtnint.net/renx/maps/
UseCompression=False 

----------------------------------------------------------------------------------------------

 

 RedirectToURL=  ;url to redirect to. Must have trailing slash. Doesn't work with alternative ports (must be port 80).
 UseCompression=    ;are the files compressed

 

 

===== ON Client ======

NOTE: These settings are included by default in public releases of Renegade X.
1. We need to change the timeout on the client to be longer. You can choose either option to change this:

Option 1: Type into the console ingame: set IpDrv.HTTPDownload ConnectionTimeout 12000

Option 2:  In Renegade X\UDKGame\Config\UDKEngine.ini under the IpDrv.UdpBeacon section create this section block:

----------------------------------------------------------------------------------------------

[IpDrv.HTTPDownload]
ConnectionTimeout=12000
ProxyServerPort=0
ProxyServerHost=

----------------------------------------------------------------------------------------------

 

ConnectionTimeout   ;how long in seconds the client can spend downloading before timingout. Can be very high, as the
 client isn't taking up a player slot while downloading maps.

ProxyServerPort   ;the clients web proxy port
ProxyServerHost   ;the clients web proxy IP

 

2. Under the [Core.System] section in the same UDKEngine.ini file, change these settings to the following values:

----------------------------------------------------------------------------------------------

[Core.System]

StaleCacheDays=365
MaxStaleCacheSize=10000
MaxOverallCacheSize=100000
PackageSizeSoftLimit=2000

----------------------------------------------------------------------------------------------

 

This stops the UDK builtin cache cleaner for wiping downloaded content. The defaults were good for 10 years ago when the engine was released, but its no where near good now. Content in the cache is simply the package(map, script, udk package), renamed with its GUID. You can look into the cache.ini file and find the real name of each cache package, and then simply rename it and move it to its proper location in udkgame if you like, but it isn't a necessity .

 

StaleCacheDays  ; How long in days until an unused file is marked as stale.
MaxStaleCacheSize; How much content in MB can be considered stale before being deleted.
MaxOverallCacheSize; How much content in MB can be in the cache before being deleted, old files first.
PackageSizeSoftLimit; Individual max package size in MB.

 

====== On Webserver ===========
On the url you provided, dump all your files into the ROOT of the folder. NO subfolders.
This can be anything, maps files, characters, effects, sounds,,, everything under UDKGame\CookedPC

I have a script that syncs the cookedpc folder and the webfolder. So that im only maintaining one folder on the server,
knowing that if the server can see it, the players will be able to aswell. Remember, on iis to enable the mime types (the files types) for the udk files. For example, create these new mime types:

.udk     application/octet-stream

.u         application/octet-stream

.upk     application/octet-stream

 

Basic Powershell Sync Script

Spoiler

$GameFilePath = "C:\Program Files (x86)\Renegade X\UDKGame\CookedPC\"
$WebFilePath = "\\iis2\c$\inetpub\Virtual\RenX\maps\"

$GameFiles = get-childitem -file -r -Path $GameFilePath
$WebFiles = get-childitem -file -r -Path $WebFilePath

foreach ($GameFile in $GameFiles)
{
    $WebFile = $WebFiles | where {$_.name -eq $GameFile.name}

    Write-Host "GameFile: " $GameFile.Name "Time: " $GameFile.LastWriteTime
    Write-host "WebFile: " $WebFile  "Time: " $WebFile.LastWriteTime

    if($webFile -eq $null -or $GameFile.LastWriteTime -gt $WebFile.LastWriteTime)
    {
        Write-host "Copying File: " $GameFile.Name
        Copy-item $Gamefile.FullName -Destination $WebFilePath -Force -verbose
    }
}

 

 

========== NOTES ================

-Current issue: It currently doesnt automatically download the next map, it just drops players to the main menu. They
can then reconnect and get the next map. I'm looking into what can be done about this. Right now, when players see the "loading map" message on the end game screen, it starts to download the next map, then the client connects to the server and the server seems to kill it. I'm thinking its a code issue and not a config issue at this point. I'm going through the code to find out.

-The pure server connection uses the per client data rate. See the following post for the config settings:

-Files downloaded via the ingame downloader will end up in the UDKGame\Cache folder.

[yosh56]

This has been known, however from what we gathered the ConnectionTimeout variable doesn't actually do ANYTHING; it will still stop then start downloading at 10KBps after about 30 seconds. 

[ShrewdTactician]

 

2 minutes ago, yosh56 said:

This has been known, however from what we gathered the ConnectionTimeout variable doesn't actually do ANYTHING; it will still stop then start downloading at 10KBps after about 30 seconds. 

No, ive tested this on my live server and on a pure server. It works.
If you change the setting on the server it wont work, since its a client setting. It has to be changed on the client.

Edited February 17, 2017 by ShrewdTactician

[yosh56]

3 minutes ago, ShrewdTactician said:

 

No, ive tested this on my live server and on a pure server. It works.
If you change the setting on the server it wont work, since its a client setting. It has to be changed on the client.

We've also tested it on client, but if you believe it works we can look into it. That was over a year ago. 

[ShrewdTactician]

I've got it running on the shrewd gaming server atm. But I can promise you that this fix is specifically to resolve the 30 second timeout which drops back to the 10kbps. If you have trouble getting it working let me know.

[kenz3001]

ConnectionTimeout=

 dose nothing ... its a broken setting

[iTweek.]

The problem we had ever or still still the Launcher seeker

 

[ShrewdTactician]

Spoiler

 

 

38 minutes ago, kenz3001 said:

ConnectionTimeout=

 dose nothing ... its a broken setting

It must do something since I've got it working. I even completely deleted all the maps from a fresh install of the client, turned off the seeker, and watched it download a map ingame at high speed.

[kenz3001]

speed ? / size? / time it takes

localhost ?

[ShrewdTactician]

locally almost instant. Over the internet with a 1mb per client limit on the webserver, about 1 or 2 min to download a map.

[ShrewdTactician]

video proof posted in op.

[yosh56]

You're like a superhero or something. 

[ShrewdTactician]

1 minute ago, yosh56 said:

You're like a superhero or something. 

I'm just a guy with a lot of free time atm.

[yosh56]

Just saying because this was looked into extensively by 3 separate server owners and like 3 different devs. Next you're gonna' tell me you know how to force GPU particle rendering. 

[ShrewdTactician]

Just now, yosh56 said:

Just saying because this was looked into extensively by 3 separate server owners and like 3 different devs. Next you're gonna' tell me you know how to force GPU particle rendering. 

Actually I was trying to work out dx11 support.

[ShrewdTactician]

added cache section to OP

[yosh56]

Also @Fffreak9999 have you tried setting this up on CT's server again?

[Fffreak9999]

I haven't had time to spend on doing it yet.

However I plan to spend some time tomorrow evening to try get this to work.

So I need to include all the Env + Maps files in the directory it points to?

 

Also I think the auto-downloader should be used in parallel since it has the ability to download maps in the rotation, and not just the one currently playing. 

[ShrewdTactician]

Yeah, just dump basically any resource, maps, packs, classes (.u, .udk, .upk) into the root of your http server. I think I found a way to do this on a ftp server as well but I havnt gotten too far into it. You can see how ive got it layed out http://www.edtnint.net/renx/maps/

I also agree in keeping both the launcher and the ingame downloader. While I do plan to fix the downloading between maps changes thing, I think its great when the user has a choice to download them all at once when going via the launcher.

The best part is, the ingame downloader uses the package guid as a way to identify it. So for instance if you have 1 version of a map in the repo, another version in the game release, and another downloaded from the server, then they don't override each other. Basically this means that servers and or players that have different versions of maps, should happily play with each other, as it will load the version it needs. I do need to experiment with this more to work out the whole pattern with this, but it seems fine so far. It even works with other things. So in a way, it is possible for a game client to get updated via connecting to an updated server.

Edited February 23, 2017 by ShrewdTactician

[Fffreak9999]

I'll have a quick look when I can, hopefully I will get it working soon.

[ShrewdTactician]

I realised that if the changes to the client are done in the defaultengine.ini, it wont propagate to current players.
I've worked out the command they can type ingame to fix it, and could be put onto server announcements to help players.

They just need to write this into the console. (console supports copy/paste). It doesn't have the cache section, but atleast they'll be able to download maps.

set IpDrv.HTTPDownload ConnectionTimeout 12000

 

[Fffreak9999]

Managed to get this working on CT servers (using Field as the test).

Will end up adding more maps shortly.

[yosh56]

On 2/23/2017 at 10:28 PM, ShrewdTactician said:

I realised that if the changes to the client are done in the defaultengine.ini, it wont propagate to current players.
I've worked out the command they can type ingame to fix it, and could be put onto server announcements to help players.

They just need to write this into the console. (console supports copy/paste). It doesn't have the cache section, but atleast they'll be able to download maps.

set IpDrv.HTTPDownload ConnectionTimeout 12000

 

Dunno.. I just had it changed in Default from the patch and it worked fine for me. 

[ShrewdTactician]

39 minutes ago, Fffreak9999 said:

Managed to get this working on CT servers (using Field as the test).

Will end up adding more maps shortly.

That's good news!

[Schmitzenbergh]

Does this mean I can finally mute all the people that scream that the seeker isn't working and rip it from the Launcher? (even though it works fine for most of the peeps)

[Fffreak9999]

Dont remove seeker, just leave it as an unmaintained method for download

[Schmitzenbergh]

2 minutes ago, Fffreak9999 said:

Dont remove seeker, just leave it as an unmaintained method for download

Aye. When this is implemented, I'll make the seeker default to not working. Peeps can Always enable it manually via the setting menu.

[Agent]

I've already removed it from the launcher on my local copy @Schmitzenbergh, and I'll commit that sometime tonight. We don't need to retain and maintain unused/deprecated features.

[ShrewdTactician]

I wouldn't be to hasty yet, while the ingame downloader is awesome, it still has some limitations.

1. The multiple map problem, which I will try and get fixed.

2. The config ini files wont transfer across. This can be turned into a non-issue by removing the need for them. As I understand it, the only place that needs it is skirmish. Since that info in the config is static, and is already essentially in the map info already put in by the map maker, the class can be changed to read that info straight out of the map, instead of from the ini files. 

3. The pre videos, this is going to be harder to work out. If you can put a video straight into a map file, and then somehow read them out again from a program like the launcher, then that would be the answer. Trying to figure out how to actually do that,,, is going to be interesting. Otherwise the seeker could always be adapted to support the preview videos as an individual file.

4. The seeker can be used server side as a way to update official/recognised maps to servers without having to release a new renx build. As a server owner, I would love if the seeker/launcher would basically automatically maintain the official/common maps. And I'm sure map makers would love a way to publish their maps without having to wait for an official renx build.

 

[Schmitzenbergh]

7 hours ago, Agent said:

I've already removed it from the launcher on my local copy @Schmitzenbergh, and I'll commit that sometime tonight. We don't need to retain and maintain unused/deprecated features.

It hasn't been touched since forever so that's not a big problem.

Lets keep both downloaders in the first phase. When the built-in version working and running for lets say a week or 2 and no serious bugs are encountered, then we can remove the seeker from the launcher. This way we can always fall back to something.

[kenz3001]

do maps come with shader cache

[ShrewdTactician]

no.

[Ruud033]

5 hours ago, ShrewdTactician said:

2. The config ini files wont transfer across. This can be turned into a non-issue by removing the need for them. As I understand it, the only place that needs it is skirmish. Since that info in the config is static, and is already essentially in the map info already put in by the map maker, the class can be changed to read that info straight out of the map, instead of from the ini files.

This. @Agent could you work this out? The only reason why I still make these .ini's is because I have to reference the map preview image here. Furthermore it's there for the skirimish stuff as @ShrewdTactician said.

As far as I'm concerned I'd rather have these ini's removed.

Just add the settings to the world info here.

5 hours ago, ShrewdTactician said:

3. The pre videos, this is going to be harder to work out. If you can put a video straight into a map file, and then somehow read them out again from a program like the launcher, then that would be the answer. Trying to figure out how to actually do that,,, is going to be interesting. Otherwise the seeker could always be adapted to support the preview videos as an individual file.

Putting a video file into a .udk or .upk file is easy. Reading that would be the trick I think. 

do maps come with shader cache

Only if you cook the map, which we don't do

[ShrewdTactician]

46 minutes ago, Ruud033 said:

Putting a video file into a .udk or .upk file is easy. Reading that would be the trick I think. 

Looking around, I think @Agent got the jump on us. https://github.com/JAJames/Rx_CustomContentPackager

 

Else, the other way to do it, /possibly/ is to use the built-in udk extractor that comes with the game anyway. https://docs.unrealengine.com/udk/Three/BatchExportCommandlet.html

If the video file is always named the same thing inside the map package, then the logic could go. 

1. launcher looks for video file for a map.
2. If not found, run the batchexport command on the upk of the map to get it.

Problem sorted (if works).

Edited February 28, 2017 by ShrewdTactician

[Agent]

Rx_CustomContentPackager was actually for something else, and was going to be used in the in-game downlaoder I was writing (which is now unnecessary).

HTTPDownload files are added to the cache which is temporary storage, whereas the loading movies and INI files are not -- it would be strange at best to extract permanent data from temporary data, which would then get stored permanently (even after the temporary data / level data gets pruned). I think a decent idea though would be to somehow give the user a button (or similar mechanism) to save a level permanently from in-game, which then instantiates an HTTP client to a level's download loading movie and INI file, as well as moving the level out of the cache (and probably renaming it). Even then, this isn't a critical issue. I'm also generally against adding the loading movie to the packages, since this would simply increase package and installation size when there are alternatives which don't (even if they might require slightly more development work).

Regarding removing INI files: I think these were added specifically for the skrimish menu, so that the menu doesn't have to load the entire package to just preview the map. I don't think it'd be a particularly great idea to remove these.

The multiple map issue can probably be fixed very easily if a specific error is thrown anywhere; if you're kicked from a server for lacking data, automatically attempt to rejoin from the main menu. Just need to find that error, more than anything.

[Ruud033]

16 minutes ago, Agent said:

since this would simply increase package and installation size when there are alternatives which don't (even if they might require slightly more development work).

I agree on this.

18 minutes ago, Agent said:

Regarding removing INI files: I think these were added specifically for the skrimish menu, so that the menu doesn't have to load the entire package to just preview the map. I don't think it'd be a particularly great idea to remove these.

Ok I can understand this