Official Servers and DDoS Attacks

Totem Arts · May 16, 2020

So by now many of y'all have probably noticed that official servers have been popping up on the server list, and I think now we can finally provide an update now that we have some confidence in our current solution.

Over the last week and some change we've been working towards official servers with the aim of mitigating DDoS attacks, as well as previously stated goals such as creating consistent vanilla Renegade X experience and stronger moderation to make Renegade X a better place. We've been bringing folks together from Constructive Tyranny, AllNoobs, and Totem Arts to encourage as much knowledge and resource sharing and cooperation as possible, and the new official servers are the fruit of that effort. Since we started these efforts, we've tried a variety of mitigation techniques and combinations of techniques in order to tackle this issue hopefully once and for all.

DDoS Mitigation Efforts

I think we can safely say at this point that we're winning the battle for DDoS mitigation (again). Over the last few days DDoS attacks have still been going strong, and those effects have definitely been felt in the game servers. Every single day, we're aiming to make those effects felt at least a little bit less than the day before. In fact mitigation has been so effective thus far that once initial connectivity issues were resolved, all disconnects to the official server have been due to minor filtering errors from our upstream provider which were quickly reported and resolved (such as yesterday when many European players were disconnected, or the other day when everyone was disconnected during an Hourglass match). We do strive to prevent these, but these are absolutely temporary/short-term and OK -- it's better to get any issues hammered out sooner rather than later, and we're still actively working on our mitigation solutions in order to try to provide as stable of an experience as possible.

We're glad to say the DDoS issues are receiving the fullest possible attention of Totem Arts and CT sysadmin staff. These efforts wouldn't be possible without both TA and CT hardware and expertise, so special thanks goes out to everyone who's been actively working towards and organizing these efforts: @Cronus, @DoctorB0NG, @Agent, @Fffreak9999, @Sho-kun, @testman, @KrypTheBear, @Sarie, hsqr, and cmg. We've been heavily monitoring our game servers, keeping communication as active as possible, and being as proactive as possible in order to minimize the impact on players who just want to play the game.

Server Highlighting

A small part of these efforts have been that we've implemented new custom server prefixes, largely to highlight/endorse servers and their DDoS mitigation status so that we can guide players to servers they won't be disconnected from. Right now, prefixes you can expect to see and mean are:

[Official] - These servers are official DDoS mitigated servers that we feel confident in. They aim to provide as consistent of a Renegade X experience as possible, and they aim to be as stable as possible. These are moderated by the global moderator team which we recently expanded with more staff members from CT and AN.
[Anti-DDoS] - These servers use the exact same DDoS mitigation as the official servers. Generally these will be unofficial servers that are helping to support our DDoS mitigation efforts, though right now we are also labeling the official servers as this just to emphasize that they're DDoS protected.
[Test] - These servers are any DDoS-mitigated servers that are either new and need testing, or using new/experimental methodologies and need testing. These will come/go constantly, or eventually be re-labeled as one of the other tags once we're confident they can provide at least as stable of an experience as the current mitigated servers.

Right now these are only visible in the launcher, and have not yet been made to be display in-game. This will likely not be in the next patch either (we've got bugs to fix!), but should end up in some upcoming patch down the line. If you use the in-game server browser and can't use the launcher, try to stick to servers with either 'Official' or 'Constructive Tyranny' in their name if they're online, since these are the ones that will have the [Anti-DDoS] and [Official] labels on the launcher.

Moderation

We aim to keep the official servers actively and effectively moderated in the same manner you'd expect other servers to moderate their servers. AFK players will be kicked in a full server, !modrequest will be made to work if it doesn't already, harassment will be dealt with swiftly, team hampering and base-to-base will be addressed, and so forth. We haven't formed any official guidelines yet, but the goal is to largely operate at least as effectively as any other server. The server-specific moderation team is currently expected to be managed mostly by: @Fffreak9999, @Sarie, and @Agent, in addition to any input from the moderation and development team at large.

To help with that effort, and since official servers are generally a cooperative effort between CT, AN, and TA anyways, we've slightly further expanded our global moderation team to include some new folks who were recommended to help out with moderation for the official servers, as well as just global moderation in general (using guidelines which are publicly posted). We've recruited most of the CT and AN staff who were both recommended and not already on the global moderation team, including @testman, @Pennydude17, and @roweboat.

Moving Forward

Moving forward, we aim to continue to provide as stable and enjoyable of an experience as possible. Once we have finished setting up additional servers, we plan to migrate between servers based on time during the day. We'll likely incrementally improve on this as well over time, by making the map vote carry over between servers for example, so nobody ends up getting their map pick stolen from them or just generally confused. For now we're just sticking to official marathon servers, but we're open to the ideas of hosting additional game modes depending on how the future goes and where it takes us.

Long term we hope to reduce server hosting costs for everybody who's been involved in mitigating these DDoS attacks. By bringing everybody together and pooling resources, we may actually be able to end up with lower cost infrastructure than we started with if we pull this off right. That's a win for everybody involved, since the servers for both the official infrastructure and the nodes hosting our game servers and other misc expenses easily add up to many hundreds of dollars every month.

We also have a much larger tool set available to us than we did before this. We have backup plans to deal with DDoS attack mitigation both in terms of traffic, and potential software solutions to aid with mitigation moving forward if the current setup gets broken. It might take time to implement and maintain these, and attacks might just strike right as you're about to win a match and ruin your night, but we hope you'll be patient and come back as we keep working forward, together, to make Renegade X a better, greater place. So, with stable servers finally in place, working together in one big team against an adversary who wishes for nothing short of destruction...

Welcome Back, Commander