We’ve been talking about the recent server rollbacks in Guild Wars 2 since they were announced back in early January, but with the months passing by the conversation has died down, and each time someone brings it up they’re met with a resounding “Meh, what does it matter?”
The development and maintenance of Guild Wars 2 was a long and arduous process for ArenaNet, the studio behind the MMORPG. To make the game more playable, the team frequently had to roll back the game’s servers, causing a ton of frustration for players who lost data or were otherwise unable to play for a period of time. There are two main reasons for this: first, the engineers at ArenaNet had to realize that the game was programming itself to become unstable under certain conditions, and second, the team needed to roll back the state of the game to ensure that the game’s core systems remained stable. Even a mild server rollback could wipe the slate clean on the world’s climate or level design, and could break the game’s ability
Guild Wars 2 was a great success for NCsoft this past year, and had a lot to do with its continued expansion into new markets. One of the key things in bringing players back to their games is its interesting and flexible content patching model. NCsoft gives players the power to run their own servers, and roll out updates to the client in a localized fashion. This approach allows players to receive new content more quickly than a centralized server rollout, and allows NCsoft to roll out patches without affecting other players.
ArenaNetGuild Wars 2 studio has been scrutinized by many outside observers in recent years, earning a reputation as a studio with unclear and opaque engineering and leadership – and the massive 2019 layoffs didn’t help matters. However, earlier this month, the company finally publicly admitted who is running the studio and Guild Wars 2 since Mike O’Brien left the studio and Mike Zadorozny secretly left at the end of 2019. And now we have an unusually detailed technical blog from Platform Team developer Robert Nekorchuk, which gives us even more hope that ArenaNet’s revival will include a reboot of its public image and transparency.
Nekorchuk describes a particularly dark day in the history of the game for players in May 2020. You may recall that this month the EU mega-server was shut down, costing players a significant amount of money and game time, which was an extremely rare event for Guild Wars 2 . ArenaNet then took steps to compensate players, but this was a minor setback as players complained about compensation gifts and mixed incentives.
Nekorchuk goes into more detail about what happened behind the scenes during those 20 hours, starting with the update from the week before, the database and driver problems (no joke) that contributed to the cause and the worsening of the disaster. It also describes the whole process by which the studio identifies problems, fixes them, gets the hamsters back on track, and prevents future problems. And you may have noticed that there has never been another downtime like this.
The main change for our databases is the increase in alerts for important database parameters, not just system parameters like CPU or disk space. For our real operation, we added some alerts to a third-party tool to improve response time to future problems. In terms of general operations, we have improved the accounting of our AWS infrastructure, we now track more than just the instance type. Our reports now include instance types, generations, drivers and memory types. We have created a generic installation package for all new servers that includes specific driver versions. All future migration plans will update this generic package to ensure this issue does not occur again. We have completed the migration of all remaining database instances and more, providing better performance for better service. Over the past fourteen months, we have achieved an uptime of 99.98%, with only five minor service interruptions affecting users’ connections.
If this is the new ArenaNet… We like it.
View
Related Tags
This article broadly covered the following related topics:
- rollback to service pack 2
- how to rollback last update in sql server
- rollback last transaction sql server
- sql server rollback last transaction
- rollback last query in sql server
