Software blog

NextGen MMO Architecture

For my first post I'm sharing an article I wrote back in 2009, about an MMO server architecture I built in Erlang. Here is a summary, and the full PDF is linked at the bottom.

At the time, a survey of 122 players (Achterbosch, Pierce and Simmons, 2008) showed what people wanted from the next generation of MMOs: a world that feels alive, with forests that burn and regrow, floods that come and go, and NPCs that follow daily routines or can be hired to gather resources and guard buildings. The server designs of the day couldn't support that kind of dynamic game design. The goal of the architecture was to make it possible.

The problem with existing designs

Most MMOs in 2009 used a central client-server model. It is easy to keep consistent and hard to cheat, but a single server runs out of hardware at some point. The usual fix is sharding: copying the world onto separate servers of a few thousand players each. That splits the player base, so friends must pick the same shard, and it causes login queues on busy shards and empty worlds on quiet ones. Peer-to-peer designs avoid the central bottleneck but make cheating and keeping a consistent world much harder.

Some research and commercial games, such as EVE Online and Second Life, split the world into sub-areas that each run on their own server. The catch is that one area can never hold more players than one computer can handle. EVE also struggles to use all its CPU cores, because Python's Global Interpreter Lock holds it back.

Why Erlang

Erlang creates a lightweight process for each object instead of running one big main loop, and an ordinary computer can run hundreds of thousands of them. Distribution, fault tolerance through supervisors, hot code upgrades and a distributed database (Mnesia) are all built in. In Java or C++ these would have to be written from scratch or taken from third-party libraries.

The architecture

The cluster has four kinds of servers, which can run on separate machines or all on one:

• Connection server: the only server clients talk to. Each client gets a small state machine that moves through connected, lobby, playing and connection lost. In the last state the player stays in the game for a while so they can reconnect.
• Account server: stores accounts in Mnesia and validates logins.
• Character server: stores and creates the characters for each account.
• Area servers: a grid of connected regions that together make up the world. Every object or character in an area is its own Erlang process.

The area server core does very little apart from event registration. All other features come from libraries that can be compiled, started and stopped while the server is running. Three libraries come by default: libstd (object registry and hot code upgrades), libchar (character login) and libdist (distribution). New code, such as a combat system, can be swapped in with libstd:upgrade(Module) without restarting anything.

Scaling one area across many machines

The key idea is to give several computers to one area, instead of one computer per area. Area nodes join a shared object registry kept in a distributed Mnesia table, for example with libstd:join(start_area@server1). Erlang process IDs work the same wherever the process runs, so the nodes behave like a single area server.

libdist keeps track of the load on each node. When a node is busier than the least-loaded node, it moves half of its processes over. Moving an object is invisible to the client:

1. The object receives a migrate message.
2. It starts a copy of itself, with the same ID and state, on the target node, and the copy takes over its entry in the registry.
3. The old process forwards any messages that still reach it to the new one.
4. When no messages have arrived for one second, the old process exits.

Results

The test machine had an AMD Phenom II X4 940 and 8 GB of RAM. It ran a 10,000 m area divided into 256 quads, filled with robots that changed direction every 5–10 seconds. With 24,000 robots all four cores were close to 100%, handling about 300,000 messages per second. The area server used 619 MB of memory, and the other servers stayed at about 40 MB each. CPU load grew faster than the robot count, because every movement is broadcast to everything in the same quad.

Limitations and future work

• Memory use will grow once real game data is added to each object.
• Login doesn't yet stop an account from logging in twice, or check that a character belongs to the account.
• Objects near a border should move to the neighbouring area automatically, and spare servers should be assigned to busy areas without manual setup.
• A hierarchy of servers (universe → galaxy → solar system → planet → area) could fit the world better than a flat grid, and would allow seamless travel from a planet's surface to space.

My conclusion back then was that the architecture could serve many thousands of concurrent players on inexpensive hardware and scale out by adding machines. That made it a good foundation for the next generation of MMOs.

Read the full 2009 article, or browse the source code on GitHub:

• NextGen_MMO_Architecture.pdf (PDF, 1.4 MB)
• github.com/flodihn/NextGen (source code)