Evennia MUD server grows a proxy

I’ve obviously been on a bit of a proxied MUD server tangent of late(see here and here). I typically haunt #evennia on FreeNode, and brought the point up to the current maintainer, Griatch, a few weeks ago. As is typical for Griatch when he finds something he likes, he cranked out his take on an AMP-based proxy (the Evennia term is “portal”) very quickly, and announced the specifics a few days ago.

The biggest benefit in the case of Evennia is that the old, complicated code reloading system is now completely gone. This is a big win in simplicity and avoiding nasty edge cases where reloading doesn’t work as expected.

For those that haven’t heard of Evennia, it’s a Python + Twisted + Django MUD server that gives you a solid foundation to immediately start adding your game-specific stuff to. The documentation (particularly the developer section) is outstanding, and is amongst the very top tier of MUD servers. If you are even partially curious, consider joining the Google Group or lurking in #evennia on FreeNode. There are lots of interesting discussions on both from people of varying levels of involvement.

dott - now with more proxy

As mentioned in my previous post about putting a MUD behind a proxy,I’ve been really intrigued by some of the things such a setup would allow. In fact, my motivation was enough to warrant actually trying to implement my own interpretation. My implementation of this split proxy/MUD setup has now been merged into my master branch of dott on GitHub (my tinker MUD that I work on when I get the urge).

A quick refresher

For those who haven’t read the first article, or are too lazy to do so, what I’m doing is sticking a proxy in front of a MUD server. The proxy is the only one of the two that handles telnet connections, maintaining them even if the MUD server is down or restarting. In my case, the proxy also handles authentication and account creation, but this is entirely optional, and just something I wanted. The MUD server only handles game world related stuff: rooms, objects, combat, AI, etc.

The primary benefit of this setup is that instead of dealing with messy live, “seamless” code reloading, I just shutdown and restart the MUD server without alerting the players. The proxy maintains the connections, and the MUD server’s startup procedures are fast enough to where very few players will ever notice that the game went down and back up. If the MUD server is down, the player will get a “The MUD is currently re-loading” message telling them to try again in a few seconds.

How it works

Since my tinker project, dott, is built with Python and the excellent Twisted framework, I had access to AMP, a very simple async messaging protocol. This allows for bi-directional communication between the proxy and the MUD server. The proxy can do stuff like pipe commands to the MUD server through various objects, and the MUD server can tell the proxy to emit messages to any player controlling an object.

Beginning with the startup process, the proxy fires up, starting a telnet server factory to accept telnet connections from players. It imports the MUD server‘s proxyamp module, which contains protocol definitions and an auto-reconnecting client factory used to communicate with the MUD server. Both the proxy and the MUD server use the same ProxyAMP class and its commands contained within.

The MUD server is started up, and the proxy auto-retries its AMP client-> server connection and finds the MUD server reachable over AMP. A connection is established, and the two pieces return to working together.

Nuances

• Both the proxy and MUD server are monitored with supervisor. If either of the two go down, supervisor immediately re-starts them. My @reload command simply shuts down the MUD server. Supervisor sees the exit and restarts it for me with very little delay. The proxy maintains connections and things automatically pick right back up where they left of, when the MUD server returns.
• The MUD server and proxy can be restarted independently and in any order. Shutting down and starting up the proxy leads to no ill effects for the MUD server. It’d just boot the players. Restarting the MUD server leads to no discernible interruption, unless the player types a command. In that case, they get an error message telling them to try again in a few seconds.
• The proxy server handles any protocol-specific display stuff. I’ll eventually be doing room descriptions in some form of markup. The proxy would parse the markup and replace it with the appropriate color/formatting codes for the protocol. For telnet, this would be ANSI color, for a theoretical future WebSocket-based client, this might be JSON or HTML. The important thing to grasp here is that the MUD server is just handling game stuff, which is really neat.

AMP enables the building of distributed MUDs

The following is rambling just for the sake of illustration. I don’t have any immediate plans to do any of the following, since MUDs are not at all resource-intensive, and are best kept simple. With that said…

If one were so inclined, next steps might be breaking out various complicated systems into their own separate services. For example, maybe we have an authentication service that the proxy communicates with over AMP for logging users in. Your website could then also communicate with this authentication service (just like the proxy), instead of hitting the MUD server directly (I don’t really like the idea of hitting a MUD server with web-related traffic). Creating new accounts from the web uses the same plumbing as the proxy.

Or maybe you run a Twisted IMAP service for your game’s mail. Users within the game using mail commands would be reading their mail via AMP messages to the mail service that runs outside of the game. Those with IMAP-enabled clients could hit the mail service separately, instead of hitting a baked-into-the-mud service with potential denial of service or security issues.

Another possible external service would be AI and/or a real-time combat system. The combat system service could handle the ticks, coordinates, and AI stuff that may be resource-intensive, messaging the MUD server when stuff happens. Users steering or moving things around would send AMP messages to the combat system letting it know to change states. This is a cheap way to make use of multi-core machines, if you really need it.

Feedback, ideas, and whatever else

Coming back to reality for a second, I’d love to get some more eyes on the code (on GitHub), or hear any ideas you may have. Please feel free to reply below with a comment. I am writing this stuff for myself for my own selfish use, but constructive criticism is welcome and encouraged. The unit testing is marginal at best at the moment, as I’m trying to figure out the best way to test AMP. However, I’ve tried to make sure the docstrings are detailed.

If there are any slick capabilities that this setup enables, but I haven’t mentioned them in either post, do speak up in case I missed them. There hasve already been some comments that pointed out things I never thought of.

A MUD behind a proxy is… potentially great

My perpetual tinker project is a MUD server that may or may not eversee the light of day. In recent adventures, I pursued using Exocet to make my goal of a mostly interruption-less experience for the players a reality. The attempt worked in most cases, but failed horribly in a few others. The failures were bad enough to make me scrap the idea. The next best thing I could think of is to stick a proxy server in front of the MUD server.

The proxy would handle all of the telnet protocol and session stuff, and just dumbly pipe input into the MUD server through Twisted’s AMP (a really neat, simple, 2-way protocol). When a user inputs something, the proxy says to MUD server “A session attached to an Object with an ID of “a20dl3da” input this. The MUD server would then have any object matches route the input through whatever command tables they are subscribed to, causing things to happen in-game.

Communication back to the proxy would happen whenever an an Object’s emit method (IE: print to any sessions controlling this object). The proxy would see if it had a session attached to the given object, and call the TelnetProtocol’s msg() method with the output.

Neat thing #1: Strictly enforced separation

Convention typically dictates that connection and protocol-level things be kept separate from business logic and other more interesting things. However, having separate proxy and MUD server sections of the codebase really enforces that separation in my mind.

Keeping session and protocol-level gunk confined to the proxy makes the MUD server easier to understand, maintain, and test. I find this layout a little easier to mentally digest.

The other cool thing in the future is that adding support for other protocols (websockets, anyone?) can be handled in the proxy, hooking the input/output into AMP commands. Protocols are already in their own island with Twisted, but this separation is much more strictly enforced under this arrangement (which again, I like). The MUD server can speak in a protocol-agnostic format like Markdown or BBcode, and the protocols can format the output for whatever they are serving.

Neat thing #2: Neither proxy nor MUD server care if the other dies

Consider the following two scenarios:

• MUD server dies, proxy stays up.

  The proxy accepts connections, but all input is left with an error message telling the user to stay put until the MUD comes back up. All sessions are maintained, and Twisted’s auto-reconnection facilities continuously tries to get back in touch with the MUD. When it does come back up, business continues as usual without interruption. The MUD server doesn’t care about sessions, and the proxy doesn’t care about in-game objects, rooms, and etc.

• The proxy goes down, but the MUD server stays up.

  This one isn’t quite as neat. In theory, this scenario should be extremely rare. If the proxy goes down, the user is unable to connect to the running game. They’ll need to re-connect once the proxy comes back up. However, the MUD server continues about its business in the meantime, so mobs are moving, the economy is ticking, etc. Once the proxy is back, it re-connects and players can interact with the game world again.

Neat thing #3: We don’t need to bother with code re-loading

The last, and most important, neat thing is that because of neat things #1 and #2, we don’t need to implement code re-loading. If both proxy and MUD server are monitored/auto-restarted by something like Supervisor, the latest version of the game code can be loaded by silently shutting down the MUD server (but leaving the proxy up). Supervisor (or runit, or launchctl, or whatever) sees the server process down, restarts it, and the proxy automatically re-connects as soon as it’s back up.

The end result is that the user may get an error or two if they’re trying to type stuff while the server is down, but the outage should be short and potentially completely unnoticed by some of the players. We don’t need to worry about all of the messyness associated with code reloading, and we can keep the MUD server focused on game logic.

Code to come

I’ve got a proof-of-concept for this arrangement “working”, but it’ll be some time before I am able to restore the existing features of the MUD server to work with the new proxy + MUD server model. I’ll continue to write posts about progress as it happens.

Exocet makes code reloading easy

One of the big philosophical “pillars” I’ve been building my tinkerPython MUD on is that cold restarts (restart the process, clients are disconnected) should be exceedingly rare. Code re-loading in Python can be challenging, but Allen Short has nailed it with his Exocet module.

The coolest part for me (as it pertains to my game) is as follows:

“Exocet is a new way to load Python modules. It separates the act of naming a dependency from the act of creating a module object. As a result, more than one instance of a module can be created from the same source file.”

As a result of this, we can deal with Python modules as objects, and can easily replace them. This is how it ends up looking for me (lots of things removed for the sake of brevity):

import exocet
# Load the general commands module, stuff it in a variable.
general_cmds = exocet.loadNamed(
   'src.game.commands.general',
   exocet.pep302Mapper
)
# Add a command to the command table.
self.add_command(general_cmds.CmdLook())

This is an excerpt from my game’s global command table. The general command module is loaded by exocet instead of Python’s regular import statement. Since we’ve tossed the module reference in a variable, the normal rules of Python garbage collection apply.

While my game does this a little differently, running the line that populates general_cmds again would mean that the old general command module’s reference count would drop to zero, hence it would be garbage collected. In its place, you have the newly loaded general commands module with any new code updates. Here is all I’d need to do to re-load the general commands module after some modifications:

import exocet
# Re-load the general commands module, replacing the old one.    general_cmds = exocet.loadNamed(
   'src.game.commands.general',
   exocet.pep302Mapper
)

Some disclaimers

Of course there had to be a catch, right? I’m still learning how to best use this, but here are a few pointers:

• Your Exocet-based imports will probably always look strange to you, because they are. However, the power you get from them is well worth the “different” look to them. This is a pretty silly point, but I am pretty silly about code cosmetics.
• Be careful about storing references to exocet-loaded modules. If you reference an exocet module from other modules/classes, you may find yourself in a situation where the old module’s reference count never drops to zero, and it isn’t garbage collected. I get around this by using properties to replace things that were previous instance attributes. I probably did a bad job explaining this, sorry!
• Exocet is still in development. It is only available on Launchpad (sadface). However, some pip+bzr magic can make that less of a problem.
• Documentation is limited to a series of blog posts by the author. They are pretty useful, but you’ll have to wade through them if you’re looking for something.

Of Markdown, REST, and others

As I’m working on my side project (You should probably watch it onGitHub to stroke my ego), I’ve been toying with how I’ll eventually handle colors and web content. I’m almost fully set against using MUX/MUSH color codes, as those don’t do the best job of handling where color starts/ends. They’re also not very friendly for web-based output.

I’ve been considering a few different approaches:

• Restructured Text
• HTML
• Something resembling customized BBcode

I really like Restructured Text, but am not quite sure how I’d substitute in the color sequences on the MUD side. HTML (or a subset of it) is possible, but I’m not sure this is a very clean solution. BBcode could work, and is probably the most simple to implement, but I wonder if it’d prove too limiting.

I’ll open this up to other MUD developers for commenting. How do you handle color in a way that plays nice with a variety of different clients/protocols (I’m very interested to hear how games that have both a Telnet and web-based service handle this).