CouchDB as a MUD server data store

I’ve been using CouchDB as the data store for my in-developmentMUDDawn of the Titans. So far it’s been very enjoyable to work with, through the CouchDB Python module. I’ll take a moment to share my experiences, for those who might be interested.

To provide some background, my MUD server is built specifically for the game I’m working on, but I’ve been developing it in the open on GitHub. The whole thing is built on Twisted, and is loosely styled after TinyMUX 2 (with Python in place of C++ and SoftCode).

Why CouchDB?

This is probably the first question on most people’s minds. There was nothing overly scientific about the choice of CouchDB. For me, this was a very uninteresting choice. I wasn’t really interested in querying whatever DB I used, as I wanted to keep almost everything memory-resident. I didn’t need much scalability at all, and I didn’t really need a relational database. The only thing I really needed a DB for was persistence.

In the end, I thought CouchDB’s use of JSON documents was pretty neat, and figured they’d allow for a really simple way to store objects. It was also a chance to learn something new (which was the biggest factor of all).

The Perks

So far, CouchDB has been a joy to work with. I realize that almost all of these are possible with X relational/non-relational DB, but I still give an approving nod to CouchDB in these cases.

The biggest benefit to using CouchDB is that my in-game object loading code looks something like this (pseudocode):

# Retrieve the object's document from CouchDB. This is a JSON
# dict, whose keys are the object's various attributes (name,
# description, location, etc).
object_doc = self._db[doc_id]
# A simplified example of how a Room is loaded. The CouchDB's keys
# are expanded as kwargs to the Room's __init__. Presto, loaded object.
loaded_object = RoomObject(**object_doc)

This may not seem like anything earth-shattering (it really isn’t), but it makes things very simple to manage and expand on. RoomObject’s __init__ method knows exactly what to do with a CouchDB object document.

Since CouchDB just deals with JSON documents, I can add/remove keys (which become object attributes in-game) without hassling with schema modifications or explicitly specifying fields. I’ve been surprised at how little time I spend mucking with the DB. I’m free to just focus on dealing in the realm of my MUD server, without worrying too much about my data store.

Another great thing about CouchDB is Futon, the web-based management console for CouchDB. It has made editing objects a breeze. I do this constantly when tinkering with objects. My set of in-game building commands is currently very limited, so this helps me keep getting things done while those materialize.

The last cool thing I’ll mention is that saving objects to the DB can easily be made asynchronous, and you can bulk submit objects (instead of saving one at a time). While async DB operations are expected for most DBs, CouchDB’s calls/queries are really easy to work into a Twisted project (without resorting to threads/processes), since the calls are all just HTTP requests (that can be deferred via Twisted’s HTTP client). Take a look at Paisley for an example of how simple it is to perform non-blocking queries.

The downsides

There is only really one downside (for what I’m doing) to CouchDB, and many non-relational stores in general: I have to manually assure that all ‘fake relations’ stay valid, and fail gracefully if they end up invalid. For example, let’s say I have an Exit that leads to ‘Test Room’. If said room is deleted, the exit should be un-linked or deleted. Leaving the exit in place means that someone attempting to travel through it (to the now non-existant room) would see an error message, since the ‘destination’ attribute points to an invalid object ID.

Most MUD servers have to do this kind of cleanup on their own anyway, I’m just somewhat spoiled from my time spent with Postgres/MySQL/SQLite on Evennia, which cleaned up after me (CASCADE!). So this is far from a show-stopper, just something I’m not used to.

The only other thing that could possibly be construed as a down-side is that querying CouchDB feels clumsy to me. This is almost certainly due to knowledge gaps on my end. I didn’t really need this in my case anyway, so no harm here.

In summation

Used in the context of a MUD, CouchDB has been awesome to work with. I’ve enjoyed it much more than my last MUD server projecet using relational DBs. In the end, the choice of data store should be made based on what lets you spend more time on your game, rather than your serialization/persistence/object saving. For any sanely designed MUD, you’re not likely to hit performance issues, and it all comes down to a matter of preference.

Over the speed bumps

One of the worse parts about writing a codebase from scratch for aspecific game is the staggering amount of foundation work that has to happen before you can really start making visible progress. Database layers, object representation and manipulation, account management, exits, and a basic set of global commands are all necessary before much of anything ‘visible’ can happen. Plus, these are only just going to get you started in breaking ground on your world.

I know right now that my project (Dawn of the Titans) lacks such basics as a communication system, a full set of builder commands, the ability to IP ban and flood control, and a sane way to handle color. Furthermore, I have yet to dig room 1 of my world, or start writing the fun stuff like combat and space flight systems.

It’s easy to look at my mile-long to-do list and get intimidated/psyched out, and at times, I have. In particular, the tasks that are either extremely uninteresting, tedious, or require a lot of re-arranging tend to really make me cringe. I end up putting these off, and further progress slows or comes to a halt.

Speed Bumps

There are probably much better terms for this, but I call issues on a to-do list that I dread the most “Speed Bumps”. Sometimes this is due to complexity, other times due to being very uninteresting or tedious. More often than not, these items are required for progress to continue. Unfortunately, these are also the most likely to go neglected.

By putting off these “Speed Bumps”, our progress is slowed, morale may drop, and motivation might go with it. Development momentum slows, and we may even come to a complete stop for periods. This is not good, given that you probably have at least a year’s worth of work to get a 100% custom MUD in playable shape.

Flying through the Speed Bumps

I have found that the best way to get past my cringeworthy tasks is to try to find something really simple and quick that I can do to at the start of a development session. It could be fixing a spelling error or writing a paragraph or two of documentation. It lets me get a commit under my belt, and I feel rewarded for making progress.

From there, the most “dreaded” of my tasks is where I start. After thinking, doodling, and writing a line or two of code, I lose the concrete shoes and I’m off to the races. The hardest part of getting past speed bumps is getting motivated enough to start working.

Attack the things you’re dreading; try to find some way to get excited about them. If it’s a particularly boring task, try to make it interesting by thinking of what you’ll be able to do down the road with what you’re working on.

Also…

For those masochistic enough to write an entire codebase for their particular game in mind, keep one very important thing in mind: You knew going into this that it was going to be a very long road. Don’t be discouraged by this, and instead be happy that you’ve got a great project to poke at when you feel like it over a year or two, or five. If it makes it out the door to live players, wonderful. If it doesn’t, hopefully you at least enjoyed working on it.

PyPy’s call for donations (NumPy)

Disclaimer: I am not at all involved with PyPy development, planning,or management. You are about to see cheer-leading, but it’s not because this is my project.

PyPy has recently posted  ”Call for donations - PyPy to support Numpy!” There has been some initial ground work laid by Alex Gaynor and others, but it looks like they’re ready to go full speed ahead with the effort now. This is great news for PyPy and Python.

NumPy is one of the defacto scientific computing packages in the Python ecosystem. There are all kinds of other modules that depend on NumPy, and it’s used heavily in research, engineering, and other general sciency stuff.

PyPy is speedy alternative implementation of Python. In many cases, PyPy is able to handily whallop traditional CPython. As of the time of this article’s writing, PyPy’s speed center says “The geometric average of all benchmarks is 0.21, or 4.9 times faster than CPython”.

So… why do I care?

CPython, being an interpreted language, often falls behind other closer-to-the-metal languages and compilers. The scientific community, along with a good number of other modules rely on NumPy. The problem is, CPython isn’t nearly as fast as some of the alternative languages. However, what Python loses in speed, it makes up for in ease-of-use and readability.

With a PyPy-compatible NumPy, we can greatly reduce our speed woes, and open up PyPy compatibility with a very large set of existing modules. The end result being PyPy is one step closer to being ready for everyday use, and it also gets a “killer package”.

Lots of drops in the bucket = ??

There’s only so much we, as individuals can do financially, but do consider making a small donation to the cause. The Python community is large, and contributions will quickly accumulate to something useful. For those who are firmly entrenched in CPython, consider what an aggressive, experimental Python implementation does for the greater Python ecosystem (subject for another post or discussion).

Read the Docs: A huge thanks!

When a free service or module proves to be invaluable to yourproject(s), it’s only right to thank the authors for their excellent work. I thought I’d take a brief moment to thank all of the contributors behind Read the Docs for this simple but wonderful site. I have found it to be extremely useful for my hobby and real-world projects.

For those who have yet to see the light

If you have Sphinx-based documentation and haven’t played with Read the Docs yet, I can’t recommend it enough. Not only do you not have to hassle with hosting your compiled HTML documentation yourself (or on a specially named branch in VC), you can point many software forges’ post-commit hooks at a specific URL to enable automatic doc updating with each commit/push.

I no longer have to manually udpate the docs with each change, or juggle the annoying gh-pages branch on GitHub. Each commit posts to a project-specific URL on Read the Docs, and RTD pulls the latest source, runs Sphinx, and posts the result up within a minute or two.

Small victories in efficiency

If all of this doesn’t seem like a big deal, that’s because it isn’t. This is a small, incremental improvement to my development process that leads to less context switching, less manual labor, and less stuff to worry about. This is not the cure for cancer and it won’t win a Nobel Peace Prize.

However, it is just the very small, simple improvement that makes a noticeable impact. Low-hanging fruit. The tastiest kind.

dott - now with more proxy

As mentioned in my previous post about putting a MUD behind a proxy,I’ve been really intrigued by some of the things such a setup would allow. In fact, my motivation was enough to warrant actually trying to implement my own interpretation. My implementation of this split proxy/MUD setup has now been merged into my master branch of dott on GitHub (my tinker MUD that I work on when I get the urge).

A quick refresher

For those who haven’t read the first article, or are too lazy to do so, what I’m doing is sticking a proxy in front of a MUD server. The proxy is the only one of the two that handles telnet connections, maintaining them even if the MUD server is down or restarting. In my case, the proxy also handles authentication and account creation, but this is entirely optional, and just something I wanted. The MUD server only handles game world related stuff: rooms, objects, combat, AI, etc.

The primary benefit of this setup is that instead of dealing with messy live, “seamless” code reloading, I just shutdown and restart the MUD server without alerting the players. The proxy maintains the connections, and the MUD server’s startup procedures are fast enough to where very few players will ever notice that the game went down and back up. If the MUD server is down, the player will get a “The MUD is currently re-loading” message telling them to try again in a few seconds.

How it works

Since my tinker project, dott, is built with Python and the excellent Twisted framework, I had access to AMP, a very simple async messaging protocol. This allows for bi-directional communication between the proxy and the MUD server. The proxy can do stuff like pipe commands to the MUD server through various objects, and the MUD server can tell the proxy to emit messages to any player controlling an object.

Beginning with the startup process, the proxy fires up, starting a telnet server factory to accept telnet connections from players. It imports the MUD server‘s proxyamp module, which contains protocol definitions and an auto-reconnecting client factory used to communicate with the MUD server. Both the proxy and the MUD server use the same ProxyAMP class and its commands contained within.

The MUD server is started up, and the proxy auto-retries its AMP client-> server connection and finds the MUD server reachable over AMP. A connection is established, and the two pieces return to working together.

Nuances

  • Both the proxy and MUD server are monitored with supervisor. If either of the two go down, supervisor immediately re-starts them. My @reload command simply shuts down the MUD server. Supervisor sees the exit and restarts it for me with very little delay. The proxy maintains connections and things automatically pick right back up where they left of, when the MUD server returns.
  • The MUD server and proxy can be restarted independently and in any order. Shutting down and starting up the proxy leads to no ill effects for the MUD server. It’d just boot the players. Restarting the MUD server leads to no discernible interruption, unless the player types a command. In that case, they get an error message telling them to try again in a few seconds.
  • The proxy server handles any protocol-specific display stuff. I’ll eventually be doing room descriptions in some form of markup. The proxy would parse the markup and replace it with the appropriate color/formatting codes for the protocol. For telnet, this would be ANSI color, for a theoretical future WebSocket-based client, this might be JSON or HTML. The important thing to grasp here is that the MUD server is just handling game stuff, which is really neat.

AMP enables the building of distributed MUDs

The following is rambling just for the sake of illustration. I don’t have any immediate plans to do any of the following, since MUDs are not at all resource-intensive, and are best kept simple. With that said…

If one were so inclined, next steps might be breaking out various complicated systems into their own separate services. For example, maybe we have an authentication service that the proxy communicates with over AMP for logging users in. Your website could then also communicate with this authentication service (just like the proxy), instead of hitting the MUD server directly (I don’t really like the idea of hitting a MUD server with web-related traffic). Creating new accounts from the web uses the same plumbing as the proxy.

Or maybe you run a Twisted IMAP service for your game’s mail. Users within the game using mail commands would be reading their mail via AMP messages to the mail service that runs outside of the game. Those with IMAP-enabled clients could hit the mail service separately, instead of hitting a baked-into-the-mud service with potential denial of service or security issues.

Another possible external service would be AI and/or a real-time combat system. The combat system service could handle the ticks, coordinates, and AI stuff that may be resource-intensive, messaging the MUD server when stuff happens. Users steering or moving things around would send AMP messages to the combat system letting it know to change states. This is a cheap way to make use of multi-core machines, if you really need it.

Feedback, ideas, and whatever else

Coming back to reality for a second, I’d love to get some more eyes on the code (on GitHub), or hear any ideas you may have. Please feel free to reply below with a comment. I am writing this stuff for myself for my own selfish use, but constructive criticism is welcome and encouraged. The unit testing is marginal at best at the moment, as I’m trying to figure out the best way to test AMP. However, I’ve tried to make sure the docstrings are detailed.

If there are any slick capabilities that this setup enables, but I haven’t mentioned them in either post, do speak up in case I missed them. There hasve already been some comments that pointed out things I never thought of.