Python and AWS Cookbook (Ebook) 50% off!

Mitch Garnaat’s excellent Python and AWS Cookbook is now 50% off($6.49) in Ebook format (ePub, Mobi, PDF). The book features some great recipes, straight from the maintainer of boto.

While the book isn’t unfriendly to those looking at boto for the first time, it really shines for those who have done some tinkering with boto in the past. Mitch gets right to the point, providing ample explanations for each recipe. The EC2 sections were particularly useful for me. We use boto heavily at DUO, but I managed to learn some great new tricks with instance management by reading through the examples.

For about $7, this is a great read, and a great way to show appreciation for an excellent project.

Addendum:There is no monetary motivation for my cheerleading; I thought I’d share this with others (the special is easy to miss).

nginx AWS ELB name resolution with resolvers

If you are running nginx as a proxy in front of An Amazon Web ServicesElastic Load Balancer (ELB), it is not safe to merely define an upstream using the hostname of ELB and call it a day. By default, nginx will only do name resolution at startup time, caching the resolved IP address infinitely.

ELB instances scale up and down based on your traffic levels, often changing IP addresses in the process. It seems to be that increased traffic leads to Amazon spawning a new, beefier ELB instance, then changing the DNS record to point at the new instance. It’ll keep the old ELB instance around for a little while to give you time to resolve the new one, but the old instance (using the old IP) will be retired after a short period of time. We need nginx to be able to periodically re-resolve the load balancer’s hostname so service interruptions aren’t encountered due to the IP address change.

The fix

Fortuantely, this one is really simple to remedy. You need only use the resolver config directive in your nginx config. By specifying a DNS server with the resolver directive from within nginx, you signify that it should check with said server every five minutes (by default) to see if the upstream ELB has changed IPs. This is done in a non-blocking manner, and should pose no real threat to your server’s throughput.

The other critical piece is that you must add a $request_uri to the end of whatever proxy_pass value you’ve specified. DNS caching will remain without this, meaning you are no better off. See the example below.

Example

http {
   [...]

   # Causes DNS resolution of upstreams every 5 minutes.
   resolver 172.16.0.23;

   [...]

   server {
      [...]

      proxy_pass http://somewhere.com$request_uri

      [...]
   }
}

The resolver directive can be used in http, server, and location sections, so you can get as specific or as broad as you’d like.

The future fix

A later version of nginx will honor DNS TTLs, so look forward to that. I’ll try to remember to update this article when this lands.

media-nommer scampers closer to initial release

Our Python-based, open source (BSD) distributed encoding system,media-nommer, is inching closer to what we’ll call an initial 1.0 release. We’d love to have other developers take a look, try our documentation, and help us chase down flaws before we stick a 1.0 on this thing and put it out there for others to rip to pieces.

Here are the basics:

  • An orchestrator/management daemon runs on some arbitrary machine of your choice (your own, in EC2, Rackspace, wherever). It provides a simple JSON API that your applications can send encoding jobs to. It also handles spinning up new instances and communicating job state to your applications.
  • Encoding nodes are spawned on Amazon EC2, with lots of configurable options to determine resource usage. The system can scale as far as EC2 will let you keep spinning up instances.
  • Encoding is handled through “Nommers“. Each Nommer wraps a different kind of encoding software. The only Nommer currently is FFmpegNommer, which wraps ffmpeg. We’d love to see some other audio and video-related Nommers added (mencoder, anyone?).
  • The EC2 encoder nodes are never in direct contact with the master management daemon. The management daemon can be stopped and restarted at a later date (or on another machine) without any interruptions or data loss. No firewall holes needed.

We are dogfooding media-nommer like crazy on two very large projects at DUO, and it’s working really well for us. However, we’d love to have some other eyes and hands on the project, so please do consider checking it out. If you find yourself using Zencoder or Encoding.com with any kind of regularity, media-nommer just may save you a lot of money.

CouchDB as a MUD server data store

I’ve been using CouchDB as the data store for my in-developmentMUDDawn of the Titans. So far it’s been very enjoyable to work with, through the CouchDB Python module. I’ll take a moment to share my experiences, for those who might be interested.

To provide some background, my MUD server is built specifically for the game I’m working on, but I’ve been developing it in the open on GitHub. The whole thing is built on Twisted, and is loosely styled after TinyMUX 2 (with Python in place of C++ and SoftCode).

Why CouchDB?

This is probably the first question on most people’s minds. There was nothing overly scientific about the choice of CouchDB. For me, this was a very uninteresting choice. I wasn’t really interested in querying whatever DB I used, as I wanted to keep almost everything memory-resident. I didn’t need much scalability at all, and I didn’t really need a relational database. The only thing I really needed a DB for was persistence.

In the end, I thought CouchDB’s use of JSON documents was pretty neat, and figured they’d allow for a really simple way to store objects. It was also a chance to learn something new (which was the biggest factor of all).

The Perks

So far, CouchDB has been a joy to work with. I realize that almost all of these are possible with X relational/non-relational DB, but I still give an approving nod to CouchDB in these cases.

The biggest benefit to using CouchDB is that my in-game object loading code looks something like this (pseudocode):

# Retrieve the object's document from CouchDB. This is a JSON
# dict, whose keys are the object's various attributes (name,
# description, location, etc).
object_doc = self._db[doc_id]
# A simplified example of how a Room is loaded. The CouchDB's keys
# are expanded as kwargs to the Room's __init__. Presto, loaded object.
loaded_object = RoomObject(**object_doc)

This may not seem like anything earth-shattering (it really isn’t), but it makes things very simple to manage and expand on. RoomObject’s __init__ method knows exactly what to do with a CouchDB object document.

Since CouchDB just deals with JSON documents, I can add/remove keys (which become object attributes in-game) without hassling with schema modifications or explicitly specifying fields. I’ve been surprised at how little time I spend mucking with the DB. I’m free to just focus on dealing in the realm of my MUD server, without worrying too much about my data store.

Another great thing about CouchDB is Futon, the web-based management console for CouchDB. It has made editing objects a breeze. I do this constantly when tinkering with objects. My set of in-game building commands is currently very limited, so this helps me keep getting things done while those materialize.

The last cool thing I’ll mention is that saving objects to the DB can easily be made asynchronous, and you can bulk submit objects (instead of saving one at a time). While async DB operations are expected for most DBs, CouchDB’s calls/queries are really easy to work into a Twisted project (without resorting to threads/processes), since the calls are all just HTTP requests (that can be deferred via Twisted’s HTTP client). Take a look at Paisley for an example of how simple it is to perform non-blocking queries.

The downsides

There is only really one downside (for what I’m doing) to CouchDB, and many non-relational stores in general: I have to manually assure that all ‘fake relations’ stay valid, and fail gracefully if they end up invalid. For example, let’s say I have an Exit that leads to ‘Test Room’. If said room is deleted, the exit should be un-linked or deleted. Leaving the exit in place means that someone attempting to travel through it (to the now non-existant room) would see an error message, since the ‘destination’ attribute points to an invalid object ID.

Most MUD servers have to do this kind of cleanup on their own anyway, I’m just somewhat spoiled from my time spent with Postgres/MySQL/SQLite on Evennia, which cleaned up after me (CASCADE!). So this is far from a show-stopper, just something I’m not used to.

The only other thing that could possibly be construed as a down-side is that querying CouchDB feels clumsy to me. This is almost certainly due to knowledge gaps on my end. I didn’t really need this in my case anyway, so no harm here.

In summation

Used in the context of a MUD, CouchDB has been awesome to work with. I’ve enjoyed it much more than my last MUD server projecet using relational DBs. In the end, the choice of data store should be made based on what lets you spend more time on your game, rather than your serialization/persistence/object saving. For any sanely designed MUD, you’re not likely to hit performance issues, and it all comes down to a matter of preference.

Over the speed bumps

One of the worse parts about writing a codebase from scratch for aspecific game is the staggering amount of foundation work that has to happen before you can really start making visible progress. Database layers, object representation and manipulation, account management, exits, and a basic set of global commands are all necessary before much of anything ‘visible’ can happen. Plus, these are only just going to get you started in breaking ground on your world.

I know right now that my project (Dawn of the Titans) lacks such basics as a communication system, a full set of builder commands, the ability to IP ban and flood control, and a sane way to handle color. Furthermore, I have yet to dig room 1 of my world, or start writing the fun stuff like combat and space flight systems.

It’s easy to look at my mile-long to-do list and get intimidated/psyched out, and at times, I have. In particular, the tasks that are either extremely uninteresting, tedious, or require a lot of re-arranging tend to really make me cringe. I end up putting these off, and further progress slows or comes to a halt.

Speed Bumps

There are probably much better terms for this, but I call issues on a to-do list that I dread the most “Speed Bumps”. Sometimes this is due to complexity, other times due to being very uninteresting or tedious. More often than not, these items are required for progress to continue. Unfortunately, these are also the most likely to go neglected.

By putting off these “Speed Bumps”, our progress is slowed, morale may drop, and motivation might go with it. Development momentum slows, and we may even come to a complete stop for periods. This is not good, given that you probably have at least a year’s worth of work to get a 100% custom MUD in playable shape.

Flying through the Speed Bumps

I have found that the best way to get past my cringeworthy tasks is to try to find something really simple and quick that I can do to at the start of a development session. It could be fixing a spelling error or writing a paragraph or two of documentation. It lets me get a commit under my belt, and I feel rewarded for making progress.

From there, the most “dreaded” of my tasks is where I start. After thinking, doodling, and writing a line or two of code, I lose the concrete shoes and I’m off to the races. The hardest part of getting past speed bumps is getting motivated enough to start working.

Attack the things you’re dreading; try to find some way to get excited about them. If it’s a particularly boring task, try to make it interesting by thinking of what you’ll be able to do down the road with what you’re working on.

Also…

For those masochistic enough to write an entire codebase for their particular game in mind, keep one very important thing in mind: You knew going into this that it was going to be a very long road. Don’t be discouraged by this, and instead be happy that you’ve got a great project to poke at when you feel like it over a year or two, or five. If it makes it out the door to live players, wonderful. If it doesn’t, hopefully you at least enjoyed working on it.