An embarrassing S3 saga

Around July of 2009, a post was made to the AWS Developer Forums for S3(which the S3 team reads and responds to) asking for the Access-Control-Allow-Origin header to be allowed on keys. The primary motivator for this is that sites would be able to store things like webfonts in S3.

The S3 team replied shortly after the initial post stating that this wasn’t in their immediate roadmap, but they’d watch the thread to help set priorities in the future. Fast-forward over two years, 101 replies, and 24,000 thread views later and we find ourselves still lacking this capability. If you look at the S3 AWS forum, you’re likely to see ‘Access-Control-Allow-Origin header‘ still near the top.

It’s a shame that a reasonably simple request, which would greatly improve S3’s utility for webfont-bearing sites, has failed to even get a “We’re working on this” from the S3 team. We can’t expect them to drop what they’re doing and get on this, but they’ve remained completely non-committal on this.

The workarounds

The way we get around this issue is that we either:

• Serve webfonts from an nginx instance (not ideal, we’re using S3 so we don’t have to host media)
• Use a competitor’s service (Rackspace CloudFiles) just for fonts. Other CDNs and object stores support this header.

PyPy’s call for donations (NumPy)

Disclaimer: I am not at all involved with PyPy development, planning,or management. You are about to see cheer-leading, but it’s not because this is my project.

PyPy has recently posted  ”Call for donations - PyPy to support Numpy!” There has been some initial ground work laid by Alex Gaynor and others, but it looks like they’re ready to go full speed ahead with the effort now. This is great news for PyPy and Python.

NumPy is one of the defacto scientific computing packages in the Python ecosystem. There are all kinds of other modules that depend on NumPy, and it’s used heavily in research, engineering, and other general sciency stuff.

PyPy is speedy alternative implementation of Python. In many cases, PyPy is able to handily whallop traditional CPython. As of the time of this article’s writing, PyPy’s speed center says “The geometric average of all benchmarks is 0.21, or 4.9 times faster than CPython”.

So… why do I care?

CPython, being an interpreted language, often falls behind other closer-to-the-metal languages and compilers. The scientific community, along with a good number of other modules rely on NumPy. The problem is, CPython isn’t nearly as fast as some of the alternative languages. However, what Python loses in speed, it makes up for in ease-of-use and readability.

With a PyPy-compatible NumPy, we can greatly reduce our speed woes, and open up PyPy compatibility with a very large set of existing modules. The end result being PyPy is one step closer to being ready for everyday use, and it also gets a “killer package”.

Lots of drops in the bucket = ??

There’s only so much we, as individuals can do financially, but do consider making a small donation to the cause. The Python community is large, and contributions will quickly accumulate to something useful. For those who are firmly entrenched in CPython, consider what an aggressive, experimental Python implementation does for the greater Python ecosystem (subject for another post or discussion).

Read the Docs: A huge thanks!

When a free service or module proves to be invaluable to yourproject(s), it’s only right to thank the authors for their excellent work. I thought I’d take a brief moment to thank all of the contributors behind Read the Docs for this simple but wonderful site. I have found it to be extremely useful for my hobby and real-world projects.

For those who have yet to see the light

If you have Sphinx-based documentation and haven’t played with Read the Docs yet, I can’t recommend it enough. Not only do you not have to hassle with hosting your compiled HTML documentation yourself (or on a specially named branch in VC), you can point many software forges’ post-commit hooks at a specific URL to enable automatic doc updating with each commit/push.

I no longer have to manually udpate the docs with each change, or juggle the annoying gh-pages branch on GitHub. Each commit posts to a project-specific URL on Read the Docs, and RTD pulls the latest source, runs Sphinx, and posts the result up within a minute or two.

Small victories in efficiency

If all of this doesn’t seem like a big deal, that’s because it isn’t. This is a small, incremental improvement to my development process that leads to less context switching, less manual labor, and less stuff to worry about. This is not the cure for cancer and it won’t win a Nobel Peace Prize.

However, it is just the very small, simple improvement that makes a noticeable impact. Low-hanging fruit. The tastiest kind.

Evennia MUD server grows a proxy

I’ve obviously been on a bit of a proxied MUD server tangent of late(see here and here). I typically haunt #evennia on FreeNode, and brought the point up to the current maintainer, Griatch, a few weeks ago. As is typical for Griatch when he finds something he likes, he cranked out his take on an AMP-based proxy (the Evennia term is “portal”) very quickly, and announced the specifics a few days ago.

The biggest benefit in the case of Evennia is that the old, complicated code reloading system is now completely gone. This is a big win in simplicity and avoiding nasty edge cases where reloading doesn’t work as expected.

For those that haven’t heard of Evennia, it’s a Python + Twisted + Django MUD server that gives you a solid foundation to immediately start adding your game-specific stuff to. The documentation (particularly the developer section) is outstanding, and is amongst the very top tier of MUD servers. If you are even partially curious, consider joining the Google Group or lurking in #evennia on FreeNode. There are lots of interesting discussions on both from people of varying levels of involvement.

dott - now with more proxy

As mentioned in my previous post about putting a MUD behind a proxy,I’ve been really intrigued by some of the things such a setup would allow. In fact, my motivation was enough to warrant actually trying to implement my own interpretation. My implementation of this split proxy/MUD setup has now been merged into my master branch of dott on GitHub (my tinker MUD that I work on when I get the urge).

A quick refresher

For those who haven’t read the first article, or are too lazy to do so, what I’m doing is sticking a proxy in front of a MUD server. The proxy is the only one of the two that handles telnet connections, maintaining them even if the MUD server is down or restarting. In my case, the proxy also handles authentication and account creation, but this is entirely optional, and just something I wanted. The MUD server only handles game world related stuff: rooms, objects, combat, AI, etc.

The primary benefit of this setup is that instead of dealing with messy live, “seamless” code reloading, I just shutdown and restart the MUD server without alerting the players. The proxy maintains the connections, and the MUD server’s startup procedures are fast enough to where very few players will ever notice that the game went down and back up. If the MUD server is down, the player will get a “The MUD is currently re-loading” message telling them to try again in a few seconds.

How it works

Since my tinker project, dott, is built with Python and the excellent Twisted framework, I had access to AMP, a very simple async messaging protocol. This allows for bi-directional communication between the proxy and the MUD server. The proxy can do stuff like pipe commands to the MUD server through various objects, and the MUD server can tell the proxy to emit messages to any player controlling an object.

Beginning with the startup process, the proxy fires up, starting a telnet server factory to accept telnet connections from players. It imports the MUD server‘s proxyamp module, which contains protocol definitions and an auto-reconnecting client factory used to communicate with the MUD server. Both the proxy and the MUD server use the same ProxyAMP class and its commands contained within.

The MUD server is started up, and the proxy auto-retries its AMP client-> server connection and finds the MUD server reachable over AMP. A connection is established, and the two pieces return to working together.

Nuances

• Both the proxy and MUD server are monitored with supervisor. If either of the two go down, supervisor immediately re-starts them. My @reload command simply shuts down the MUD server. Supervisor sees the exit and restarts it for me with very little delay. The proxy maintains connections and things automatically pick right back up where they left of, when the MUD server returns.
• The MUD server and proxy can be restarted independently and in any order. Shutting down and starting up the proxy leads to no ill effects for the MUD server. It’d just boot the players. Restarting the MUD server leads to no discernible interruption, unless the player types a command. In that case, they get an error message telling them to try again in a few seconds.
• The proxy server handles any protocol-specific display stuff. I’ll eventually be doing room descriptions in some form of markup. The proxy would parse the markup and replace it with the appropriate color/formatting codes for the protocol. For telnet, this would be ANSI color, for a theoretical future WebSocket-based client, this might be JSON or HTML. The important thing to grasp here is that the MUD server is just handling game stuff, which is really neat.

AMP enables the building of distributed MUDs

The following is rambling just for the sake of illustration. I don’t have any immediate plans to do any of the following, since MUDs are not at all resource-intensive, and are best kept simple. With that said…

If one were so inclined, next steps might be breaking out various complicated systems into their own separate services. For example, maybe we have an authentication service that the proxy communicates with over AMP for logging users in. Your website could then also communicate with this authentication service (just like the proxy), instead of hitting the MUD server directly (I don’t really like the idea of hitting a MUD server with web-related traffic). Creating new accounts from the web uses the same plumbing as the proxy.

Or maybe you run a Twisted IMAP service for your game’s mail. Users within the game using mail commands would be reading their mail via AMP messages to the mail service that runs outside of the game. Those with IMAP-enabled clients could hit the mail service separately, instead of hitting a baked-into-the-mud service with potential denial of service or security issues.

Another possible external service would be AI and/or a real-time combat system. The combat system service could handle the ticks, coordinates, and AI stuff that may be resource-intensive, messaging the MUD server when stuff happens. Users steering or moving things around would send AMP messages to the combat system letting it know to change states. This is a cheap way to make use of multi-core machines, if you really need it.

Feedback, ideas, and whatever else

Coming back to reality for a second, I’d love to get some more eyes on the code (on GitHub), or hear any ideas you may have. Please feel free to reply below with a comment. I am writing this stuff for myself for my own selfish use, but constructive criticism is welcome and encouraged. The unit testing is marginal at best at the moment, as I’m trying to figure out the best way to test AMP. However, I’ve tried to make sure the docstrings are detailed.

If there are any slick capabilities that this setup enables, but I haven’t mentioned them in either post, do speak up in case I missed them. There hasve already been some comments that pointed out things I never thought of.