Let’s play: python-gotalk

A recent HackerNews post announced Gotalk, a simple bidirectional protocol.I can imagine your collective eyeballs rolling. “Oh great, yet another half-baked way for… things to talk to one other”. But keep following along, maybe you’ll see something you like. Here are some highlights:

• By Rasmus Andersson - You may know him from his work at Facebook, Spotify, and Dropbox.
• Bidirectional — There’s no discrimination on capabilities depending on who connected or who accepted. Both “servers” and “clients” can expose operations as well as send requests to the other side.
• Concurrent - Not too earthshattering, but this is a non-blocking, multiplexed protocol.
• Debuggable - ASCII-based wire format. You can, of course, encode your payloads any number of way, but you’ll have an easy time with your packet sniffers and debug tools.
• Pretty damned simple - There’s not a whole lot to Gotalk. A handful of message types. A pretty simple, easy-to-parse wire format. Nothing crazy for terminology or concepts. If you were curious about network protocols and wanted a gentle intro, this would be a great one to look at.

The official Gotalk repo has a bunch of examples and even some helper libraries in Go and JS. The official implementation is in Go, with the JS libraries being built on top of WebSockets.

But what about Python?!?

“But Greg, I too want to play with this fledging version 0.0 protocol! And I want to do it in Python!”

You, my friend, are in luck! I have a rough, version 0.0 Python module to go with this new, version 0.0 protocol.

For now, most of the effort is on message serialization and deserialization. We’ll be keeping that separate from any of the naughty bits (sockets, IO, and other things). The goal is to provide some reusable components that people can tinker with.

And more importantly, I haven’t ever bothered to mess around at the protocol level very often. This has been a great excuse to play around with a spec that is just getting started.

Pull requests, issues, suggestions, and the whole lot are welcome. Tell me how bad I screwed up!

Closing notes

If it wasn’t already evident, do not use this for anything but tinkering right now!.

Also, to stem the tide of “Well, why didn’t you just use X instead?”, this is a fun little experiment for me. Yes, I wrote a protocol serialization/deserialization for funzies. No, I’m not mentally ill.

EMDR Relay 1.1 Released

EVE Market Data Relay (EMDR) has been chugging along behind the scenesin the EVE Online developer community, quietly delivering large volumes of player-supplied market data. But the winds of change are arriving, as CCP has released a set of HTTP APIs for obtaining much of the data directly. EMDR will continue to function for those who don’t want to poll on their own.

The first step in putting a fresh coat of paint on EMDR is to freshen up the relays. We have done just that, updating and modernizing a few things.

Some hilights of the 1.1 release:

• I have built a Docker image that is now the officially endorsed way to set up and run an EMDR relay.
• We’ve upgraded (and now require) ZeroMQ 4.x. If you use the Docker image, you don’t need to worry about this.
• We now auto-restart the process every 12 hours. This works around some ZeroMQ edge cases where connections aren’t restored correctly. Most of our relay operators are already doing this, but now we all will, just in case.

See our Docker Hub repo for full setup instructions. Current relay operators are encouraged to upgrade, though nothing will break if you don’t.

python-colormath 2.1.0 released

python-colormath 2.1.0 haslanded, bringing with it some excellent new features and bug fixes. See the release notes for a more detailed look at the changes.

The headlining feature is the replacement of our hardcoded conversion tables with NetworkX-based resolution of color conversions (courtesy, Michael Mauderer). Color Appearance Models and IPT round out the rest of the new features.

Installation

The easiest way to get python-colormath is through pip/easy_install:

pip install colormath

Alternatively, grab it from PyPi.

Halp!

If you get stuck, create an issue in the issue tracker and we can figure it out.

Networked, multi-container image crawling with Docker and fig

Like many, I’ve been playing with Docker as it has been developing. I feltreasonably comfortable creating images and running isolated containers, but I had yet to tinker with automatically provisioning and networking multiple containers together.

To remedy this, I spent a weekend writing a simple distributed dockerized image crawler using Twisted, Docker, and fig. The full source (under a BSD license) can be found in the Github repo.

The README covers a lot of this in more detail, but the application is comprised of three container types:

• An HTTP web API for enqueuing jobs. For the sake of this example, you only run one of these, but it’d be easy to adapt the code to support multiple instances.
• A worker container that carries out the image crawling. You can run as many as these as you’d like.
• A Redis container that is used for tracking job state and results.

The worker daemons connect to the HTTP web API container over ZeroMQ, PULLing jobs that the HTTP daemon PUSHes. If this were anything but a toy app, you’d probably want a proper message broker that could offer stronger deliverability and persistence guarantees.

Assorted Observations

• I may have been doing something wrong, but boot2docker seems to be pretty rough with volume mounting. It looks like it’d be pretty easy to mount a directory within the boot2docker VM, but the host machine->container sharing didn’t work in real-time. I expect this will improve with time, or I could have missed something.
• fig is a great idea, and is very handy. I’m looking forward to seeing where this concept is taken over the next few years.
• There is a great python:2.7 image on Docker Hub that serves as a great starting point. They’ve got all sorts of other versions and alternative implementations (PyPy) covered.

python-colormath 2.0 released!

python-colormath was startedback in 2008, when I was an undergraduate at Clemson University (Go Tigers!). While there are a good number of people out there making use of the module effectively, there were a lot of things I wanted to do differently in an eventual 2.0 release. There were some usability issues that arose from my being relatively new to Python at the time.

But all has been made well now. I am happy to announce the immediate release of python-colormath 2.0! A few hilights:

• Better documentation
• Python 3.3+ support added. Python 2.6 and lower no longer supported.
• A complete re-working of RGB and RGB conversions. I’m biased, but I think we’ve now got more correct RGB handling than the vast majority of color math libraries out there, regardless of language.
• While the color space conversion math remains largely untouched, there is now a dedicated color_conversions submodule that is devoted to the cause. I think this is a good clarity/usability win.
• Our unit test suite saw a lot of improvement. They are now easier to write, more complete, and more helpful when failures occur.
• Numpy matrix-driven Delta E functions were incorporated for a nice speed win. In addition to the standard Delta E calls, you can use a vectorized equivalent that is much faster.

See the 2.0 release notes for a more detailed look at the differences.

NOTE: There are backwards incompatible changes in this release. In order to set things right, there were quite a few breakages, but I’ve done the best I can to document these.

Installation

The easiest way to get python-colormath is through pip/easy_install:

pip install colormath

Halp!

If you get stuck, create an issue in the issue tracker and we can figure it out.