EMDR Relay 1.1 Released

EVE Market Data Relay (EMDR) has been chugging along behind the scenesin the EVE Online developer community, quietly delivering large volumes of player-supplied market data. But the winds of change are arriving, as CCP has released a set of HTTP APIs for obtaining much of the data directly. EMDR will continue to function for those who don’t want to poll on their own.

The first step in putting a fresh coat of paint on EMDR is to freshen up the relays. We have done just that, updating and modernizing a few things.

Some hilights of the 1.1 release:

  • I have built a Docker image that is now the officially endorsed way to set up and run an EMDR relay.
  • We’ve upgraded (and now require) ZeroMQ 4.x. If you use the Docker image, you don’t need to worry about this.
  • We now auto-restart the process every 12 hours. This works around some ZeroMQ edge cases where connections aren’t restored correctly. Most of our relay operators are already doing this, but now we all will, just in case.

See our Docker Hub repo for full setup instructions. Current relay operators are encouraged to upgrade, though nothing will break if you don’t.

python-colormath 2.1.0 released

python-colormath 2.1.0 haslanded, bringing with it some excellent new features and bug fixes. See the release notes for a more detailed look at the changes.

The headlining feature is the replacement of our hardcoded conversion tables with NetworkX-based resolution of color conversions (courtesy, Michael Mauderer). Color Appearance Models and IPT round out the rest of the new features.

Installation

The easiest way to get python-colormath is through pip/easy_install:

pip install colormath

Alternatively, grab it from PyPi.

Halp!

If you get stuck, create an issue in the issue tracker and we can figure it out.

Networked, multi-container image crawling with Docker and fig

Like many, I’ve been playing with Docker as it has been developing. I feltreasonably comfortable creating images and running isolated containers, but I had yet to tinker with automatically provisioning and networking multiple containers together.

To remedy this, I spent a weekend writing a simple distributed dockerized image crawler using Twisted, Docker, and fig. The full source (under a BSD license) can be found in the Github repo.

The README covers a lot of this in more detail, but the application is comprised of three container types:

  • An HTTP web API for enqueuing jobs. For the sake of this example, you only run one of these, but it’d be easy to adapt the code to support multiple instances.
  • A worker container that carries out the image crawling. You can run as many as these as you’d like.
  • A Redis container that is used for tracking job state and results.

The worker daemons connect to the HTTP web API container over ZeroMQ, PULLing jobs that the HTTP daemon PUSHes. If this were anything but a toy app, you’d probably want a proper message broker that could offer stronger deliverability and persistence guarantees.

Assorted Observations

  • I may have been doing something wrong, but boot2docker seems to be pretty rough with volume mounting. It looks like it’d be pretty easy to mount a directory within the boot2docker VM, but the host machine->container sharing didn’t work in real-time. I expect this will improve with time, or I could have missed something.
  • fig is a great idea, and is very handy. I’m looking forward to seeing where this concept is taken over the next few years.
  • There is a great python:2.7 image on Docker Hub that serves as a great starting point. They’ve got all sorts of other versions and alternative implementations (PyPy) covered.

python-colormath 2.0 released!

python-colormath was startedback in 2008, when I was an undergraduate at Clemson University (Go Tigers!). While there are a good number of people out there making use of the module effectively, there were a lot of things I wanted to do differently in an eventual 2.0 release. There were some usability issues that arose from my being relatively new to Python at the time.

But all has been made well now. I am happy to announce the immediate release of python-colormath 2.0! A few hilights:

  • Better documentation
  • Python 3.3+ support added. Python 2.6 and lower no longer supported.
  • A complete re-working of RGB and RGB conversions. I’m biased, but I think we’ve now got more correct RGB handling than the vast majority of color math libraries out there, regardless of language.
  • While the color space conversion math remains largely untouched, there is now a dedicated color_conversions submodule that is devoted to the cause. I think this is a good clarity/usability win.
  • Our unit test suite saw a lot of improvement. They are now easier to write, more complete, and more helpful when failures occur.
  • Numpy matrix-driven Delta E functions were incorporated for a nice speed win. In addition to the standard Delta E calls, you can use a vectorized equivalent that is much faster.

See the 2.0 release notes for a more detailed look at the differences.

NOTE: There are backwards incompatible changes in this release. In order to set things right, there were quite a few breakages, but I’ve done the best I can to document these.

Installation

The easiest way to get python-colormath is through pip/easy_install:

pip install colormath

Halp!

If you get stuck, create an issue in the issue tracker and we can figure it out.

Oft-misused adjectives: Bloated and Lightweight

In software development, we like to re-purpose everyday adjectives. We’llcall a project “unstable”, or “mature”. Maybe we give a nod where it’s due and say a piece of software is “elegant”. For the most part, this works pretty well. However, I’m going to take a moment to rant about two that I see constantly misused:

  • Bloated
  • Lightweight

Every time I see either of these used, I ignore what follows. There are cases where these two adjectives make sense, but I argue that these are few and far between. Some of my hangups may be completely irrational, so take this mostly as a rant, because who doesn’t like a good rant?

Let’s started with “bloated”.

Bloated

The Dictionary.com definition:

bloat·ed [bloh-tid] adjective

  1. swollen; puffed up; overlarge.
  2. excessively vain; conceited.
  3. excessively fat; obese.

Applied in the context of software, this is often used to describe a project that is larger than it needs to be. It is almost always used as a derogatory term, used to describe something overly complex. However, in most cases this term is a too vague and non-specific to be of any use. How “large” should the project be? What defines “large”? If it were stripped down, would it still be useful to the same audience? Is it really a problem that you aren’t using all of the bells and whistles in a module?

When you find yourself considering using “bloated”, ask yourself to more specifically define your grievances. If the software/library/module is a sprawling, badly organized mess, you probably want something like “sloppy” or “tangled”. If you are having a hard time wrapping your head around the basic usage of said software, maybe you’re looking for “complex” or “ergonomically challenged”. These point to more specific issues.

Bloated” can often appear when a developer is having a hard time understanding a piece of software. Perhaps the real problem is bad documentation, but the dev may instead just call said software “bloated” and move on. I also see it used as justification to re-invent the wheel or justify NIH Syndrome due to a lack of understanding for the original library.

Now that I’ve ranted for a bit, here’s a summary:

  • Find a better way to describe your objection. Be more specific!
  • Understand that a general-purpose library/module/software product has to cover more than just your usage case. If your usage case is specific and narrow, you’re more likely to toss around the “B word”.
  • It’s OK if you don’t use all of a software product’s features. Disk space is cheap.
  • If a software product is hard to understand, it may not be “bloated”. It may have ergonomic or documentation issues. It could just be messy code. Mention these issues instead of referring to the more vague “bloat”.

There are legitimate usage cases for this term, but there is almost always a better, more specific way to describe why you dislike a piece of software.

Lightweight

A library being described as “bloated” is almost always followed up with a recommendation for a more “lightweight” alternative. The Dictionary.com definition:

light·weight [lahyt-weyt] adjective

  1. light in weight.
  2. being lighter in weight, texture, etc., than another item or object of identical use, quality, or function: a lightweight topcoat; a lightweight alloy for ship construction.

This is probably the more obnoxious of the two terms for me. The intent is to describe a module as being smaller and more simple than an alternative. The amusing thing is that the supposed more “lightweight” alternative is of similar size and complexity to the “bloated” bit. “lightweight” is used to paint a module in a positive light, opposite of “bloated“‘s negative connotation.

As is the case with “bloated”, lightweight is a cop out for justifying why something is better. When you find yourself considering the use of “lightweight”:

  • Re-visit and re-define your objection to the first library you were considering. Is it sloppy? Does the documentation suck? Does the API have ergonomic issues? Does it use too much of your system’s resources? If so, pick a more descriptive adjective.
  • Once you have a concise reason for not liking your first candidate library, see if your alternative does things better. If so, specifically mention what it does better. Does it have better documentation? Alternative is “better documented”. Is the API easier to use? Alternative is subjectively “easier to use”. Is the code better organized and of higher quality? Alternative is “more elegantly designed”.

Lightweight” doesn’t tell us much. Amusingly, it is often used by developers who were looking to re-invent a more “bloated” piece of software by stripping a ton of features out, pooping out some documentation, then calling it a day. Said alternative often lacks the community to see much maintenance, and it often lacks widely used features that its predecessor had.

There are legitimate usage cases for this term, but there is almost always a better, more specific way to describe why an alternative to something is better.

Summary

In closing, it’s important to remember that open source software is created and developed by other real humans. They tend to take some amount of pride in their work. It’s OK to criticize software, it often motivates positive change. However, it’s important to be specific and fair with criticism. Don’t cop out when describing what makes you dislike a certain library/module/software product. Get specific, be fair, and maybe everyone will benefit.