More efficient market web APIs for EVE Online

There are a handful of market data sites (EVE-Central, Eve Marketeers,Eve Marketdata) out there now, each with their own developer APIs. All but EVE-Central are relatively new sites, and most seem to suffer from the occasional, or permanent, sluggishness. It doesn’t appear to be for lack of hardware, I know several of these sites are running on some pretty good metal. There appears to be a combination of duties (accepting incoming data, serving the website, serving the developer APIs) that really slows the sites down, along with their respective developer-exposed web APIs. Let’s muse on a cheap, sturdy way to architect around this somewhat.

For the sake of discussion, I’ll just concern myself with the web-exposed developer APIs in this post. These are open-to-the-public services that let other applications grab the market site’s data at will. This is often done with high frequency, and high volume. From what I can tell, some of these sites serve their APIs from their primary app server, which is where the rest of the general web traffic goes through. In some cases, the same process that serves the site and the developer API may also accept incoming market data.

Wouldn’t it be nice if we could shove that API traffic off somewhere else? That would mean sluggishness on the website wouldn’t mean sluggishness in the API, and vice-versa. For market sites with tons of API users, this could free up resources for other things. Let’s see where we can go with this…

The APIs are relatively simple

The good news here is that the APIs on all three sites are relatively simple. To get the price for an item in a certain region, you just pass in the region and the item ID. A “recently updated” query may not require any parameters at all. For the most part, our input is going to be small, and dictated by EVE’s identifiers for various things (items, regions, characters).

Here’s what we know

  • We need a relatively small core set of capabilities for developers to find our service useful. Price by region being the biggest.
  • We don’t want to have to serve the API requests ourselves, since that is boring.
  • We’re probably already doing some processing and aggregation of incoming data.

Here’s what we can do

  • Create a daemon whose only purpose is to accept incoming market data from the uploaders. This gets queued and pumped into another process that does validation, statistical aggregation, and saves such things to a DB.
  • Based on what new data came in, the process uploads JSON documents to Amazon S3, to paths that mimic the current developer APIs (IE: /region/12345/item/123556/).
  • External developers can then form the URLs much like they already do, hitting S3 instead of your servers.
  • Your API is now infinitely scalable, and pretty much impossible to bring down under load.
  • Access can be left public, or restricted by S3 keys or signed URLs.
  • If you are wanting to be self-sufficient, you could even make developers pay for the bandwidth they use with Amazon DevPay. Given that a few other sites offer free APIs, this may no go over well, though.

Caveats

  • You will need to keep track of how long it’s been since various S3 keys were updated. It may not make sense to always upload the new data as it comes in for super active items. You can afford to wait a minute or two between updates for very active item+region combos.
  • Data transfer into S3 is free, but the bandwidth on your market data upload accepter machine may not be.
  • You will be uploading to S3 pretty constantly.

Reference Implementation?

I briefly debated whether to attempt to write a reference implementation for this, but it looks like the state of market uploaders is pretty bad right now. There’s Contribtastic, but it appears to be very much centered on EVE-Central. There’s also a Unified Uploader, but it’s closed-source and Windows-only.

We’ll re-visit this idea if the uploader scene changes.