Pathwright launches, powered by Snakes and Jazz

/

After over two years of development, Pathwright has emerged frombeta, and is open to the masses. For those with knowledge to share, we hope this web application will give you an easy-to-use, easy-on-the-eyes way to teach others. We provide a structured, social way for your students to learn.

See the Pathwright landing page for more details, as I’m going to leave marketing mode and get into the more philosophical and technical topics that most of you will probably be more interested in.

What’s special about Pathwright?

There is no longer a shortage of online learning software to choose from. Players ranging from “free” to expensive exist, with wildly varying approaches and feature lists. We chose a very specific set of core principles to develop against:

  • Online learning should be easy for both teachers and students.
  • The software should be visually appealing and polished.
  • Do less than the competition, but do it better.
  • Prices are posted upfront, no sales calls or interaction with us is required to get started. We’re eager to talk to anyone who’d like to get in touch, but some of our competitors strong arm you into sales calls before you can even see the software, which we are loath to do.
  • It should be easy and affordable for organizations of all sizes to sign up, try the software free of charge, then elect to start selling courses as soon as they’d like.

We are a hosted solution that handles all of the unpleasant stuff for you (servers, billing, scaling, continuous software upgrades, support) for you. Our hope is that our customers can spend their time creating content and interacting with their students, instead of dealing with infrastructure or dialing IT.

A peek under the hood

Pathwright is powered by the Python and Django. The two were chosen for their ease of use, and being good enough to accomodate our fast-paced development process. Almost any modern, widely used framework would have done the trick, our team was just very comfortable with this particular stack.

All of our servers are on Amazon Web Services. We’ve found the pricing to be great, and the ability to effectively “outsource” portions of our stack has freed up a lot of development time. We run no media servers (S3 + CloudFront), no mail servers (SES), no self-hosted load balancers (ELB), and have drastically simplified a number of other things with AWS’s services.

A few really critical parts of our stack:

Some crucial Python packages:

Tell us what you think!

We’ve worked long and hard on Pathwright, bootstrapping it from nothing. At this point, we’re what you could say “too close” to the project to get all the feedback we need from our team. Please let us know what you think in the comments or via email.

More efficient market web APIs for EVE Online

/

There are a handful of market data sites (EVE-Central, Eve Marketeers,Eve Marketdata) out there now, each with their own developer APIs. All but EVE-Central are relatively new sites, and most seem to suffer from the occasional, or permanent, sluggishness. It doesn’t appear to be for lack of hardware, I know several of these sites are running on some pretty good metal. There appears to be a combination of duties (accepting incoming data, serving the website, serving the developer APIs) that really slows the sites down, along with their respective developer-exposed web APIs. Let’s muse on a cheap, sturdy way to architect around this somewhat.

For the sake of discussion, I’ll just concern myself with the web-exposed developer APIs in this post. These are open-to-the-public services that let other applications grab the market site’s data at will. This is often done with high frequency, and high volume. From what I can tell, some of these sites serve their APIs from their primary app server, which is where the rest of the general web traffic goes through. In some cases, the same process that serves the site and the developer API may also accept incoming market data.

Wouldn’t it be nice if we could shove that API traffic off somewhere else? That would mean sluggishness on the website wouldn’t mean sluggishness in the API, and vice-versa. For market sites with tons of API users, this could free up resources for other things. Let’s see where we can go with this…

The APIs are relatively simple

The good news here is that the APIs on all three sites are relatively simple. To get the price for an item in a certain region, you just pass in the region and the item ID. A “recently updated” query may not require any parameters at all. For the most part, our input is going to be small, and dictated by EVE’s identifiers for various things (items, regions, characters).

Here’s what we know

  • We need a relatively small core set of capabilities for developers to find our service useful. Price by region being the biggest.
  • We don’t want to have to serve the API requests ourselves, since that is boring.
  • We’re probably already doing some processing and aggregation of incoming data.

Here’s what we can do

  • Create a daemon whose only purpose is to accept incoming market data from the uploaders. This gets queued and pumped into another process that does validation, statistical aggregation, and saves such things to a DB.
  • Based on what new data came in, the process uploads JSON documents to Amazon S3, to paths that mimic the current developer APIs (IE: /region/12345/item/123556/).
  • External developers can then form the URLs much like they already do, hitting S3 instead of your servers.
  • Your API is now infinitely scalable, and pretty much impossible to bring down under load.
  • Access can be left public, or restricted by S3 keys or signed URLs.
  • If you are wanting to be self-sufficient, you could even make developers pay for the bandwidth they use with Amazon DevPay. Given that a few other sites offer free APIs, this may no go over well, though.

Caveats

  • You will need to keep track of how long it’s been since various S3 keys were updated. It may not make sense to always upload the new data as it comes in for super active items. You can afford to wait a minute or two between updates for very active item+region combos.
  • Data transfer into S3 is free, but the bandwidth on your market data upload accepter machine may not be.
  • You will be uploading to S3 pretty constantly.

Reference Implementation?

I briefly debated whether to attempt to write a reference implementation for this, but it looks like the state of market uploaders is pretty bad right now. There’s Contribtastic, but it appears to be very much centered on EVE-Central. There’s also a Unified Uploader, but it’s closed-source and Windows-only.

We’ll re-visit this idea if the uploader scene changes.

boto 2.2.2 has arrived

/

boto 2.2.2 was released this morning, comprised mostly of bug fixes.Of particular interest is the DynamoDB stabilization that has happened over the last few weeks, although a number of other minor bugs were fixed with the other services. For anyone currently running on older versions, we highly recommend upgrading to boto 2.2.2.

If you run into an problems, have questions, or would like to suggest something, hit #boto on FreeNode IRC, the issue tracker, or the mailing list.