Simple outbound request rate limiting with App Engine

I've been doing a lot of playing with Google App Engine (GAE) of late, since it is a cheap/free way for me to quickly toss ideas against the wall and see what sticks. One of the tinker projects I've been working on is a sub-Reddit stat tracker (warning: hastily put together and un-finished) that records and ranks technical sub-Reddit activity over time.

Retrieving the data I needed required polling the Reddit API. There is an existing high-quality Python API client (PRAW), but I ran into a GAE + requests + HTTPS issue that prevented me from using it (PRAW uses requests). Said issue will be fixed in requests 2.10.0, but there's been no indication that requests 2.10.0 is arriving anytime soon. This was significant in that PRAW handles rate limiting and oauth authentication for you. Rather than wait for requests 2.10.0 or forking PRAW to use GAE's urlfetch service (which supports HTTPS), I decided to hit Reddit's public API directly without authenticating.

Warning: The days of unauthenticated Reddit API access may be coming to an end. I don't recommend unauthenticated API access for anyone doing anything more simple than a tinker project like mine.

Early goings

My initial draft used App Engine cron and task queues to schedule and parallelize the work. Once an hour, cron triggered a background task that would create sub-tasks for each sub-Reddit I'm tracking. These sub-tasks would hit the Reddit API once or twice, muddle through the return value, and toss some values into a Custom Metric on Google Stackdriver.

Since the first proof-of-concept wasn't rate limited, I ran into HTTP 429 (Too Many Requests) as I tracked increasingly more sub-Reddits.

View fullsize

App Engine Queue definitions to the rescue

I only needed to do a full scan of all of my tracked sub-Reddits once per hour. I had to make sure that I'm not doing more than 30 API calls per minute. I wanted to try to spread the requests out evenly, rather than exhaust my quota at the beginning of each minute. I also wanted to do this with minimal complexity.

Fortunately, App Engine task queues can be configured with a queue.yaml file in your project root. There are two directives in here that are particularly interesting:

• rate - How often jobs are popped from the queue and distributed to your workers. The number of jobs that are popped at a time is determined by max_concurrent_requests. For example, a value of 30/m will mean the queue is popped at most 30 times per minute.
• max_concurrent_requests - The max number of concurrently executing jobs.

Since the unauthenticated Reddit API rate limit is 30 requests per minute, I was able to enforce this at the queue level by using a rate of 30/m and a max_concurrent_requests of 1. Here is my full queue.yaml file. The end result:

• Tasks are popped from the queue up to 30 times a minute (rate = 30/m).
• We only pop one task at a time (max_concurrent_requests = 1).
• We won't pop a new task until the currently running one is ACK'd.
• As long as everything works as described in the docs, we stay under the rate limit at all times.

As a result, we went from rate limiting errors all over the place to:

View fullsize

So you can rate limit. What's the big deal?

Rate limiting is not an especially difficult thing to implement, but I thought it was interesting to see how easy App Engine made this. My code doesn't know or care that it's being rate limited, which is nice. The most beautiful lines of code are the ones you don't have to write at all!

In the future, I'll want to either move over to PRAW when requests 2.10.0 lands or implement the bare minimum for oauth authentication with App Engine's urlfetch service. At that point, I'll be able to twiddle my rate and concurrency values to get some more throughput.

Nothing earth-shattering here, but I thought I'd share!

Oft-misused adjectives: Bloated and Lightweight

In software development, we like to re-purpose everyday adjectives. We’llcall a project “unstable”, or “mature”. Maybe we give a nod where it’s due and say a piece of software is “elegant”. For the most part, this works pretty well. However, I’m going to take a moment to rant about two that I see constantly misused:

• Bloated
• Lightweight

Every time I see either of these used, I ignore what follows. There are cases where these two adjectives make sense, but I argue that these are few and far between. Some of my hangups may be completely irrational, so take this mostly as a rant, because who doesn’t like a good rant?

Let’s started with “bloated”.

Bloated

The Dictionary.com definition:

bloat·ed [bloh-tid] adjective

1. swollen; puffed up; overlarge.
2. excessively vain; conceited.
3. excessively fat; obese.

Applied in the context of software, this is often used to describe a project that is larger than it needs to be. It is almost always used as a derogatory term, used to describe something overly complex. However, in most cases this term is a too vague and non-specific to be of any use. How “large” should the project be? What defines “large”? If it were stripped down, would it still be useful to the same audience? Is it really a problem that you aren’t using all of the bells and whistles in a module?

When you find yourself considering using “bloated”, ask yourself to more specifically define your grievances. If the software/library/module is a sprawling, badly organized mess, you probably want something like “sloppy” or “tangled”. If you are having a hard time wrapping your head around the basic usage of said software, maybe you’re looking for “complex” or “ergonomically challenged”. These point to more specific issues.

“Bloated” can often appear when a developer is having a hard time understanding a piece of software. Perhaps the real problem is bad documentation, but the dev may instead just call said software “bloated” and move on. I also see it used as justification to re-invent the wheel or justify NIH Syndrome due to a lack of understanding for the original library.

Now that I’ve ranted for a bit, here’s a summary:

• Find a better way to describe your objection. Be more specific!
• Understand that a general-purpose library/module/software product has to cover more than just your usage case. If your usage case is specific and narrow, you’re more likely to toss around the “B word”.
• It’s OK if you don’t use all of a software product’s features. Disk space is cheap.
• If a software product is hard to understand, it may not be “bloated”. It may have ergonomic or documentation issues. It could just be messy code. Mention these issues instead of referring to the more vague “bloat”.

There are legitimate usage cases for this term, but there is almost always a better, more specific way to describe why you dislike a piece of software.

Lightweight

A library being described as “bloated” is almost always followed up with a recommendation for a more “lightweight” alternative. The Dictionary.com definition:

light·weight [lahyt-weyt] adjective

1. light in weight.
2. being lighter in weight, texture, etc., than another item or object of identical use, quality, or function: a lightweight topcoat; a lightweight alloy for ship construction.

This is probably the more obnoxious of the two terms for me. The intent is to describe a module as being smaller and more simple than an alternative. The amusing thing is that the supposed more “lightweight” alternative is of similar size and complexity to the “bloated” bit. “lightweight” is used to paint a module in a positive light, opposite of “bloated“‘s negative connotation.

As is the case with “bloated”, lightweight is a cop out for justifying why something is better. When you find yourself considering the use of “lightweight”:

• Re-visit and re-define your objection to the first library you were considering. Is it sloppy? Does the documentation suck? Does the API have ergonomic issues? Does it use too much of your system’s resources? If so, pick a more descriptive adjective.
• Once you have a concise reason for not liking your first candidate library, see if your alternative does things better. If so, specifically mention what it does better. Does it have better documentation? Alternative is “better documented”. Is the API easier to use? Alternative is subjectively “easier to use”. Is the code better organized and of higher quality? Alternative is “more elegantly designed”.

“Lightweight” doesn’t tell us much. Amusingly, it is often used by developers who were looking to re-invent a more “bloated” piece of software by stripping a ton of features out, pooping out some documentation, then calling it a day. Said alternative often lacks the community to see much maintenance, and it often lacks widely used features that its predecessor had.

There are legitimate usage cases for this term, but there is almost always a better, more specific way to describe why an alternative to something is better.

Summary

In closing, it’s important to remember that open source software is created and developed by other real humans. They tend to take some amount of pride in their work. It’s OK to criticize software, it often motivates positive change. However, it’s important to be specific and fair with criticism. Don’t cop out when describing what makes you dislike a certain library/module/software product. Get specific, be fair, and maybe everyone will benefit.

When (and when not) to use EC2

There is a lot of advice on the internet regarding the suitability of EC2.One can easily find all kinds of benchmark and price comparisons, long rants on Cloud vs Bare metal, and any number of other reasons for or against using EC2. I feel that a lot of these articles miss the mark entirely, and figured I’d toss up my own attempt at guidelines for choosing (or avoiding) EC2.

This article will attempt to provide a simple list of some (but not all) cases where EC2 is and isn’t a good fit. I will undoubtedly miss some, and there are exceptions to every case, but take these as a set of general guidelines. Also, for the sake of brevity, I assume that you have a basic grasp of what EC2 is. If not, check out the EC2 page before continuing.

EC2’s killer feature

Before we get to the fun stuff, I want to take a moment to highlight EC2’s killer feature: Elasticity. Here are some important bits extracted from the AWS EC2 page:

Amazon EC2’s simple web service interface allows you to obtain and
configure capacity with minimal friction.
[...]
Amazon EC2 reduces the time required to obtain and boot new server instances
to minutes, allowing you to quickly scale capacity, both up and down, as
your computing requirements change. Amazon EC2 changes the economics of
cloud computing servers by allowing you to pay only for capacity that
you actually use.

EC2’s primary goal is to make it easy to provision and destroy VMs as needed. They’ve got an advanced set of HTTP APIs for managing your fleet, a very good set of imaging/provisioning utilities, a wide variety of instance sizes to choose from, and they even excellent auto-scaling capabilities built into ELB. The hourly billing allows you to handle bursts of traffic without a long-term commitment, and you can also fire up experimental/tinker instances without much expense.

There are other services that also offer some of these things individually, but EC2 is at the front of the pack when it comes to provisioning instances and scaling up/down.

It’s not all sunshine and roses

“That sounds great! Why would I ever want to use anything else?”

• The performance per dollar is atrocious. Be prepared to shell out for a larger instance to get consistent performance. This is more of a barrier for projects with smaller budgets. In the case of a very well-funded project, infrastructure spend is (probably) much less of a concern.
• Amazon Web Services in general features a pretty steep learning curve before one can make informed infrastructure decisions. You’ll need to learn about security groups, EBS, EC2’s SSH key management system, how snapshots/AMIs work, performance characteristics, common sources of failures, etc.
• The lower end instances (below m1.large) are embarrassingly underpowered and erratic. For services or applications with low resource requirements, this may or may not be an issue. Make sure you benchmark and test before committing to a reservation!
• EBS has had serious reliability issues and is very inconsistent performance-wise without Provisioned IOPS or RAID. The majority of the larger EC2 outages have been due to EBS issues.
• The customer service is abysmally bad. Unless you pay for a Support Plan, your only option is to post to a public forum and hope that someone from AWS replies. I understand that AWS operates at a huge scale, but I expect better from a company full of brilliant people like Amazon. Telling paying customers (without a support plan) to post in the forums for help is inexcusable. A support plan should be for above-and-beyond service.

“That was discouraging. Let’s get back to the good stuff.”

EC2 is a potential good fit for your application when…

• You want/need to be able to scale up and down to meet handle traffic. This can be done automatically via ELB (after some setup work), via the EC2 HTTP API, or by the more traditional web based management console.
• Your application has to have consistently fast, low latency access to at least one other AWS service.
• Your application pumps enough traffic into/out of an AWS service and you don’t want to pay the higher external traffic tolls.
• You plan on using a number of managed services that typically pair with EC2. For example, ElastiCache, RDS, CloudSearch, and etc. These can be a life-saver if your team doesn’t have the ops/administrative skill to manage the equivalent EC2 instances yourselves. Though, they’re not always a good value for teams with some ops chops in-house.
• You have enough budgeted to pay for the instance sizes that are appropriate for your performance needs. For many/most, this probably involves buying at least some instance reservations.
• EC2 would allow you to streamline your operations processes enough to allow you to run with a smaller ops team. Spending more for infrastructure convenience is a lot less expensive than hiring another employee.

EC2 is probably not a good fit for your application when…

• You don’t need to scale up/down much. This is one of the biggest strengths of EC2. If you aren’t using it, you can go with something much cheaper/faster/more reliable/more consistent.
• You can’t afford to pay for redundancy, but your application has high stability/availability requirements. EC2 instances fail, EBS volumes fail or slow down randomly, entire availability zones fail. The systems supporting EC2 are incredibly complex and operate at a massive scale. Stuff happens. If you can’t afford some measure of cross-availability-zone redundancy (at minimum) and your application has stability requirements, EC2 is probably not for you.
• The need to purchase 1-3 year instance reservations to get decent hourly rates on instances bothers you. The On Demand rates can be incredibly expensive if you run instances with them for extended periods of times. A reservation requires the upfront payment of a large chunk of the next 1-3 year’s costs. This can be a problem for those without enough liquid capital. It also means that adding additional capacity can be a larger business decision.
• Your application has components that require very consistent performance, but you can’t afford one of the very large instance sizes that have few to no neighbors on the host machine. Very high activity DB servers often fall into this category.
• You are wanting to eventually incorporate bare metal servers into your fleet. EC2 currently only offers virtualized instances, though you can rent out an entire host machine for yourself. Alternatively, you can use Direct Connect to bridge to certain data centers, but you’ll want to make sure this fits in your architecture and budget.

Advice on deciding for or against EC2

Read through the cases outlined above again and keep a tally of how many apply for your project. If you find that a good number of the cases under “good fit” apply, EC2 is probably worth further consideration. If you find that more than 1-2 of the scenarios under “not a good fit” are true, you should probably tread more carefully.

Exceptions, omissions, and etc

“But you forgot X case, or your advice isn’t appropriate for my usage case!”

This article attempts to be decent one-size-fits-most guide, but there are exceptions and other scenarios that I didn’t cover. If you know some of these, you probably don’t need this guide. If I missed something super obvious, leave a comment and I’ll update the article.

I don’t cover comparing prices against other providers in this article, but I’d like to point out that some of EC2’s price is there because of convenience and flexibility. You are paying extra for the ability to interact with stuff like ELB, CloudWatch, and other services. Perhaps I’ll follow up with another article expanding on this more if there’s interest…

In the end, it will come down to you identifying your requirements and finding the best fit.

On a related note, if you need help or advice with infrastructure planning, get in touch with me via the Contact link on the top navbar.