nginx AWS ELB name resolution with resolvers

Thursday, November 10 2011

If you are running nginx as a proxy in front of An Amazon Web Services Elastic Load Balancer (ELB), it is not safe to merely define an upstream using the hostname of ELB and call it a day. By default, nginx will only do name resolution at startup time, caching the resolved IP address infinitely.

ELB instances scale up and down based on your traffic levels, often changing IP addresses in the process. It seems to be that increased traffic leads to Amazon spawning a new, beefier ELB instance, then changing the DNS record to point at the new instance. It’ll keep the old ELB instance around for a little while to give you time to resolve the new one, but the old instance (using the old IP) will be retired after a short period of time. We need nginx to be able to periodically re-resolve the load balancer’s hostname so service interruptions aren’t encountered due to the IP address change.

The fix

Fortuantely, this one is really simple to remedy. You need only use the resolver config directive in your nginx config. By specifying a DNS server with the resolver directive from within nginx, you signify that it should check with said server every five minutes (by default) to see if the upstream ELB has changed IPs. This is done in a non-blocking manner, and should pose no real threat to your server’s throughput.

The other critical piece is that you must add a $request_uri to the end of whatever proxy_pass value you’ve specified. DNS caching will remain without this, meaning you are no better off. See the example below.

Example

http {
   [...]

   # Causes DNS resolution of upstreams every 5 minutes.
   resolver 172.16.0.23;

   [...]

   server {
      [...]

      proxy_pass http://somewhere.com$request_uri

      [...]
   }
}

The resolver directive can be used in http, server, and location sections, so you can get as specific or as broad as you’d like.

The future fix

A later version of nginx will honor DNS TTLs, so look forward to that. I’ll try to remember to update this article when this lands.


A simple explanation of Linux load averages

Tuesday, July 26 2011

I’ve noticed that some people coming from Windows or other backgrounds often mis-interpret Linux load averages, or don’t understand them at all. Check out this great rough overview post, Understanding Linux CPU Load. While there is, of course, more to it than the article goes over, this will ...

read more

Rabbits for the celery

Monday, July 18 2011

I run an Arch Linux desktop as my primary development workstation. We use celery pretty heavily on some of our Django projects, and I was working to get my local environment at least somewhat closer to our production setup, only to find there isn’t a non-AUR package for ...

read more

Amazon EC2 and long restart delays

Wednesday, March 23 2011

For the benefit of others either considering Amazon’s EC2, or who are already there, I thought I’d point something out. I am not sure if this is an Ubuntu EC2 AMI issue, an EC2 issue, an EBS issue, or some combination of all of these, but we are ...

read more

Toshiba Tecra A4 on Ubuntu Ibex Alpha

Sunday, September 28 2008

I haven’t had enough time to really get some wear and tear testing in, but it looks like as of today, the latest Ubuntu Ibex Alpha release runs almost flawlessly on the Toshiba Tecra A4 (PTA40E). Wireless and the Wired NIC work without any kernel boot options, and performance ...

read more