I’ve been finding myself frequently enumerating all public and elastic IPs on various AWS accounts fairly often in order to audit the set of services exposed to the Internet. This is a tedious process through the AWS console UI, and doing it through the AWS CLI interface involves an impossible to remember incantation.

For the curious, that incantation is:

1
2

aws ec2 describe-instances --query "Reservations[*].Instances[*].PublicIpAddress" --output=text > /tmp/publicips.txt
nmap -il /tmp/publicips.txt -sC -sV

I finally decided to stop spending 30 minutes on Google every time I needed to do this and wrote a quick utility I could actually remember how to use.

1
2

pip install aws-nmap
aws-nmap -sC -sV

Any arguments passed to aws-nmap will simply be forwarded to nmap along with any public IPs in use on your AWS account instances. Great! Now stop putting Redis/MongoDB on public IPs bound to 0.0.0.0!

Services like SendGrid, Mailgun, or Mandrill are a great way to handle outbound email from a web application, and each will certainly allow you to transmit email directly to their mail servers. However, any kind of outage in between your app server and your mail transfer service could easily result in mail getting routed to /dev/null with no easy way to identify and resend the lost mail. This is particularly problematic for emails sent from a worker process since they will often keep right on churning away even if other things are down. For some applications, it may be a perfectly acceptable risk to drop a few emails, as hopefully such service interuptions are rare, but losing emails would be a major problem for the project I’m working on and I suspect many others.

To mitigate this, I’m configuring Postfix on each app server to relay out to Mandrill. You can do the same for SendGrid or Mailgun, and while this post is going to be mostly Mandrill-specific, since that’s what I’m using, the equivalent for SendGrid/Mailgun ought to be pretty close as all three services use SASL authentication for relay configurations.

This is a decent solution because Postfix, being a full mail transfer agent in its own right, has useful things like automatic retry logic built-in. Rather than reimplement all that in my app server, no doubt in a much less reliable way, I’m going to set up Postfix running on each app server’s localhost. By doing it that way, I can leverage the fact that it’s hard to screw up communication over loopback and just outsource my email reliability problem to software that’s been around for just shy of two decades. Postfix uses very little in the way of resources, so it’s not a problem to run it on each app server.

The Opscode Postfix cookbook is the main tool we need. I’m using OpsWorks, but this cookbook isn’t specific to OpsWorks and should work in any Chef setup.

The custom Chef JSON for a Postfix null client relay looks like this:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26

{
  "postfix": {
    "mail_type": "master",
    "main": {
      "relayhost": "smtp.mandrillapp.com",
      "myhostname": "myapp.com",
      "mydomain": "myapp.com",
      "mynetworks": [
        "127.0.0.0/8",
        "[::1]/128",
        "[fe80::]/10"
      ],
      "mydestination": "",
      "inet_interfaces": "loopback-only",
      "smtp_use_tls": "yes",
      "smtp_tls_CAfile": "/etc/ssl/certs/ca-certificates.crt",
      "smtpd_tls_CAfile": "/etc/ssl/certs/ca-certificates.crt",
      "smtp_tls_security_level": "encrypt",
      "smtp_sasl_auth_enable": "yes"
    },
    "sasl": {
      "smtp_sasl_user_name": "[redacted]",
      "smtp_sasl_passwd": "[redacted]"
    }
  }
}

The main attributes in the JSON config that we care about are the values for smtp_tls_security_level and smtp_sasl_auth_enable. The yes value for smtp_sasl_auth_enable switches on SASL authentication, which is how Mandrill knows which account is sending mail. Setting the value to encrypt for smtp_tls_security_level forces TLS for everything. We ought to just care about encrypting outbound mail since this is a null client, but, especially in the world in which BCP 188 was necessary…

Since we’re using SASL for authentication, postfix::sasl_auth will be the recipe we want to use. For OpsWorks, this can go into your ‘Setup’ lifecycle event.

I’m using Ubuntu on EC2. The Postfix cookbook doesn’t do a great job of locating the CA certificate bundle for that platform, so you may need to specify the correct location for that or alternatively download a bundle and place it the fallback location of /etc/postfix/cacert.pem. You can test that it’s working via telnet.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18

$ telnet localhost 25
Trying 127.0.0.1...
Connected to myhostname.localdomain.
Escape character is '^]'.
220 myapp.com ESMTP Postfix
EHLO test.com
250-myapp.com
250-PIPELINING
250-SIZE 10240000
250-VRFY
250-ETRN
250-STARTTLS
250-ENHANCEDSTATUSCODES
250-8BITMIME
250 DSN
STARTTLS
220 2.0.0 Ready to start TLS
QUIT

If you send STARTTLS and in response get a message like 454 4.7.0 TLS not available due to local problem, that usually means Postfix couldn’t find your CA bundle.

Finally, verify that relaying works with sendmail.

1
2
3
4
5

$ sendmail your.email@myapp.com
From: admin@myapp.com
Subject: Testing from Postfix
This is a test email
.

Over the years, I’ve frequently found myself implementing systems that follow Postel’s Law on a fairly regular basis. An extremely common scenario that comes up when building such liberal systems and parsers is a short-circuited sequence of searches.

Often, these search sequences will involve long if chains that repeatedly check for nil. However, this is both inefficient and inelegant. I’ve just (re-)discovered that Ruby has a much better way of accomodating this style of branching logic.

Take, for example, a particularly liberal feed parser that needs to return the title of a feed:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37

def feed_node
  @feed_node ||= (
    root.xpath(
      '/atom:feed',
      'atom' => 'http://www.w3.org/2005/Atom'
    ).first or
    root.xpath('/feed').first or
    root.xpath(
      '/rdf:RDF/rdf:channel',
      'rdf' => 'http://www.w3.org/1999/02/22-rdf-syntax-ns#'
    ).first or
    root.xpath('/rss/channel').first
  )
end

def title
  @title ||= (
    begin
      node = feed_node.xpath('title').first
      node.text if node
    end or
    begin
      node = feed_node.xpath(
        'dc:title',
        'dc' => 'http://purl.org/dc/elements/1.1/'
      ).first
      node.text if node
    end or
    begin
      node = feed_node.xpath(
        'rdf:title',
        'rdf' => 'http://www.w3.org/1999/02/22-rdf-syntax-ns#'
      ).first
      node.text if node
    end
  )
end

Admittedly, the example above would allow for completely mental mixes of RSS, Atom, and ancient RDF, and it’s perhaps a bit contrived, but hopefully it illustrates how much cleaner you can make this kind of logic with some well-placed control-flow or’s. Because both or and and have extremely low operator precedence, there’s little risk of accidentally doing something unintended, and the use of begin; end allows for longer blocks of short-circuited logic. Additionally, instead of begin; end, you can also use method calls when chunks of logic start to get too long or unwieldy.

Of course, neither or nor and are anything new, but I rarely see them used across multiple lines, and it turns out that they’re really useful in longer expressions.

When you’re setting up dependencies in your Gemfile, you generally want to give gem authors the benefit of the doubt. Assume that they will adhere to semantic versioning unless you have explicit knowledge that this is not the case for a particular gem.

Many developers make a habit of specifying their dependencies far too pessimistically, which tends to result in dependency conflicts, especially in larger projects. Instead of "~> x.y.z", consider using "~> x.y", ">= x.y.z". This compound requirement should be your instinctive default way of specifying a version constraint.