The following is part of a series of guest posts from Windows IT Pro that takes an in-depth look at SMTP. So far, we’ve learned about the history of Simple Mail Transfer Protocol (SMTP) and examined various definitions for when an email is considered “delivered.” In this final post from Windows IT Pro, they examine 4 key factors that affect email deliverability.
Many factors can affect deliverability, including having a good sending reputation, properly authenticating your mail through SPF and DKIM, and having strong permission practices. But four factors really stand out:
- The health of the messaging infrastructure
- What’s in the message
- Who’s sending it
- The receiving system’s availability
- Message Infrastructure Health
Network and infrastructure health will obviously have a huge impact on whether messages can be delivered. SMTP assumes the presence of reliable network connectivity and name resolution through the Domain Name Service (DNS). Anything that interferes with these functions can prevent messages from being delivered.
Most administrators are aware of this and take measures to protect their inbound traffic; for example, it’s common to use multiple Mail Exchanger (MX) DNS records to provide redundant routes for inbound mail. However, as a sender, you can’t control the recipients’ servers or DNS configuration, and if either the sending or receiving system has flaky connectivity or DNS problems, it will be difficult to deliver mail in a timely, reliable way.
- Message Content
Message content is a key influence on how deliverable a message is, for a variety of reasons. Recognizing a message as spam or malware by examining its content (including attachments, URLs, message text, and headers) has proven to be a pretty good way to identify and block unwanted content; however, legitimate email about dating sites, mortgages, and various kinds of pharmaceuticals can end up being caught in filters.
The first few generations of message filters were fairly limited; they could only match specific patterns and content items. One of the key improvements in filtering technology was the introduction of collaborative filters, where multiple sites and services can contribute spam reports to a centralized service. The problem with collaborative filtering is that once one filter somewhere decides that a message is spam, other sites that consume the same filtering data will block it, no questions asked.
- Sender History and Filtering
Collaborative filtering was further enhanced by the introduction of reputation-based filtering. Although the exact way that a sender’s reputation is calculated varies from filter to filter, the idea is the same: reputation systems combine information about the source of the message (including the sender’s IP address and purported domain), the behavior of the sender in the past (including the number of messages sent per unit time and whether those messages were considered suspicious), the content of the message, and even feedback from the recipient when they press the “this is spam” button.
For example, a legitimate business that normally sends out 1,000 messages per day might be flagged by a reputation filter if the business suddenly starts sending out 10,000 messages per day. A business that begins sending out messages with questionable content might also trigger reputation filtering. These filters are effective, but they have problems.
For one, if your messages begin triggering the filter, it can be difficult to find out why because ISPs often protect the algorithms they use to score email so spammers aren’t able to game the system. For another, reputation systems that use the source IP address of the sender as an input can cause your mail to be filtered after your IP address changes, and the new address comes from a range that has a bad reputation. So, through no fault of your own, you might find your messages being filtered after you change IPs or move your mail servers to a different network.
Of course, when you run your own SMTP servers you’re taking on responsibility for a host of deliverability issues. For example, if your server is set to be an open relay, you’re likely to very quickly find your server blacklisted once that configuration mistake is discovered, either by spammers or by the automated scanning tools that all the major RBL vendors use. When you put an SMTP server onto the Internet, you own the responsibility to monitor and maintain it.
- Receiving System’s Availability
Deliverability also depends on the receiving system’s availability. Because SMTP is a store-and-forward protocol, a sending server will typically queue messages for a period of time when the receiving server is unavailable. In multi-server environ- ments, the message might be delivered to a perimeter server, but if the recipient’s mailbox is unavailable (perhaps because the target mailbox database is offline, or an inter-site link is unavailable), the perimeter server can hold the message and deliver it later.
Improving Message Deliverability
Improving the deliverability of your outbound messages helps to make your business communications more efficient and cost-effective. The most important single step you can take is to have good instrumentation and measurement processes in place so that you have reliable data about the number of messages that aren’t delivered. Tracking these metrics over time will help you quickly identify trends and patterns that can give you early warning of problems before they get out of hand.
Analyzing this data will tell you a great deal about the source of any email deliverability problems you’re having. For example, a server that’s configured to use the wrong DNS server, or that contains incorrect reverse DNS, SPF, or DKIM records, might be slow, or unable, to deliver messages to certain destinations, and that change in delivery patterns will show up clearly when you look at delivery times and queue lengths on the affected server—but only if you do look.
Also, look at server logs more than queue lengths. Many ISPs will return deliverability specific errors in their return codes and these typically can be seen in the SMTP server (MTA) logs. Some of these “errors” indicate that a sender should fill out lengthy cumbersome forms at the ISP to establish bulk sender status or verify their legitimacy.
Your ongoing operational monitoring should give you early warning of network or infrastructure problems that will affect your deliverability for outbound messages. If you don’t have the resources to dedicate to this sort of monitoring, SendGrid has robust tracking and analytics as well as a team of email experts to help you identify and rectify these delivery issues.
Many thanks to Windows IT Pro for this in-depth look at the history of SMTP and email deliverability. If you’d like to learn more about email delivery best practices and staying compliant, you can read our Deliverability Guide and our guide on The ABCs of ISPs.