Best Practices with Sender Policy Framework (SPF)
What is SPF?
Sender Policy Framework (SPF) is one of the primary means of authenticating email messages as they transit the Internet. The importance of SPF is growing as additional technologies, such as DMARC, are built on top of the standard. As more domains implement SPF records, greater value is derived from the framework for anti-spam systems and mailbox providers.
Creating and managing SPF records can be difficult though. A poorly configured record can cause a number of deliverability issues. We have seen some of the wackiest attempts at creating a record when working with clients, but a new research study reveals just how pervasive SPF issues are across the Internet. Keith Coleman and his team at Fraudmarc build tools that help organizations simplify SPF management. They recently analyzed the SPF records for all one million of the top web domains according to Alexa. Their research showed that about one out of every six domains that published an SPF record did so with invalid syntax or violated a restriction outlined in RFC7208.
We wanted take a look at the most common root causes of a bad SPF record and explain how they can be overcome. We will look at each problem in order of how negatively they will impact your deliverability.
Overly Permissive Records
A nice feature of SPF records is the ability to include ranges of IP addresses, rather having to specify them all individually. SPF allows for ranges to be provided in CIDR notation. A single typo though can dramatically change the value. For example, the range of IPs that ARIN has assigned to SocketLabs is:
This range includes 4,096 IP addresses, all of which we use for our email delivery network. If we were to forget the 0 at the end of this prefix we would have:
This single character changes the range from 4,096 IPs to 1,073,741,824. That is 25% of the entire Internet. If a malicious user is able to compromise a few machines, odds are one of them has an IP address in this range which could then be used to send spam messages that properly authenticate.
Anyone can mess this up. Not too long ago I received a spam message that properly authenticated as a fairly popular software company. A quick Twitter search revealed that lots of their users were frustrated about not getting their email messages. We noticed that their SPF record included the majority of the Internet and their domain was being heavily abused by spammers.
Google believed that these spam messages were truly originating from the software company. Since the spam volume likely outnumbered the good mail, all messages that authenticated as this domain were likely getting put into the spam folder.
This issue is so prevalent that now many mailbox providers are able to help reduce the potential issues from this kind of mistake. They do this by limiting the permitted CIDR size ranges they will view as valid in an SPF record. Would a domain ever allow 25% of the Internet to send mail on its behalf legitimately? We’ve heard that any inclusion greater than a /16 (65,536 IPs) may not be honored at some mailbox providers.
Another way that an SPF record can be overly permissive is if you terminate your SPF record in “+all”. This essentially causes your record to include the entire Internet as permitted to send mail on your behalf. Malicious domains registered for only spam distribution will often have records that end in “+all”. This way they can spew spam from any IP address on any machine their malware infects.
Identifying this type of issue can be hard as overly permissive records are often syntactically valid. Tools and resources to test your records may not identify one that is overly permissive. Here at SocketLabs we recommend for you to check that your SPF record does not include any ranges that end in a /16 or larger range, as these will rarely be valid.
Duplicate SPF TXT records
Another commonly violated aspect of SPF is that a domain may only have a single SPF record. That means you can only have a single DNS TXT record that begins with “v=spf1”. Having multiple records by definition of the RFCs should result in a permanent error.
Why are multiple SPF records so common? Part of the cause is that as an organization deploys different services such as SocketLabs, each provider often instructs them to create an SPF record. We’ve updated our knowledge base to be clear that our only requirement is the portion “include:email-od.com” is present in your SPF record. You can read our knowledge base article on SPF records here.
For organizations that have multiple SPF records, this is easily resolved by merging the records into a single statement. For example, if you had the following two records to authenticate both SocketLabs and Google Apps:
These records can easily be combined into:
Note: When combining two records ensure that the text “v=spf1” is only present once at the beginning of the record and the “all” mechanism is only present at the end of the record.
The amount of deliverability damage which duplicate records can cause is limited by some intelligence by large mailbox providers. Microsoft and Google both automatically fix duplicate SPF records. Smaller mail systems may not be so forgiving though, so this is an important and easily fixed issue that should be addressed when identified.
SPF Character Limits
Per the RFC, SPF records have a 255-character limit for a single string.
This is an inherent limit of DNS TXT records. Records that do not comply may be cause for temporary or permanent errors by receiving mail systems. Your DNS management system is hopefully smart enough to prevent you from exceeding this limit, but it may cause you issues trying to store a longer record.
While you can only have 255 characters in a string, as stated in RFC 1035, sections 3.3 and 3.3.14, you can have more than one string in a single DNS record. When a receiving mail server begins the process to analyze an SPF record, and it finds a record with multiple strings, it concatenates them together in order and without adding spaces before it evaluates the content.
A record that looks as follows:
will be read as:
While you can have multiple strings there is still a general limit in the total characters that should be in your record. This limit is harder to understand, but is related to ensuring that you do not cause DNS packets to exceed the reliable UDP packet length of 512 bytes. We recommend limiting records to around 460 characters. Steve Atkins of the email consulting firm Word to the Wise has a great post diving into the gritty details on these limits.
If your organization’s SPF record is going to exceed 460 characters, to prevent any SPF validation issues you will need to create sub records to split your single record across multiple records. You may also need to consider any other DNS TXT records on the domain as they are included in the 512 byte limitation of UDP.
When splitting SPF records, each sub record is then included in the primary. This type of work around is popular with very large mail systems, but you need to be careful doing this as it can created excessive DNS requests, an issue which we will dive into next. Here is how an extremely long record may look:
“v=spf1 ip4:10.0.0.0/24 include:email-od.com ip4:10.3.2.0/24 ip4:10.16.4.0/28 ip4:10.66.9.0/24 include:customerexample.com ip4:192.168.1.0 ip4:10.232.87.0/25 ip4:10.190.185.0/24 include:spf.protection.outlook.com ip4:10.10.119.0/24 ip4:10.99.14.0/24 ip4:10.240.176.0/20 ip4:10.0.14.0/24 include:_spf.google.com ip4:10.71.102.0/24 ip4:10.240.14.0/26 ip4:10.10.180.0/24 include:qa-test.spf.example.org ip4:10.15.190.0/28 ip4:10.46.188.0/22 include:production.spf.example.net ~all”
This could be broken apart into two sub-records of:
example.com TXT “v=spf1 ip4:10.0.0.0/24 include:email-od.com ip4:10.3.2.0/24 ip4:10.16.4.0/28 ip4:10.66.9.0/24 include:customerexample.com ip4:192.168.1.0 ip4:10.232.87.0/25 ip4:10.190.185.0/24 include:spf.protection.outlook.com ip4:10.10.119.0/24 include:spf2.example.com ~all”
spf2.example.com TXT “ip4:10.99.14.0/24 ip4:10.240.176.0/20 ip4:10.0.14.0/24 include:_spf.google.com ip4:10.71.102.0/24 ip4:10.240.14.0/26 ip4:10.10.180.0/24 include:qa.spf.example.org ip4:10.15.190.0/28 ip4:10.46.188.0/22 include:dev.spf.example.net ~all”
You will want to be as efficient as possible with how many times you split your record as each split will cause an additional DNS query by the receiving system.
Number of DNS Lookups Allowed in SPF
In addition to the 255-character limit, it is against RFC specifications to cause more than ten DNS look-ups in your record. This limit is designed to prevent denial of service attacks and infinite DNS loops. If you have more than ten lookups in your record, a permanent error could be returned during the SPF authentication process. The following mechanisms will cause a DNS query to occur:
- Include
- A
- MX
- PTR
- Exists
- Redirect
The only mechanisms that do not count against your DNS query total are:
- All
- ip4
- ip6
Be aware that nested include statements like the aforementioned use of sub records to split up a larger record can quickly increase the number of DNS queries your SPF record generates.
There are some well known organizations that use excessive numbers of DNS queries in their SPF record. Be careful when including a new service provider. For example, Microsoft’s SharePoint Online platform uses nine of the permitted ten DNS queries when using their recommendation of “include:sharepointonline.com”. Here at SocketLabs we do not cause any additional DNS queries beyond the initial inclusion.
Duplicate IP Addresses and Subnets
Ultimately, SPF entries all resolve to an IP address or subnet. When attempting to build an exhaustive list of all systems that send email, it might be easy to generate a duplicate entry. This may happen often when your SPF record has an include statement for your mailbox provider, and your domain’s MX records. They may be the same IPs, or IPs within subnets you are already including.
If the server’s IP rarely changes, consider using the ip4:x.x.x.x (or ip6) notation so recipients can avoid DNS lookups entirely. Since there is a limit of ten DNS lookups per SPF record, specifying an IP address or address range is preferable for long lists of outgoing mail servers
Other Considerations
- Don’t use the “exists” mechanism of SPF. If not used properly, it can be a major security risk.
- Make sure that all mechanisms (a, mx, include, etc) that use a domain based implementation (i.e. a:mail.example.com) properly resolve. We often see system administrators fail to remove deprecated and legacy systems such as an on-premise exchange server, that was replaced by Office365 cloud services. While the SPF record was updated to add Office365, it may not have been updated to remove the on-premise system. Now the “a:mail.example.com” which is still present in the record causes a temporary SPF failure because the domain doesn’t exist.
- Don’t use the SPF DNS record type. This has been deprecated in favor of TXT records.
- Don’t use the ‘ptr” mechanism as it has been deprecated in the most recent RFC draft of SPF.
Getting More Help On SPF Issues
Here at SocketLabs our team of email delivery experts are able to help our customers with creating the perfect SPF record. There are also lots of great online tools from a number of different organizations to help with record analysis and creation.
Word to the Wise’s Authentication Toolbox
Kitterman SPF Record Testing Tools
If you are not currently a SocketLabs customer but are interested in how we can help you with SPF or getting messages delivered, sign-up now and try it for free today!
