I run an automated, non-intrusive SPF survey by testing 24,853 domains in the DNSWL list as of 10 May 2012. The test and its result are described below.
To discuss this survey, please subscribe to the Dnswl-users.
If you need to contact me:
Please answer the non-automated SPF-deployment-survey if you haven't done so already.
Readers may also be interested in the IETF's SPF survey.
My surveyor-client connected to a domain's MX port 25, like so:
Server: 220 Server.example ESMTP Client: HELO spftest.tana.it Server: 250 Server.example pleased to meet you Client: MAIL FROM:<email@example.com>
I (ab)used the
because their SPF policy is as succinct and strict as possible:
That way, they tell no mail is going to originate from theirs.
Thus, SPF-aware servers know that the message that the client would send is forged.
What do they reply? A possibility is the following:
Server: 550 5.7.0 Please see http://www.openspf.org/why.html?sender=dummy%40spf-all.com&ip=220.127.116.11&receiver=example.com Client: RSET Server: 250 2.0.0 Reset state Client: MAIL FROM:<firstname.lastname@example.org> Server: 250 2.1.0 Sender <email@example.com> ok Client: QUIT Server: 221 2.0.0 SMTP closing connection
I used an unknown IP (18.104.22.168) to run the test, so I did not expect to be whitelisted. However, in roughly 99% of cases, the test went on just like so:
Server: 250 2.1.0 Sender <firstname.lastname@example.org> ok Client: QUIT Server: 221 2.0.0 SMTP closing connection
Those two behaviors are labeled
R for reject) and
The latter behavior does not allow to infer that servers don't do SPF authentication of incoming messages.
Possibly, they'd use the result at some later stage of the SMTP transaction, or even on delivery.
I heard that Postfix servers do SPF-rejection after
RCPT TO, which I don't know how to test.
In addition to reject/accept behavior, the survey checked what SPF records the domain under test published. Part of the latter survey can be compared with the IETF's SPF survey (mentioned in the Intro). The DNSWL's classification provides some further insight on SPF deployment.
The server accepted |
|791||No good connection to port 25 could be established.|
|287||DNS lookup of domain failed|
The server rejected |
The server rejected |
|153||DNS lookup of domain failed|
|151||A connection was established, but the SMTP greeting code was not 220.|
The server neither accepted nor rejected |
The server rejected both |
|8||The server did not accept the HELO command. 5 servers mentioned greylisting explicitly.|
The exiguous number of domains deploying SPF's reject-on-fail feature seems to be at odds with the almost 50% awareness implied by the presence of some type of SPF record in the reachable domains. (The following table roughly agrees with the results of the IETF's SPF survey.)
Domains that publish no usable SPF policy, including 572 |
|TXT||10,915||Domains that publish SPF policies using the TXT RRTYPE.|
All and only those domains with connection problems (result=|
|BOTH||587||Domains that publish a policy using both TXT and SPF RRTYPEs consistently.|
|SPF||43||Domains that publish SPF policies using the SPF RRTYPE only.|
|VARY||20||Domains that publish a different policy for each RRTYPE.|
|BAD||9||Domains that caused an exception during the evaluation of their TXT policy.|
SPF-aware domains can publish loose or strict policies, the former are about twice as many as the latter. A noticeable number of maybe-aware domains publish invalid policies. In the following table, the evaluation was done using TXT RRTYPEs only, against an unused test-address (203.0.113.99).
|none||11,515||Domains that don't publish a TXT SPF record.|
|softfail or neutral||6,476||
Loose policies, ending in either |
Policies ending in |
|permerror or temperror||1,713||All possible failures, either permanent (1,663) or temporary (50).|
|NULL||1,240||All and only those domains with SPF problems (spf_type=NULL, BAD).|
Policies ending in |
Those results can be broken down according to DNSWL's classification. DNSWL assigns a category and a score to each IP block it lists, thus one can derive a list of categories and a list of scores for each domain name. (I arbitrarily re-categorized[*] the 17 domains having two categories; none has more.) I discarded the problematic NULL results.
X-axis order: categories are arranged according to their ability to
provide the receiving server with a meaningful SPF result. That is,
the ratio of
permerror/ valid, increasing left to right,
where valid is any of
The resulting plot shows two facts very clearly:
Except for Email Marketing Providers (cat. 15),
followed by loose policies (
except in some special cases (cat. 10).
temperror are nearly invisible, policies afflicted by permanent
errors are rather common.
I think readers can quickly work out how the user-admin relationship is
characterized for each category, and thus explain why bald
-all's are more
common in some categories than in others.
For example, Personal/private servers (cat. 6) obviously can control their
users more closely.
Fig. 1: Result of SPF evaluation by category
Now, for the corresponding behavior, the cases different from
ACCEPT are so rare
that we have to resort to logarithmic scale just to be able to see them
--look at the numbers on the axis to see that the highest percentage of
reject-on-fail, found at Service/Network providers (cat. 5), is just
above 1%; 1.66% to be more precise.
Fig. 2: Reject/accept behavior by category
As for categories, the breakdown can be done by score classes.
I put the domains with multiple scores in their own mixed class.
Classes none, low, and med appear in an odd order, according to the SPF-oriented
x-axis order used.
Quite similar to one another, their
permerror/ valid ratios are:
neutral + 919
softfail + 653
fail + 17
pass) = 0.15,
neutral + 2470
softfail + 1845
fail + 46
pass) = 0.16, and
neutral + 1020
softfail + 858
fail + 20
pass) = 0.17 respectively.
Fig. 3: Result of SPF evaluation by score
Fig. 4: Reject/accept behavior by score
Finally, we can count how many blocks are listed for each domain, and use that number as an additional classification, presumably related to the domain's size. The SPF-oriented placement of small domains seems to be in contrast with the fact that Personal/private servers (cat. 6) is leading the categories (Fig. 1). Indeed, the 1561 domains in category 6 have an average of 1.2 blocks each; topped by one domain with seven blocks. However, only 9% of the single-block domains is in category 6.
Fig. 5: Result of SPF evaluation by no-of-blocks ranges
Fig. 6: Reject/accept behavior by no-of-blocks ranges
About a half of mail domains publish an SPF record of some kind, and the majority
of them are oriented toward
neutral default values.
The choice of whether to publish SPF and how to qualify the
mechanism seems to be related to the type of mail domain; that is, its users and admins.
-all default values are the most common.
They are only different inasmuch as the receiving servers honor the sender's policy.
However, reject-on-fail is so rare that it is not even obvious how to characterize it.
We may say that publishing a strict policy is easier than honoring one.
In fact, reject-on-fail requires a tight control on users:
A mail admin must whitelist all the domains who legitimately forward mail to her site,
but there is no well-established method to learn and maintain such a domain list.
The script accepts input lines like
domain.example [ranges [category [score]]]
where the optional values are provided by dnswl.
Copyright (C) 2012 Alessandro Vesely, all rights reserved except as noted.