SPF Test
Check what MX servers reject on spf fail
[Intro] [Description] [Results] [Conclusions] [Perl script]

Intro

I run an automated, non-intrusive SPF survey by testing 24,853 domains in the DNSWL list as of 10 May 2012. The test and its result are described below.

To discuss this survey, please subscribe to the Dnswl-users.

If you need to contact me: vesely at tana dot it.

Please answer the non-automated SPF-deployment-survey if you haven't done so already.

Readers may also be interested in the IETF's SPF survey.

up↑

Description

My surveyor-client connected to a domain's MX port 25, like so:

Server: 220 Server.example ESMTP
Client: HELO spftest.tana.it
Server: 250 Server.example pleased to meet you
Client: MAIL FROM:<dummy@spf-all.com>

I (ab)used the spf-all.com name because their SPF policy is as succinct and strict as possible: "v=spf1 -all". That way, they tell no mail is going to originate from theirs. Thus, SPF-aware servers know that the message that the client would send is forged. What do they reply? A possibility is the following:

Server: 550 5.7.0 Please see http://www.openspf.org/why.html?sender=dummy%40spf-all.com&ip=62.94.243.229&receiver=example.com
Client: RSET
Server: 250 2.0.0 Reset state
Client: MAIL FROM:<dummy@spftest.tana.it>
Server: 250 2.1.0 Sender <dummy@spftest.tana.it> ok
Client: QUIT
Server: 221 2.0.0 SMTP closing connection

I used an unknown IP (62.94.243.229) to run the test, so I did not expect to be whitelisted. However, in roughly 99% of cases, the test went on just like so:

Server: 250 2.1.0 Sender <dummy@spf-all.com> ok
Client: QUIT
Server: 221 2.0.0 SMTP closing connection

Those two behaviors are labeled R-ON-FAIL (R for reject) and ACCEPT respectively. The latter behavior does not allow to infer that servers don't do SPF authentication of incoming messages. Possibly, they'd use the result at some later stage of the SMTP transaction, or even on delivery. I heard that Postfix servers do SPF-rejection after RCPT TO, which I don't know how to test.

In addition to reject/accept behavior, the survey checked what SPF records the domain under test published. Part of the latter survey can be compared with the IETF's SPF survey (mentioned in the Intro). The DNSWL's classification provides some further insight on SPF deployment.

up↑

Results

Overall reject/accept behavior, in descending popularity. See the script for the exact meaning of result names.
resultcountexplanation
ACCEPT23,001 The server accepted dummy@spf-all.com.
SMTPNOCONN791 No good connection to port 25 could be established.
NXDOMAIN287 DNS lookup of domain failed
R-ON-FAIL221 The server rejected dummy@spf-all.com but accepted a non SPF-tainted sender after RSET.
NONE156 The server rejected dummy@spf-all.com and didn't accept RSET. Possibly, the server dropped the connection right away.
SERVFAIL153 DNS lookup of domain failed
SMTPERROR151 A connection was established, but the SMTP greeting code was not 220.
OTHER54 The server neither accepted nor rejected dummy@spf-all.com. This includes some kind of greylisting.
REJECT31 The server rejected both dummy@spf-all.com as well as the non SPF-tainted sender after RSET.
REJECTHELO8 The server did not accept the HELO command. 5 servers mentioned greylisting explicitly.

The exiguous number of domains deploying SPF's reject-on-fail feature seems to be at odds with the almost 50% awareness implied by the presence of some type of SPF record in the reachable domains. (The following table roughly agrees with the results of the IETF's SPF survey.)

Resource record type used
spf_typecountexplanation
NONE12,048 Domains that publish no usable SPF policy, including 572 permerror/temperror cases.
TXT10,915 Domains that publish SPF policies using the TXT RRTYPE.
NULL1,231 All and only those domains with connection problems (result=NXDOMAIN, SERVFAIL, or SMTPNOCONN).
BOTH587 Domains that publish a policy using both TXT and SPF RRTYPEs consistently.
SPF43 Domains that publish SPF policies using the SPF RRTYPE only.
VARY20 Domains that publish a different policy for each RRTYPE.
BAD9 Domains that caused an exception during the evaluation of their TXT policy.

SPF-aware domains can publish loose or strict policies, the former are about twice as many as the latter. A noticeable number of maybe-aware domains publish invalid policies. In the following table, the evaluation was done using TXT RRTYPEs only, against an unused test-address (203.0.113.99).

Result of SPF evaluation on a "default" IP address
spf_defaultcountexplanation
none11,515 Domains that don't publish a TXT SPF record.
softfail or neutral6,476 Loose policies, ending in either ~all (4,966) or ?all (1,510).
fail3,816 Policies ending in -all, the strict ones.
permerror or temperror1,713 All possible failures, either permanent (1,663) or temporary (50).
NULL1,240 All and only those domains with SPF problems (spf_type=NULL, BAD).
pass93 Policies ending in +all (or just all, since + is the default qualifier).

Those results can be broken down according to DNSWL's classification. DNSWL assigns a category and a score to each IP block it lists, thus one can derive a list of categories and a list of scores for each domain name. (I arbitrarily re-categorized[*] the 17 domains having two categories; none has more.) I discarded the problematic NULL results.

X-axis order: categories are arranged according to their ability to provide the receiving server with a meaningful SPF result. That is, the ratio of permerror/ valid, increasing left to right, where valid is any of fail, softfail, neutral or pass.

The resulting plot shows two facts very clearly:

I think readers can quickly work out how the user-admin relationship is characterized for each category, and thus explain why bald -all's are more common in some categories than in others. For example, Personal/private servers (cat. 6) obviously can control their users more closely.

Fig. 1: Result of SPF evaluation by category

Result of SPF evaluation by category

Now, for the corresponding behavior, the cases different from ACCEPT are so rare that we have to resort to logarithmic scale just to be able to see them --look at the numbers on the axis to see that the highest percentage of reject-on-fail, found at Service/Network providers (cat. 5), is just above 1%; 1.66% to be more precise.

Fig. 2: Reject/accept behavior by category

Reject/accept behavior by category

As for categories, the breakdown can be done by score classes. I put the domains with multiple scores in their own mixed class. Classes none, low, and med appear in an odd order, according to the SPF-oriented x-axis order used. Quite similar to one another, their permerror/ valid ratios are: 275permerror/ (277neutral + 919softfail + 653fail + 17pass) = 0.15, 835permerror/ (741neutral + 2470softfail + 1845fail + 46pass) = 0.16, and 365permerror/ (305neutral + 1020softfail + 858fail + 20pass) = 0.17 respectively.

Fig. 3: Result of SPF evaluation by score

Result of SPF evaluation by score

Again, we also look at the fraction of those domains that deploy a behavior different different from ACCEPT.

Fig. 4: Reject/accept behavior by score

Reject/accept behavior by score

Finally, we can count how many blocks are listed for each domain, and use that number as an additional classification, presumably related to the domain's size. The SPF-oriented placement of small domains seems to be in contrast with the fact that Personal/private servers (cat. 6) is leading the categories (Fig. 1). Indeed, the 1561 domains in category 6 have an average of 1.2 blocks each; topped by one domain with seven blocks. However, only 9% of the single-block domains is in category 6.

Fig. 5: Result of SPF evaluation by no-of-blocks ranges

Result of SPF evaluation by ranges

The class with 20-29 blocks features the top reject-on-fail percentage with 1.74%.

Fig. 6: Reject/accept behavior by no-of-blocks ranges

Reject/accept behavior by ranges

up↑

Conclusions

About a half of mail domains publish an SPF record of some kind, and the majority of them are oriented toward softfail or neutral default values. The choice of whether to publish SPF and how to qualify the all mechanism seems to be related to the type of mail domain; that is, its users and admins.

The ~all and -all default values are the most common. They are only different inasmuch as the receiving servers honor the sender's policy. However, reject-on-fail is so rare that it is not even obvious how to characterize it. We may say that publishing a strict policy is easier than honoring one. In fact, reject-on-fail requires a tight control on users: A mail admin must whitelist all the domains who legitimately forward mail to her site, but there is no well-established method to learn and maintain such a domain list.

up↑

Perl script

The script accepts input lines like
domain.example [ranges [category [score]]]
where the optional values are provided by dnswl.

#! /usr/bin/perl
# use: shuf < domain-list | runtest.pl
# check whether MX servers reject on fail

use strict;
use warnings;
use Net::DNS;
use Net::DNS::RR;
use NetAddr::IP;
use Mail::SPF;
use Error qw(:try);
use IO::Socket::INET;
use Net::Cmd;
use DBI;

@main::ISA = qw(Net::Cmd IO::Socket::INET);

# test options:
my $spf_server = Mail::SPF::Server->new(
	query_rr_types => Mail::SPF::Server->query_rr_type_txt);
my $spf_test_ip = new NetAddr::IP::Lite('203.0.113.99'); # TEST-NET-3 from RFC 5737

my $table_name = 'survey';
my $local_ip = '62.94.243.229';          # IP (or name) to use for SMTP client
my $helo_cmd = 'HELO spftest.tana.it';   # all cmd except trailing CRLF
my $mail_cmd = 'MAIL FROM:<dummy@spf-all.com>';
my $mail_chk = 'MAIL FROM:<dummy@spftest.tana.it>';
my $resolver = Net::DNS::Resolver->new;  # set any DNS options here

# DB options:
my $dbh = DBI->connect("DBI:mysql:database=mix;", "web") or
	die $DBI::err.": ".$DBI::errstr;
$dbh->do(' CREATE TABLE IF NOT EXISTS '. $table_name .' ('.
	' domain VARCHAR(64) NOT NULL PRIMARY KEY,'.
	' test_date TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,'.
	# the following column names are used in SET <column-name>=<value>
	' up_date TIMESTAMP,'.
	' ranges INT,'.            # number of different IP ranges with domain
	' score VARCHAR(16),'.     # from dnswl scores
	' category VARCHAR(16),'.  # categories
	' spf_type VARCHAR(4),'.   # NONE, TXT, SPF, BOTH, VARY, BAD
	' spf_queries TINYINT,'.   # additional queries to establish default result
	' spf_default VARCHAR(10),'.  # spf result for spf_test_ip (fail/softfail...)
	' result VARCHAR(10) NOT NULL,'. # R-ON_FAIL for reject-on-fail
	' host VARCHAR(64),'.    # mx server
	' mx_order TINYINT,'.    # 0 if using fallback-to-a, 1.. otherwise
	' banner TINYTEXT,'.
	' reply_mail TINYTEXT,'. # what did it say for mail_cmd
	' mail_code SMALLINT,'.
	' mail_chk TINYTEXT,'.   # what did it say for mail_chk command
	' mail_chk_code SMALLINT)') or
	die $DBI::err.": ".$DBI::errstr;
my $insert_stmt = 'REPLACE INTO '. $table_name .' SET';


# utility
sub trim
{
	my $s = shift;
	return '' unless defined $s;
	$s =~ s/^\s+|\s+$//g;
	return $s
}

my %results = ();
print 'Started ', $$, $/;

# main loop
while (<STDIN>)
{
	chomp;
	#              domain                 ranges              category  score
	next unless m/^([-._a-zA-Z0-9]+)(?:\s+([1-9][0-9]*)(?:\s+(\S+)(?:\s+(\S+))?)?)?\s*$/;
	my $domain = $1;
	my $result = 'UNEXPECTED';
	my $mx_order = 0;

	# Initialize db variables
	my $db_values = ' domain='. $dbh->quote($domain) .',up_date=NOW(),';
	$db_values .= 'ranges='. $2 .',' if defined($2);
	$db_values .= 'category='. $dbh->quote(trim($3)) .',' if defined($3);
	$db_values .= 'score='. $dbh->quote(trim($4)) .',' if defined($4);


	# Lookup MXes.  If none try domain name
	my @mx = mx($resolver, $domain);
	if (@mx <= 0)
	{
		if ($resolver->errorstring eq 'NOERROR')
		{
			$mx_order = -1;
			@mx = Net::DNS::RR->new("$domain. MX 0 $domain.");
		}
		else # SERVFAIL, NXDOMAIN
		{
			$result = $resolver->errorstring;
		}
	}

	# Try each MX until can connect to one
	foreach my $rr (@mx)
	{
		my $host = lc($rr->exchange);
		chomp($host);
		++$mx_order;
		$result = 'SMTPNOCONN';
		my $s = IO::Socket::INET->new(
			LocalAddr => $local_ip,
			PeerAddr => $host,
			PeerPort => 'smtp(25)',
			Timeout => 60)
				or next;
		
		# connected?
		my $sock = bless($s);
		$sock->response();
		next unless defined($sock->code) && defined($sock->message);
		
		$db_values .= 'host='. $dbh->quote(trim($host)) .','.
			"mx_order=$mx_order,".
			'banner='. $dbh->quote(trim($sock->message)) .',';

		# test if they define an SPF record, and its result on a dummy IP
		try
		{
			my $spf_SPF = $resolver->bgsend($domain, 'SPF');
			my $spf_request = Mail::SPF::Request->new(
				scope => 'helo', identity => $domain, ip_address  => $spf_test_ip);
			my $spf_result = $spf_server->process($spf_request);
			my $spf_SPF_packet = $resolver->bgisready($spf_SPF)?
				$resolver->bgread($spf_SPF) : undef;
			$spf_SPF = undef;
			if (defined($spf_SPF_packet) &&
				$spf_SPF_packet->header->rcode eq 'NOERROR')
			{
				my @versions = $spf_request->versions;
				my @recs =
					$spf_server->get_acceptable_records_from_packet(
						$spf_SPF_packet,
						'SPF',
						\@versions,
						$spf_request->scope,
						$spf_request->authority_domain);
				$spf_SPF = $recs[0] if (@recs == 1);
			}
			$db_values .= sprintf('spf_queries=%d,',
				$spf_request->state('dns_interactive_terms_count'));

			my $spf_type;
			my $record = $spf_request->record;
			if (defined($record) and defined($spf_SPF))
			{
				$spf_type = ($spf_SPF eq $record)? 'BOTH': 'VARY';
			}
			elsif (defined($record))
			{
				$spf_type = 'TXT';
			}
			elsif (defined($spf_SPF))
			{
				$spf_type = 'SPF';
			}
			else
			{
				$spf_type = 'NONE';
			}
			$db_values .= 'spf_type='. $dbh->quote(trim($spf_type)) .',';

			if (defined($spf_result))
			{
				$db_values .= 'spf_default='. $dbh->quote(trim($spf_result->code)) .',';
				#	'spf record: ', $record->text, "\n";
			}
		}
		otherwise
		{
			$db_values .= 'spf_type='. $dbh->quote('BAD') .',';
		};

		# Try sending helo + mailfrom(bad).
		# If rejected, RSET and try mailfrom(chk).
		# Then quit.
		if ($sock->code != 220)
		{
			$result = 'SMTPERROR';
		}
		else
		{
			$sock->command($helo_cmd)->response();
			if ($sock->code != 250)
			{
				$result = 'REJECTHELO';
				$db_values .= 'reply_mail='.
					$dbh->quote(trim($sock->message)) .',';
			}
			else
			{
				$sock->command($mail_cmd)->response();
				$db_values .= 'mail_code='. $sock->code .',';
				if ($sock->code == 250)
				{
					$result = 'ACCEPT';
				}
				elsif ($sock->code >= 500)
				{
					$db_values .= 'reply_mail='.
						$dbh->quote(trim($sock->message)) .',';
					$sock->command('RSET')->response();
					if ($sock->code == 250)
					{
						$result = 'REJECT';
						$sock->command($mail_chk)->response();
						$result = 'R-ON-FAIL' if ($sock->code == 250);
						$db_values .= 'mail_chk_code='. $sock->code .','.
							'mail_chk='. $dbh->quote(trim($sock->message)) .',';
					}
					else # doesn't take RSET
					{
						$result = 'NONE';
					}
				}
				else # invalid code (4xx?)
				{
					$result = 'OTHER';
				}
			}
		}
		$sock->command("QUIT");
		$sock->close();
		last;
	}
	
	# report result
	$results{$result} += 1;
	$db_values .= 'result='. $dbh->quote(trim($result));
	$dbh->do($insert_stmt . $db_values) or
		warn $DBI::err.": ".$DBI::errstr;
}

print $$, ' ', $_, ' ', $results{$_}, $/ foreach (sort (keys %results));
print 'End ', $$, $/;


zero rights

up↑