<nettime> greenhost.nl: Spam? Not Spam? Tracking a hijacked Spamhaus IP

["nettime's_spam_kr!k!t" <nettime@kein.org>]

<https://greenhost.nl/2013/03/21/spam-not-spam-tracking-hijacked-spamhaus-ip/>


Spam? Not Spam? Tracking a hijacked Spamhaus IP
 
Today we noticed we that a lot of our incoming email was being marked as
spam. Since this included many messages that were clearly not spam, we
decided to investigate more deeply. We found that a third party was
subverting the Spamhaus service using a routing-based attack.
 
First clues: Spamhaus misbehaving?

To filter out spam we use a combination of about 30 different rules. One of
them is based on the Spamhaus advisory services, which are a family of
DNS-based block lists (DNSBLs) commonly used to filter email. Spamhaus
keeps track of servers which are sending large amounts of spam, and
publishes the IP addresses of these servers. In our system, when an email
comes in, we use Spamhaus to look up the IP address of the remote server.
If we find a listing, we compute that the chances are higher that the
message is spam.

As each incoming email goes through our system, we record which anti-spam
rules contributed to its overall spam score. Today there were many more
points attributed to Spamhaus advice then usual, so we suspected something
was wrong with Spamhaus or that they changed their systems in some major
way. But we checked with them, and found nothing had changed. Even more
surprising, the IP addresses our system was recognizing as "listed in
Spamhaus" did not have official records on the Spamhaus web site.

Spamhaus is checked using DNS, so we can use simple DNS client tools to
make queries by hand. For example, if an email is received from IP address
24.43.30.128, we check whether that address exists in the Spamhaus "ZEN"
list on the UNIX command line as follows:

	$ host 128.30.43.24.zen.spamhaus.org
	128.30.43.24.zen.spamhaus.org has address 127.0.0.2

A reply (always in the form 127.x.x.x) means that there is a positive match
in the database.

Spamhaus presents 22 public DNS servers for querying their system. After
some testing, we discovered that one of these (ns0.spamhaus.org,
204.16.254.40) was always returning "127.0.0.2" to any query. So it
seemed one of their systems was serving bad data, reporting that every IP
address on the internet was a source of spam. This meant about 5% (1/22) of
our incoming emails were getting a much higher spam score than they
deserved, and thus a larger proportion of messages were being marked as
spam.

Locating a rogue Spamhaus server

As Spamhaus has a good track record, we thought this behavior was
uncharacteristic. An error like this would normally be fixed quickly. So we
suspected something more complex was going on. We looked up the server in
the RIPE database and found it was in a group of addresses controlled by
Vitalwerks Internet Solutions. Then we performed a traceroute to the
address, which produced some strikingly unusual results:

	# traceroute -n 0.ns.spamhaus.org
	traceroute to 0.ns.spamhaus.org (204.16.254.40), 30 hops max, 60 byte packets
	 1  xxx.xxx.xxx.xxx  0.213 ms  0.189 ms  0.150 ms
	 2  xxx.xxx.xxx.xxx  0.945 ms  0.971 ms  0.945 ms
	 3  * * *
	 4  * * *
	 5  * * *
	 6  204.16.254.40  17.831 ms  17.752 ms  17.716 ms

Several 'hops' along the path (3, 4 and 5) did not reveal themselves to the
traceroute at all. We expected to see some hops belonging to Vitalwerks.
One 'missing' hop is not very unusual, but several consecutive hops (almost
the entire route!) being missing was another clue to something fishy. We
had to look more deeply, by asking our BGP routers, to find out where this
misbehaving Spamhaus server was really located:

	router# show ip bgp 204.16.254.40
	BGP routing table entry for 204.16.254.40/32
	Paths: (1 available, best #1, table Default-IP-Routing-Table)
	  Advertised to non peer-group peers:
	  xxx.xxx.xxx.xxx
	  34109 51787 1198
		193.239.116.204 from 193.239.116.204 (84.22.127.134)
		 Origin IGP, metric 10, localpref 140, valid, external, best
		  Last update: Tue Jan  5 11:57:27 1971

Finding the truth, getting back to normal

As can seen from the BGP output, we were using a /32 route going over AS
34109. This was highly suspicious for two reasons. First, a /32 route
refers only to a single IP address. Except in special cases, routes are
normally /24 (256 hosts) or larger. Second, the AS 34109 belongs to CB3ROB
which is an Internet provider that has actually been in conflict with
Spamhaus (see: spamhaus; allspammedup; theregister). Certainly they weren't
running a legitimate Spamhaus server.

It seems clear that the CB3ROB network hijacked one (or more) of the IP
addresses of Spamhaus, and installed a DNS server there which incorrectly
returns positive results to every query. The result causes harm to Spamhaus
users and their customers, making Spamhaus unusable for anyone unable to
correct the problem as we did, and perhaps even undermining the credibility
of Spamhaus itself.

As we take measures to protect our routers from route hijacking, we were
surprised to find this rogue entry appeared in our routing tables. We
discovered a small misconfiguration in our software, and after fixing that
the problem was solved. We could once again reach the real
0.ns.spamhaus.org server:

	# traceroute -q1 0.ns.spamhaus.org
	traceroute to 0.ns.spamhaus.org (204.16.254.40), 30 hops max, 60 byte packets
	 1  xxx.xxx.xxx.xxx  0.190 ms
	 2  xxx.xxx.xxx.xxx  0.394 ms
	 3  xxx.xxx.xxx.xxx  10.967 ms
	 4  r22.amstnl02.nl.bb.gin.ntt.net (195.69.144.36)  1.961 ms
	 5  ae-2.r03.amstnl02.nl.bb.gin.ntt.net (129.250.2.211)  3.695 ms
	 6  xe-3-0-3.ar1.ams3.nl.nlayer.net (69.22.139.202)  3.700 ms
	 7  as23352.vlan-102.ar1.ams3.nl.nlayer.net (69.22.139.123)  2.562 ms
	 8  ge0-4.aggrB3.ams3.nl.scnet.net (205.234.220.231)  3.953 ms
	 9  204.16.254.40 (204.16.254.40)  2.393 ms

Route hijacking has happened before, such as when Pakistan Telecom started
announcing itself as the route to YouTube in 2008, but it is still rather
unusual.

As the core routing protocol of the Internet, playing "tricks" with BGP can
be hazardous to the whole network. Moreover, Internet peering (route
exchanging) relationships are based on building trust, so we don't think
BGP is the place to make political demonstrations that cause network
interruptions. We need to build trust between small network participants
like Greenhost and CB3ROB, to foster digital diversity and innovation. But
for now, we have suspended our BGP peering with the network that caused us
all this trouble today -- and our email services are back to normal.


#  distributed via <nettime>: no commercial use without permission
#  <nettime>  is a moderated mailing list for net criticism,
#  collaborative text filtering and cultural politics of the nets
#  more info: http://mx.kein.org/mailman/listinfo/nettime-l
#  archive: http://www.nettime.org contact: nettime@kein.org