Acquiring a large SMTP pcap for testing
Recently I needed a large PCAP of SMTP traffic for testing zeek scripts. Obtaining a capture from a production network would be a straightforward process. However, I needed a file that could be included in test suites, or copied to other hosts without having to worry about leaking sensitive data.
I could have taken an archived MBOX file and replayed it into a server. This would have worked, but wouldn't have included any variability due to client differences. When testing tools that are analyzing traffic, you want the traffic to be as real world as possible. Replaying a MBOX file would have given me real world mail data, but it would have only consisted of SMTP behaviour from a single client implementation.
My solution was to set-up a SMTP server configured to accept all email sent to it. Concurrently, tcpdump would be running to capture every packet transmitted over port 25.
I set this up using a VPS and the following script
Create a locked user to act as the destination for the email
adduser spam passwd -l spam
Install postfix with the pcre module
apt-get install postfix-pcre
Route all email to the spam user
echo '/.*/ spam' > /etc/postfix/virtual echo 'virtual_alias_maps = pcre:/etc/postfix/virtual' >> /etc/postfix/main.cf
Disable TLS - we want to analyze SMTP commands, not TLS handshakes
perl -pi -e 's/smtpd_use_tls=yes/smtpd_use_tls=no/' /etc/postfix/main.cf
Create a directory for storing the pcap files
mkdir -p /data/pcaps
Create a wrapper script
This will create pcap files based on the date and limited to at most 100MB. The filenames aren't super important because things can be merged and re-split later on.
cat <<'END' > /usr/local/bin/start_pcap #!/bin/sh exec /usr/sbin/tcpdump -i eth0 -s 0 -C 100 -w /data/pcaps/smtp.$(date +'%Y-%m-%d_%H_%m').pcap 'port 25' END chmod +x /usr/local/bin/start_pcap
Have systemd run this script at boot and ensure it's always running.
cat <<END > /etc/systemd/system/pcap.service [Unit] Description=full pcap for smtp After=network.target [Service] User=root Group=root ExecStart=/usr/local/bin/start_pcap Restart=always RestartSec=2s [Install] WantedBy=multi-user.target END systemctl enable pcap systemctl start pcap
Once I had this running I figured I would start receiving spam, but that turned out to be harder than I thought. Tweeting out an email address resulted in zero emails. Putting an address into a cryptocurrency newsletter sign-up form resulted in exactly the one email per week as promised. Various “sign-an-address-up-for-a-lot-of-email” websites failed to work at all.
To get the process going I signed an address up to the Linux kernel mailing list. Living up to its reputation as a high volume list, I have received about 13,000 emails. This isn't ideal, but it's a start.
After two weeks I have about 100MB of MBOX and PCAP files:
96M /data/pcaps/smtp.2019-12-13_18_12.pcap 29M /data/pcaps/smtp.2019-12-24_02_12.pcap 97M /var/spool/mail/spam