About a week ago I realised that an email I had sent to a Gmail address had not reached the recipient, who later found it in her spam folder. I had known about an issue with my mail setup for a long time but now was the time to finally do something about it.
Google has been phasing in an email validation system called
Sender Policy Framework (SPF) which provides a mechanism for their receiving mail exchanger to check that incoming mail from a domain comes from a host authorised by that domain’s administrators. I have a few domains hosted by Mythic Beasts and one of them (zenatode.org.uk) I use for my primary email address. I deal with all my mail on an iMac at home where I get my Internet connection from Andrews & Arnold (which I will refer to as AA). Until recently I had been sending mail via the AA outgoing SMTP server, but without an SPF record to tell mail exchangers that the AA server was authorised. In order to reliably send mail to Gmail addresses I was going to have to create an SPF record.
Now I could have created a record authorising the AA server to send mail from zenatode but there were a few reasons to make some changes. To start with, I have occasionally taken my iMac with me when house-sitting for friends and since they use a different ISP I had to either authenticate to the AA server to prove I was entitled to use it, or change my configuration to send mail via the server appropriate to the setup where I was staying. I chose the latter, but if I had created an SPF record authorising only AA to send then that wouldn’t have worked. So what about setting things up to authenticate to the AA server allowing me to use it from anywhere? I could have done that, and it would have forced me to also use a secure connection (until now, since I hadn’t been sending a password to authenticate I hadn’t bothered, which has meant that my outgoing mail has been travelling from my computer to the AA server in the clear). However, there was another option, which was to switch to using the Mythic Beasts outgoing SMTP server with a secure authenticated connection. In the end that is what I decided to do, partly because Mythic Beasts provide a simple one-click mechanism for creating an SPF record for a hosted domain – provided you are using their own outgoing server.
I looked at the Mythic Beasts support pages to find the name of their outgoing server and the required settings. What about a username and password to authenticate? It said you could use the email address of any mailbox you had set up on their system as a username and the mailbox password as the password. The trouble is I had not created any mailboxes! I had been funnelling mail for all my domains into my Mythic Beasts main account mailbox which used my main account username and password, and those don’t work as credentials for authenticating to the outgoing server. So I created a mailbox using the Mythic Beasts control panel and while I was at it redirected mail for all my domains into that. I now had a single new username and password for both sending and receiving mail. On my iMac I use a minimal SMTP client called msmtp to do the actual sending and its settings go in a configuration file. I edited the configuration file to create a new “account” with the required settings, then made that the default account and tried sending something. It failed with an error saying that TLS (Transport Layer Security) requires either tls_trust_file or tls_fingerprint or a disabled tls_certcheck – for the time being I just disabled tls_certcheck. I then tried again and got a different error which said that the TLS handshake had failed. Changing tls_starttls from off to on fixed that and I was now able to send mail again.
Having got this far it was now trivial to go to the Mythic Beasts control panel again and click the button to create an SPF record for each of my domains. The SPF record is published in the DNS as a specially formatted TXT record. I searched for SPF record testing tools and found
these at Kitterman Technical Services. I entered zenatode.org.uk as the domain in the first box and clicked “Get SPF Record (if any)”. It found a valid SPF record which said “v=spf1 include:_spf.mythic-beasts.com ~all”.
The final test was to send something to a Gmail address and see what happens. I have a Gmail address which is set to forward to my zenatode address so I sent a test message to that. If Google had put it in my spam folder it would not have been forwarded so the fact that it got back to me straight away was promising. It didn’t prove that SPF was working because most mail was getting through before. However, I was able to look at the message headers, where I found a Received-SPF header which said that the message had passed Google’s SPF test. Hopefully this will dramatically reduce the chance that mail I send ends up in a spam folder.