So, this is the part of my job/work which I love, but sometimes not so dearly: unsuspecting items cropping up from no-where when you are working on something really important or having an off-time :'(
Yesterday, post lunch, I was excited-ingly working on the architecture of a new platform plus apps which we need to deliver on a really short notice. And my Manager pinged me to let me know emails are not being dispatched from Imbibe’s Careers site.
Shouldn’t be too difficult I said to myself? After all, maybe some SMTP setting has gone awry or misplaced and I would just need to re-verify the same on the site’s admin panel. This was important to get it working for HR as they were actively interviewing candidates and having the notifications for form submissions was important to address them meaningfully.
For the record, the Careers site is built with Drupal and uses a G Suite account (basically a Gmail account) to send out emails. So I logged in to the site, verified the settings and voila, all was okay. But naah, emails weren’t going out. So I headed over to the associated Gmail account to verify the settings there.
Power users would know Google has made it increasingly protected (and correspondingly secure) to be able to use Gmail to send emails using SMTP. They first introduced the “Allow less secure apps” option for which I blogged years ago. Then they deprecated that feature encouraging users to enable 2-step verification to use Gmail. Most email clients like Outlook or Thunderbird updated themselves to use the 2-step auth work-flow to use such accounts. However, when using automated emails from sites/portals, 2-step auth is not an option and hence Google allowed using secure “App Passwords” to enable such use-cases.
I headed over to the account’s App Passwords section, and verified all was okay. The app password for the careers site was there:
It was last used on Mar 15, which obviously meant something was wrong and the emails were not being sent out since then. I even created a new App Password and tried using it on the Careers site and still got the same error:
phpmailerException: StartTLS not supported by server or could not initiate session. in PHPMailer->SmtpConnect() (line 780 of /home/public_html/site/sites/all/modules/smtp/smtp.phpmailer.inc).
Error in Drupal’s log while trying to send out emails
I swear I tried every approach I could find to fix that error, including changing Drupal’s SMTP’s module’s code to increase timeouts, manually change allowed TLS protocols to any that PHP makes available (check crypto_type section), playing between TLS and SSL protocols and what not. Every time, the error changed to something else, including:
Warning: stream_socket_enable_crypto(): SSL operation failed with code 1. OpenSSL Error messages: error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed in SMTP->StartTLS() (line 211 of /home/public_html/site/sites/all/modules/smtp/smtp.transport.inc).
Further error in Drupal’s log while trying to send out emails
And I kept playing, including disabling certificate validation completely (check this and this link) for catering to certificate verify failed error, making TLS protocols complaint with the ones Gmail supported, and what else not. I ended up parsing every single line of Drupal’s SMTP module, playing with it, scanning the excellent PHPMailer library, and performing every step from its troubleshooting guide, testing SSL outside PHP on the command-line (which worked) but the mails won’t simply go out from the site giving one or another of the above errors shared.
I enabled the Full-logging option from SMTP module which provided a lot of useful information while I was changing the module’s code:
Ultimately, working on it since yesterday, into the first half of today, I made 2 observations, I was able to get to a point where SSL errors went away, and all I was left with authentication errors trying to connect to SMTP server:
Error sending e-mail from [email protected] to [email protected] : SMTP Error: Could not authenticate.
Authentication error trying to use Gmail’s SMTP
phpmailerException: SMTP Error: Could not authenticate. in PHPMailer->SmtpConnect() (line 790 of /home/public_html/site/sites/all/modules/smtp/smtp.phpmailer.inc)
And another observation on Google App’s Password section where the “Last Used” column wasn’t getting updated. This meant the request wasn’t even reaching Gmail’s SMTP server even after SSL/TLS issues were resolved. At this point, I decided to use Zoho’s SMTP servers and switched configuration to use a Zoho’s email account instead.
To my absolute surprise, I got the exact same authentication error. Remember I had Full Logging enabled on SMTP module and I was stumped to see the verbose output matching verbatim for Gmail’s and Zoho’s SMTP servers. Which was kind of unexpected, the responses especially for HELO messages are not expected to be the same, character by character for bigger players like Gmail and Zoho which have their proprietary SMTP servers.
At this point, I moved my focus to whether the SMTP request was leaving my Linux VPS server at all. I tried playing with Local and Remote Mail Exchanger options on the associated CPanel account in WHM’s panel:
Tried switching between Local and Remote Mail Exchanger’s and eliminating the MX entries in WHM for the careers’ site domain, and everytime I tried, I still got the same authentication error. My focus was now on WHM settings and after some Googgling, I finally arrived on WHM’s SMTP Restrictions screen:
And this was my WTF moment. Over 6-7 hours of head-scratching, hair pulling, crypto protocol playing, and all you needed to do was push a button to disable SMTP Restrictions in WHM :-#
And it made sense to me now. To lighten load on our existing VPS server, we had procured a new Linux VPS in second week of March and moved some of our sites including Imbibe’s Careers site there. Looks like WHM introduced this option in a recent version and I wasn’t aware of the same.
The good thing is I ended up learning a lot about SMTP protocol’s internals, including the difference in working between SSL and TLS encryption methods for SMTP. And also a lot about how PHPMailer supports SMTP inside a single file. Happy Learning 🙂
So back to my architecture project now :D. But hey’s its already past end of working day 😉
Recent Comments