In troubleshooting TLS encryption problems with Exchange 2010, there can potentially be several places that an administrator should look to find where TLS encryption is breaking down. There are a number of issues that can exist just in the server setup itself that need to be checked before expanding the search out elsewhere. Having a properly setup digital certificate from a trusted certificate authority is one of the first checks an administrator needs to make. There are also many really good step by step articles straight from Microsoft that an administrator should at least give a once over just to make sure there isn’t something obvious with the cert itself or how Exchange is using it.
Taking a look at the link below is a good place to start with TLS and securing transport servers:
https://technet.microsoft.com/en-us/library/bb430764.aspx
A good article on understanding TLS certificates and looking at the terminology and configuration with these:
https://technet.microsoft.com/en-us/library/aa998840.aspx
However, there comes a point where an administrator must look to other possibilities when TLS just isn’t functioning correctly and a properly functioning certificate is verified.
There are a couple of areas that we want to mention in this post that could be a possibility, however, they exist in very specific environment setups.
Watchguard Firewalls
There is a known issue with Watchguard Firewall products that are caused by setting up SMTP proxy’s instead of SMTP filters. By default Watchguard configures the SMTP services by using these proxy setups instead of filters. Take a look at the post below:
https://support.google.com/postini/bin/answer.py?hl=en&answer=138468
Untangle UTM
In working in a client environment recently we discovered an environment where TLS was not working for the client. In digging a little deeper into their firewall and UTM setup, we found that the default value under the Untangle >> Config >> System >> Protocol settings was set to not allow TLS encryption. The screenshot below is after we changed the setting. After changing the setting in Untangle, the TLS issues were resolved!
A really great website for system administrators and mail administrators to bookmark if they don’t have it bookmarked already is mxtoolbox.com. Their free web tools are great for checking several of the big ticket items when it comes to the health of your mail server(s) from the outside in. You can check your MX records, blacklists, Reverse DNS, Whois, and many other items. If you suspect you are having problems with DNS or your MX configuration, this is a great place to verify that you have issues in those areas.
