Skip to content

TLS and Certificates

Overview

Transport Layer Security (TLS) provides encryption, authentication, and data integrity for network communications. It uses asymmetric cryptography (public/private key pairs) to establish a secure channel, after which symmetric encryption handles the actual data transfer for performance reasons.

Certificates bind a public key to an identity (e.g., a domain name) and are signed by a Certificate Authority (CA) that vouches for the binding. The chain of trust — from root CAs pre-installed in browsers/operating systems, through intermediate CAs, to the end-entity certificate — is how clients verify they are talking to the legitimate server.

How It Works

Asymmetric Cryptography

Asymmetric cryptography allows you to generate a key pair consisting of:

  • A private key — kept secret, never shared with anyone
  • A public certificate (or public key) — freely distributed

When someone wants to send you private data, they encrypt it using your public certificate. After encryption, the only way to decrypt the data is with your private key. This is the same principle SSH keys use.

In the context of web services, the server presents its public certificate to connecting clients. Clients encrypt their data with this certificate before sending it. The server then decrypts using its private key. This allows secure data exchange while the data remains unreadable to anyone intercepting the traffic.

You can inspect any server's public certificate with:

openssl s_client -connect google.com:443

Certificate Issuance and Trust

If anyone could generate a certificate for any domain, it would be trivial to impersonate websites (e.g., a bank). To prevent this, certificates go through a validation process:

  1. Generate a private key on the server
  2. Create a Certificate Signing Request (CSR) based on the key
  3. Submit the CSR to a Certificate Authority (CA)
  4. Validate domain ownership — done via email, phone, DNS challenge, or HTTP challenge
  5. Receive and install the signed certificate

This process allows clients to verify trust. Most operating systems and browsers ship with a built-in list of trusted root CAs and their public keys. When a client connects to a server, it cryptographically validates the certificate chain:

Root CA (pre-installed in browser/OS)
  └── Intermediate CA (signed by Root CA)
        └── Server Certificate (signed by Intermediate CA)

If the chain is valid and leads back to a trusted root, the connection is trusted. If not (e.g., self-signed certificates), the browser displays a security warning.

Certificate Types

Standard certificates are issued for a specific FQDN (e.g., www.example.com).

Wildcard certificates use * to cover all subdomains of a domain (e.g., *.example.com covers www.example.com, mail.example.com, shop.example.com). While convenient, wildcard certificates carry a risk: if one service is compromised and the private key is exposed, all services using that certificate become vulnerable.

Certificate File Formats and Locations

Certificates are most commonly stored in PEM format (Base64-encoded, enclosed between -----BEGIN CERTIFICATE----- and -----END CERTIFICATE----- markers).

Standard file locations on RHEL/CentOS systems:

  • Certificate: /etc/pki/tls/certs/<service>.crt
  • Private key: /etc/pki/tls/private/<service>.key
  • CA chain: /etc/pki/tls/certs/cacert.crt

The CA chain file contains the intermediate and/or root CA certificates that form the trust chain. Each certificate in the chain is a separate PEM block in the same file.

TLS in Practice

TLS is applied to existing protocols by either:

  • Wrapping the entire connection in TLS from the start (implicit TLS) — used by HTTPS (443), IMAPS (993), SMTPS (465)
  • Upgrading a plaintext connection to TLS mid-stream using STARTTLS — used by SMTP submission (587), IMAP (143)

TLS in Apache (HTTPS)

HTTPS uses port 443 instead of port 80. Apache requires the mod_ssl module and three directives in the virtual host configuration:

<VirtualHost *:443>
    ServerName www.example.com
    SSLEngine on
    SSLCertificateFile /etc/pki/tls/certs/www_server.crt
    SSLCertificateKeyFile /etc/pki/tls/private/www_server.key
    SSLCACertificateFile /etc/pki/tls/certs/cacert.crt
    # ... rest of virtual host config
</VirtualHost>

TLS in Postfix (SMTP)

Postfix uses TLS for both server-to-server relay and client submission:

  • Port 25 (relay): smtpd_tls_security_level = may — offers TLS but doesn't require it (other servers may not support it)
  • Port 587 (submission): smtpd_tls_security_level = encrypt — requires TLS before accepting authentication
  • Port 465 (SMTPS): smtpd_tls_wrappermode = yes — implicit TLS from connection start

Key main.cf settings:

smtpd_tls_security_level = may
smtpd_tls_cert_file = /etc/pki/tls/certs/postfix.pem
smtpd_tls_key_file = /etc/pki/tls/private/postfix.key
smtpd_tls_loglevel = 1

TLS in Dovecot (IMAP)

Dovecot's TLS configuration in 10-ssl.conf:

ssl = yes
ssl_cert = </etc/pki/dovecot/certs/dovecot.pem
ssl_key = </etc/pki/dovecot/private/dovecot.pem

Note the < prefix — this tells Dovecot to read the file contents, not treat the path as a string value. Port 993 provides implicit TLS (IMAPS).

Verifying TLS Configuration

Use openssl s_client to verify TLS is working correctly:

# HTTPS
openssl s_client -connect www.example.com:443 -CAfile /path/to/cacert.crt

# SMTP with STARTTLS
openssl s_client -connect mail.example.com:587 -starttls smtp -CAfile /path/to/cacert.crt

# SMTPS (implicit TLS)
openssl s_client -connect mail.example.com:465 -CAfile /path/to/cacert.crt

# IMAPS
openssl s_client -connect mail.example.com:993 -CAfile /path/to/cacert.crt

The critical field in the output is Verify return code: 0 (ok). A non-zero return code indicates a trust chain problem.

Key Terminology

PKI (Public Key Infrastructure)
The framework of policies, hardware, software, and procedures needed to create, manage, distribute, and revoke digital certificates.
Certificate Authority (CA)
An entity that issues digital certificates, vouching that the public key in the certificate belongs to the named subject.
CSR (Certificate Signing Request)
A message sent to a CA to request a signed digital certificate. Contains the public key and identifying information.
CN (Common Name)
The fully qualified domain name (FQDN) that the certificate is issued for, specified in the CSR.
Chain of Trust
The hierarchical path from a server's certificate through intermediate CAs to a trusted root CA.
Self-Signed Certificate
A certificate signed by its own private key rather than a CA. Useful for testing but triggers browser warnings in production.
Let's Encrypt
A free, automated Certificate Authority that issues domain-validated certificates. Requires the server to be publicly accessible.
STARTTLS
A protocol command that upgrades an existing plaintext connection to an encrypted TLS connection on the same port.

Why It Matters

As a system administrator, you will:

  • Obtain and install TLS certificates for all public-facing services (web, mail, etc.)
  • Configure HTTPS, SMTPS, and IMAPS to protect data in transit
  • Manage certificate renewals before expiration (certificates have a limited TTL)
  • Maintain the chain of trust by correctly configuring CA chain files
  • Protect private keys with proper file permissions (typically 0600)
  • Use tools like Wireshark to verify that traffic is actually encrypted

Without TLS, all data travels in plaintext — passwords, emails, form submissions, and session cookies are visible to anyone who can intercept the network traffic.

Common Pitfalls

  1. Expired certificates — certificates have a validity period. Letting them expire causes service outages and browser warnings. Automate renewal where possible.
  2. Incorrect CA chain — forgetting to include intermediate CA certificates means clients can't validate the trust chain, even if the server certificate itself is valid.
  3. Wrong PEM format — CA chain certificates must be in proper PEM format with each certificate as a separate block. Comma-separated or malformed chains cause validation failures.
  4. Permissive private key files — private keys should be readable only by the service that needs them (chmod 0600). World-readable keys are a critical security issue.
  5. Forgetting firewall rules — TLS-enabled services use different ports (443, 465, 587, 993). These must be opened in the firewall.
  6. Mixing up implicit TLS and STARTTLS — port 465 uses implicit TLS (connection starts encrypted), while port 587 uses STARTTLS (starts plaintext, then upgrades). Using the wrong openssl command to test will fail.
  7. Self-signed certificates in production — while useful for development, self-signed certificates cause browser warnings and should not be used for public services.
  8. Not testing after configuration — always verify with openssl s_client that the certificate, chain, and key are correctly configured. Check that Verify return code is 0.

Further Reading