How to Run a Mail Server on a Dedicated Server (and Keep IP Reputation Clean)

Running your own mail server on a dedicated server gives you full control over your email: your own dedicated IP address, your own sending limits, your own data and no per-mailbox fees. It also makes you responsible for deliverability. If your messages land in spam, the cause is almost always a missing DNS record, a mismatched reverse DNS entry or a damaged IP reputation, not the mail software itself.

This tutorial shows you how to set up a secure mail server on Ubuntu 24.04 with Postfix (sending and receiving), Dovecot (mailbox access) and OpenDKIM (message signing), then how to configure SPF, DKIM, DMARC and reverse DNS, and how to keep your IP reputation clean over time.

What you will build: A mail server that sends and receives email for your own domain, encrypted connections (TLS) for SMTP and IMAP, SPF, DKIM and DMARC authentication, a dedicated IP with matching reverse DNS (PTR), and a routine for warming up, monitoring and protecting your sending reputation.

Time needed: about 60 to 90 minutes, plus DNS propagation. Difficulty: intermediate. You should be comfortable with SSH and editing config files on Linux.

1. Should You Self-Host Email on a Dedicated Server?

Self-hosting email is worthwhile in some situations and a poor idea in others.

It makes sense when you:

• Want full ownership of your email data and metadata

• Send transactional email (receipts, alerts, password resets) from your own applications

• Need many mailboxes without per-user pricing

• Have compliance or data-location requirements

• Want a dedicated IP whose reputation depends only on you

It may not be the right choice when you:

• Send large volumes of bulk marketing email with no deliverability experience (a specialist email delivery service is usually easier)

• Cannot spend time on monitoring and updates

• Need a guaranteed inbox-placement SLA

A dedicated server helps because you get a dedicated IPv4 address, control over reverse DNS, full control over open ports, and no other tenant sharing your IP and its reputation. A VPS can work too, but shared or recycled IP ranges more often come with blocklist history.

2. Prerequisites

Check these before you start. Most failed mail server setups fail on one of them.

A small server is enough for a typical mailbox workload. Memory needs grow with spam filtering and mailbox count. See our guide to how much RAM a dedicated server needs if you plan to add filtering or webmail later.

Test outbound port 25 first:

If this connects, outbound SMTP is open. If it times out, ask your provider to allow port 25 before you continue.

Acceptable use: Follow your provider's acceptable use policy and applicable anti-spam laws. Send only to recipients who expect your mail, and include a working unsubscribe method for marketing email.

Step 1: Set the Hostname and Prepare the Server

Choose a fully qualified domain name (FQDN) for your mail server, such as mail.example.com. Use your own domain throughout this guide.

Confirm the hostname resolves locally by adding it to /etc/hosts:

Replace 203.0.113.10 with your server's public IPv4 address.

Open the firewall ports you need. If you use UFW:

Securing the server itself is just as important as configuring mail. Follow our dedicated server hardening checklist before you put the server on the public internet.

Step 2: Create the DNS Records

Mail depends on DNS more than any other service. Create these records at your DNS provider. If you are new to DNS, our DNS zone guide for beginners explains how zones and records work.

The DKIM record is added in Step 6, after you generate the key.

About the reverse DNS (PTR) record: You cannot set it in your domain's DNS. It belongs to the IP address owner, so set it in your provider's control panel or ask support. The PTR hostname should match your server's hostname (mail.example.com), and that hostname's A record should point back to the same IP. This forward-confirmed reverse DNS match is one of the first things receiving servers check.

About SPF: Start with ~all (soft fail) while you test, then change to -all (hard fail) once you confirm that legitimate mail passes.

IPv6 note: If you do not have complete IPv6 setup (AAAA, PTR and SPF coverage), send over IPv4 only. Our guide to IPv4 vs IPv6 on dedicated servers explains the trade-offs.

Verify records after they propagate:

Step 3: Get a TLS Certificate

Mail clients and other servers expect encrypted connections. Use a free Let's Encrypt certificate.

Port 80 must be free during issuance. Certificates renew automatically, but Postfix and Dovecot need to reload after each renewal. Create a deploy hook:

Step 4: Install and Configure Postfix

Postfix is the mail transfer agent (MTA). It accepts mail from other servers, sends your outgoing mail and hands authenticated submissions from your mail clients to the queue.

When prompted, choose Internet Site and enter example.com as the system mail name.

Apply the core settings:

Add TLS settings:

may means opportunistic encryption: Postfix uses TLS when the other server supports it, which is the right default for server-to-server mail.

Prevent open relay. An open relay lets anyone send spam through your server and is the fastest way to get blocklisted:

Connect Postfix to Dovecot for authentication:

Enable the submission service (port 587) for authenticated mail clients. Edit /etc/postfix/master.cf, find the submission lines (commented out by default) and make them look like this:

This forces TLS and requires a login on port 587, so only your users can send through it.

Step 5: Install and Configure Dovecot (IMAP)

Dovecot stores mailboxes and lets mail clients read mail over IMAP.

Mailbox location. In /etc/dovecot/conf.d/10-mail.conf:

Authentication. In /etc/dovecot/conf.d/10-auth.conf:

TLS. In /etc/dovecot/conf.d/10-ssl.conf:

Let Postfix use Dovecot for logins. In /etc/dovecot/conf.d/10-master.conf, find the service auth block and add the Postfix socket:

Restart both services and create a first mailbox user:

This guide uses system accounts, which is the simplest way to start. For many domains or hundreds of mailboxes, plan a virtual mailbox setup (a database or file-based user map) instead.

Step 6: Set Up DKIM Signing

DKIM adds a cryptographic signature to every outgoing message so receivers can verify it really came from your domain and was not altered.

Edit /etc/opendkim.conf. Comment out any existing Socket line, then set:

Create the three supporting files:

Connect Postfix to OpenDKIM:

Now publish the public key. Print it:

Create a TXT record named mail._domainkey.example.com with the value shown inside the quotes (join the pieces into one string if it is split). When DNS propagates, test the key:

A result of key OK means DKIM is working.

Step 7: Test Sending, Receiving and Authentication

1. Send a test message

Send a test message from the server (install mailutils if the mail command is missing):

2. Read the headers

In Gmail, open the message and choose Show original. You want to see SPF: PASS, DKIM: PASS and DMARC: PASS.

3. Check for open relay

From a different machine, try to send mail through your server to an outside address without logging in. It must be rejected.

4. Score your setup

Send a message to a deliverability testing service such as mail-tester.com and review the report for missing or failing checks.

5. Check the connection

A valid certificate for mail.example.com should appear.

6. Watch the logs

How to Keep Your IP Reputation Clean

A working mail server is only half of the job. Mailbox providers such as Gmail, Outlook and Yahoo decide whether to trust you based on your IP address, your domain, your authentication and how recipients react. These practices protect that trust.

1. Start with a clean IP

Before you send anything, check whether your IP appears on a blocklist using a lookup tool such as MXToolbox or Spamhaus. If your IP is already listed, ask your provider for a different one. A previous tenant's history becomes yours.

2. Get authentication and DNS exactly right

• PTR and hostname match: the IP resolves to mail.example.com, and that name resolves back to the IP.

• SPF, DKIM and DMARC all pass, and the domains align with your visible From address.

• Consistent HELO name: Postfix announces your myhostname, which should match the PTR.

Large providers now require authentication, and bulk senders face stricter rules, including easy unsubscribe and low spam-complaint rates. Check Google's and Microsoft's current sender requirements before you send at volume, because thresholds change.

3. Warm up the IP gradually

A new IP has no history, and a sudden burst of mail looks suspicious. Increase volume slowly and send first to engaged recipients. A typical approach:

These are guidelines, not provider rules. If you see deferrals or spam-folder placement, slow down.

4. Send only wanted mail

• Use confirmed opt-in lists

• Remove hard bounces immediately

• Remove recipients who never engage

• Include an unsubscribe link in bulk mail

• Never buy or scrape lists

Spam complaints damage reputation faster than any technical mistake.

5. Separate different kinds of mail

Keep transactional mail (receipts, password resets) apart from marketing or bulk mail, ideally on separate IPs or at least separate subdomains. A complaint wave on a newsletter should never block your password-reset emails.

6. Limit your sending rate

Throttling prevents burst sending and reduces the damage if an account is compromised:

Tune these to match your actual volume.

7. Protect the server and the accounts

Compromised mailboxes are a leading cause of blocklisting.

• Use strong passwords and consider app-specific passwords or key-based access

• Install Fail2ban with Postfix and Dovecot jails to block password-guessing attacks

• Keep Postfix, Dovecot and the OS patched

• Never leave the server as an open relay

8. Monitor reputation signals

• Google Postmaster Tools: domain and IP reputation, spam rate, authentication results

• Microsoft SNDS and JMRP: IP data and complaint feedback for Outlook.com recipients

• DMARC reports: the rua address in your DMARC record receives aggregate reports showing who sends mail as your domain

• Blocklist checks: run regular lookups on your IP and domain

• Queue and bounce review: postqueue -p shows stuck mail, and bounce messages tell you which providers reject you and why

9. Tighten DMARC gradually

Start with p=none to collect reports and find legitimate senders you forgot about. When reports show only authorized mail passing, move to p=quarantine, then to p=reject. This protects your domain from spoofing and strengthens your reputation.

Maintenance Checklist

For monitoring, our Prometheus and Grafana tutorial shows how to graph server health. For backups, see our dedicated server backup guide.

Troubleshooting Common Problems

Conclusion

Running a mail server on a dedicated server is practical when you take the DNS and reputation side as seriously as the software. Install Postfix and Dovecot, sign your mail with DKIM, publish correct SPF and DMARC records, match your reverse DNS to your hostname, and then protect your IP by sending only wanted mail at a steady, gradually increasing pace.

Ready to set it up? Explore KW Servers dedicated servers with dedicated IPv4 addresses and the control you need for a reliable mail server, or continue with our cPanel server management guide if you prefer a control-panel-based email setup.

This tutorial is reviewed and updated as mail software versions and mailbox provider requirements change. Commands target Ubuntu 24.04 LTS; verify package versions and sender requirements before deploying to production.