Latest revision as of 08:14, 25 November 2024

Email services components

Originally there was only one "email" protocol: SMTP. It was designed to address, route, and receive messages for a destination user on a destination server. Later an extra couple of protocols (POP3 and IMAP) were developed so that people wouldn't have to read their email on the command-line on the SMTP server. Now there are many components, making a powerful, reliable, but complex system. Here's a simplified diagram of what typically happens when you send an email to someone else:

By the end of this course you should be familiar enough with all these components to be able to speak about them comfortably.

We'll set up an SMTP server in this lab and an IMAP server in the next lab.

New VM: email

Create a new VM, similar to the www VM you created earlier. Make sure that it:
- Named email
- AMI version Amazon Linux 2023
- Is in vpc-ops345 (where no public IP address should be assigned by default).
- In the subnet subnet-ops345
- Is in a new security group named ops345emailsg, which will allow SSH acccess from the router only and SMTP access from anywhere.
- Has the IP address 10.3.45.12
- Gets an extra 4GB EBS volume.
After it starts: do some configuration you should be very comfortable with by now:
- Set up port forwarding on the router so you can SSH to email via port 2212 on the router. The email server is going to have its own public IP, but remember we're pretending that we're monitoring all SSH traffic using special software on the router machine.
- Create a new user and delete ec2-user.
- Set the hostname to email.yoursenecaid.ops345.ca
- Allocate a new, permanent static IP named email_public_ip and associate it with your email server.

DNS records

Email used as you're used to it requires a couple of DNS entries. One A record for the server, and on MX record for your domain.

Log in to your Bindistrar.
Create an A record for email.yoursenecaid.ops345.ca with the static IP you assigned to your email server. This is just a typical A record.
Create an MX record with the value email.yoursenecaid.ops345.ca.
- This record is for other people's email servers who want to send email to anyone@yoursenecaid.ops345.ca
- It specifies that the receiving SMTP server for yoursenecaid.ops345.ca is email.yoursenecaid.ops345.ca
- The priority is there for when you have one or more backup email servers. You would if this email server were important, but in this course you don't so you can pick any number for the priority.

Remember that DNS records are cached, so if you make a mistake: you'll need for the TTL to expire before you can test your updated records. One way too give yourself two chances to get it right is to first test your MX record on the router. Then if it's wrong: you can test your second attempt on your workstation before the TTL expires on the router's caching DNS server.

Test your MX record on the router:

Storage

For the www VM we have set up an external storage device so that we wouldn't lose our website if the server was damaged or deleted. We'll do the same for the email VM. You can look back at lab 3 if you need a reminder for how to do the following steps.

Create a volume group vg_email with the second drive as the only physical volume.
Create a logical volume lv_email

Don't get locked out
With the setup we're building all the emails will be stored in each user's home directory. That means lv_email will need to be mounted on /home. That makes it tricky because other important things (like your SSH keys) are stored under /home as well, so be careful with the following steps, you may end up locking yourself out of your VM, and you'll have to rebuild it.

Create a temporary directory /tmp/home and mount lv_email there.
Move all the contents of /home to /tmp/home
Edit your fstab to make sure lv_email gets mounted automatically on /home when your system boots.
Hold your breath and reboot. If you did it right: you should be able to ssh back in and everything will look the same, except now there's a lost+found directory in /home, which tells you that there's a filesystem mounted on there.

SMTP server: Postfix

There are many SMTP server software on Linux to choose from. Postfix is the most used commonly used so that's the one we're going to use. It can be configured to do all sorts of advanced and interesting things, but we'll keep it simple. Once you feel less overwhelmed with all the exotic configuration options: it will help you to read the description of what those options do, over time the meaning of those comments will become more and more clear.

Postfix is not installed by default on Amazon Linux 2023, install it using yum.
Then start it and make sure it starts automatically on boot.

In the default configuration Postfix only listens on localhost, it will not accept inbound connections to port 25 even if the firewall allows access to that port.

Use the ss command to see what ports are open (i.e. there is a service listening on that port):

Edit /etc/postfix/main.cf and change four of the options:
- Set mydomain to yoursenecaid.ops345.ca
- Set myorigin to $mydomain. This is appended to your username when you send email directly from the server. It doesn't have to be the same as $mydomain, but it makes sense for it to be the same in most cases.
- Set inet_interfaces to all. This will configure Postfix to listen for external incomming connections, rather than just connections from localhost.
- Change mydestination to include $mydomain. This will tell postfix that when it sees an email for anyone@yoursenecaid.ops345.ca: it should deliver it rather than forward it to a different SMTP server.
Restart the Postfix service. Check the service status, and the postfix logs using journalctl -u postfix -e to confirm that it restarted successfully.
Run ss again, learn to see the difference between a service that listens on localhost and a service that listens on all interfaces:

Test receiving email

People often think of SMTP servers as "sending" servers, but they are responsible for receiving email too. Since setting up a full email client involves many steps, and at least two services: we'll test access to the SMTP server manually.

See if you can connect from your workstation to your SMTP server using telnet. If the connection is successful: run a single SMTP protocol command just because you can:

Lame ISPs and VPNs
I used to only know one ISP lame enough to block outgoing connections to port 25 for their customers (Rogers). But that plague is spreading. Some students tell me Bell has the same problem, and I've even experienced this block using a VPN. If you're having a problem establishing the following connection: try connecting to Seneca's SMTP server first. If that doesn't work: just skip this test.

ISP	Works?	Date tested
SenecaNET	Sometimes	2024-11
TekSavvy DSL	Yes	2024-10
TekSavvy Fibre	Yes	2024-11
Bell DSL/Fibre	No	2024-11
Rogers Cable	No	2024-11
Koodo	Yes	2024-11
Freedom Mobile	Yes	2024-11
Fido	Yes	2024-11
Bell Mobility	No	2024-11

You can use the quit command or the escape sequence ^] to get out of that session.
Now use whatever email service you want (e.g. myseneca, or your personal account) to send an email to yourusername@yourusername.ops345.ca. Note that you don't need any special configuration on the sender's email client or server. All someone needs to know to send email to you is in your server's MX record.
- Before you send it: run journalctl -u postfix -f in a terminal so you can see what happens when a message arrives. You can press enter a few times to insert some blank lines so you can more easily separate unrelated messages.
- There's a lot of log for a single email. I typically look for from, to, and status=sent. If the status is something different: I start looking at other information.
"Delivered to mailbox" means that Postfix decided it is the destination for this particular message and saves it where it was configured to save it.

Use Maildir and dovecot-lda

By default Postfix saves emails in the file /var/mail/username in the MBOX format. That means all the emails you get will be appended to this file. That works fine under normal circumstances but most people don't use MBOX for serious email servers. We'll configure Postfix to save messages in the Maildir format, under each user's home directory.

We'll use dovecot-lda as a LDA. It's an application that comes with the Dovecot package.

Install dovecot, but don't start or enable the service. Leave the Dovecot configuration for the next lab.
Edit /etc/postfix/main.cf and add the following mailbox_command, to tell Postfix to deliver using dovecot-lda:
```
mailbox_command = /usr/libexec/dovecot/dovecot-lda -f "$SENDER" -a "$RECIPIENT"
```
Restart postfix.
Edit /etc/dovecot/conf.d/10-mail.conf and add the following mail_location parameter, to tell dovecot-lda where to store emails it is asked to store and in which format:
```
mail_location = maildir:~/Maildir
```
No need to restart any service after changing that, dovecot-lda is executed as an application for every email that Postfix wants delivered.
Send yourself another email and find where it ended up. The maildir format isn't meant for reading on the command-line, but you should be able to find a newly received message so that you can troubleshoot when it's not working properly.

Aliases

Sometimes you want to receive emails which are sent to different addressed. For example the system administrator typically receives emails sent to postmaster, webmaster, hostmaster, and root. Also dovecot-lda will not deliver email to root at all as a security measure.

Create an alias to deliver emails meant for root to andrew instead:

Edit /etc/aliases (it's a text file).
Uncomment the bottom line and change "mark" to your username.
Run "newaliases".
Send an email to root@yourusername.ops345.ca and make sure it ends up in your user's Maildir

Submit evidence of your work

For this lab, please submit screenshots that show you've completed the work, unless your professor has given you different instructions. As a minimum that's:

Your email VM details in the AWS console.
A dig query for your two new DNS records.
The output of the mount command.
The received emails in your Maildir.

@@ Line 1: / Line 1: @@
-= THIS PAGE IS A DRAFT, NOT READY FOR USE YET =
+= Email services components =
+Originally there was only one "email" protocol: SMTP. It was designed to address, route, and receive messages for a destination user on a destination server. Later an extra couple of protocols (POP3 and IMAP) were developed so that people wouldn't have to read their email on the command-line on the SMTP server. Now there are many components, making a powerful, reliable, but complex system. Here's a simplified diagram of what typically happens when you send an email to someone else:
-= SMTP Encryption =
+[[File:Email-servers.png|800px|border|center]]
+By the end of this course you should be familiar enough with all these components to be able to speak about them comfortably.
+We'll set up an SMTP server in this lab and an IMAP server in the next lab.
+= New VM: email =
+* Create a new VM, similar to the www VM you created earlier. Make sure that it:
+** Named '''email'''
+** AMI version '''Amazon Linux 2023'''
+** Is in '''vpc-ops345''' (where no public IP address should be assigned by default).
+** In the subnet '''subnet-ops345'''
+** Is in a new security group named '''ops345emailsg''', which will allow SSH acccess from the router only and SMTP access from anywhere.
+** Has the IP address '''10.3.45.12'''
+** Gets an extra '''4GB EBS''' volume.
+* After it starts: do some configuration you should be very comfortable with by now:
+** Set up port forwarding on the router so you can SSH to email via port 2212 on the router. The email server is going to have its own public IP, but remember we're pretending that we're monitoring all SSH traffic using special software on the router machine.
+** Create a new user and delete ec2-user.
+** Set the hostname to '''email.yoursenecaid.ops345.ca'''
+** Allocate a new, permanent static IP named '''email_public_ip''' and associate it with your email server.
+== DNS records ==
+Email used as you're used to it requires a couple of DNS entries. One A record for the server, and on MX record for your domain.
+* Log in to your [http://ops345.ca Bindistrar].
+* Create an A record for email.yoursenecaid.ops345.ca with the static IP you assigned to your email server. This is just a typical A record.
+* Create an MX record with the value email.yoursenecaid.ops345.ca.
+** This record is for other people's email servers who want to send email to anyone@yoursenecaid.ops345.ca
+** It specifies that the receiving SMTP server for yoursenecaid.ops345.ca is email.yoursenecaid.ops345.ca
+** The priority is there for when you have one or more backup email servers. You would if this email server were important, but in this course you don't so you can pick any number for the priority.
+Remember that DNS records are cached, so if you make a mistake: you'll need for the TTL to expire before you can test your updated records. One way too give yourself two chances to get it right is to first test your MX record on the router. Then if it's wrong: you can test your second attempt on your workstation before the TTL expires on the router's caching DNS server.
+* Test your MX record on the router:
+[[File:AWSDigEmail.png|border|center]]
+== Storage ==
+For the www VM we have set up an external storage device so that we wouldn't lose our website if the server was damaged or deleted. We'll do the same for the email VM. You can look back at lab 3 if you need a reminder for how to do the following steps.
+* Create a volume group '''vg_email''' with the second drive as the only physical volume.
+* Create a logical volume '''lv_email'''
+{{Admon/important|Don't get locked out|With the setup we're building all the emails will be stored in each user's home directory. That means '''lv_email''' will need to be mounted on '''/home'''. That makes it tricky because other important things (like your SSH keys) are stored under /home as well, so be careful with the following steps, you may end up locking yourself out of your VM, and you'll have to rebuild it.}}
-One thing we haven't spent time on last week is the connection from your email client to your email server. This will very often be on a very untrusted network, such as a free wifi network you found in some random place.
+* Create a temporary directory /tmp/home and mount lv_email there.
+* Move all the contents of /home to /tmp/home
+* Edit your fstab to make sure lv_email gets mounted automatically on /home when your system boots.
+* Hold your breath and reboot. If you did it right: you should be able to ssh back in and everything will look the same, except now there's a lost+found directory in /home, which tells you that there's a filesystem mounted on there.
-In this lab we'll set up your MTA to accept encrypted incomming connections, so that it won't matter what sort of network your workstation computer or phone is connected to.
+== SMTP server: Postfix ==
-[[File:Email-servers.png|800px|border|center]]
+There are many SMTP server software on Linux to choose from. Postfix is the most used commonly used so that's the one we're going to use. It can be configured to do all sorts of advanced and interesting things, but we'll keep it simple. Once you feel less overwhelmed with all the exotic configuration options: it will help you to read the description of what those options do, over time the meaning of those comments will become more and more clear.
+* Postfix is not installed by default on Amazon Linux 2023, install it using yum.
+* Then start it and make sure it starts automatically on boot.
+In the default configuration Postfix only listens on localhost, it will not accept inbound connections to port 25 even if the firewall allows access to that port.
+* Use the '''ss''' command to see what ports are open (i.e. there is a service listening on that port):
+[[File:AWSssSMTPlocalhost.png|800px|border|center]]
+* Edit '''/etc/postfix/main.cf''' and change four of the options:
+** Set '''mydomain''' to yoursenecaid.ops345.ca
+** Set '''myorigin''' to $mydomain. This is appended to your username when you send email directly from the server. It doesn't have to be the same as $mydomain, but it makes sense for it to be the same in most cases.
+** Set '''inet_interfaces''' to all. This will configure Postfix to listen for external incomming connections, rather than just connections from localhost.
+** Change '''mydestination''' to include $mydomain. This will tell postfix that when it sees an email for anyone@yoursenecaid.ops345.ca: it should deliver it rather than forward it to a different SMTP server.
+* Restart the Postfix service. Check the service status, and the postfix logs using '''journalctl -u postfix -e''' to confirm that it restarted successfully.
+* Run '''ss''' again, learn to see the difference between a service that listens on localhost and a service that listens on all interfaces:
+[[File:AWSSsSMTPOpen.png|800px|border|center]]
+=== Test receiving email ===
+People often think of SMTP servers as "sending" servers, but they are responsible for receiving email too. Since setting up a full email client involves many steps, and at least two services: we'll test access to the SMTP server manually.
+* See if you can connect from your workstation to your SMTP server using telnet. If the connection is successful: run a single SMTP protocol command just because you can:
+{{Admon/important|Lame ISPs and VPNs|I used to only know one ISP lame enough to block outgoing connections to port 25 for their customers (Rogers). But that plague is spreading. Some students tell me Bell has the same problem, and I've even experienced this block using a VPN. If you're having a problem establishing the following connection: try connecting to Seneca's SMTP server first. If that doesn't work: just skip this test.}}
+{| class="wikitable"
+!ISP
+!Works?
+!Date tested
+|-
+|SenecaNET
+|Sometimes
+|2024-11
+|-
+|TekSavvy DSL
+|Yes
+|2024-10
+|-
+|TekSavvy Fibre
+|Yes
+|2024-11
+|-
+|Bell DSL/Fibre
+|No
+|2024-11
+|-
+|Rogers Cable
+|No
+|2024-11
+|-
+|Koodo
+|Yes
+|2024-11
+|-
+|Freedom Mobile
+|Yes
+|2024-11
+|-
+|Fido
+|Yes
+|2024-11
+|-
+|Bell Mobility
+|No
+|2024-11
+|}
+[[File:AWSTelnet25-2023.png|border|center]]
+* You can use the '''quit''' command or the escape sequence ^] to get out of that session.
+* Now use whatever email service you want (e.g. myseneca, or your personal account) to send an email to yourusername@yourusername.ops345.ca. Note that you don't need any special configuration on the sender's email client or server. All someone needs to know to send email to you is in your server's MX record.
+** Before you send it: run '''journalctl -u postfix -f''' in a terminal so you can see what happens when a message arrives. You can press enter a few times to insert some blank lines so you can more easily separate unrelated messages.[[File:AWSMaillogReceivedMail-2023.png|800px|border|center]]
+** There's a lot of log for a single email. I typically look for '''from''', '''to''', and '''status=sent'''. If the status is something different: I start looking at other information.
+* "Delivered to mailbox" means that Postfix decided it is the destination for this particular message and saves it where it was configured to save it.
-Sadly today there is still no widespread means to encrypt MTA->MTA SMTP connections, which means that fundamentally you cannot trust that your email hasn't been read or modified in transit. This is a problem we can't solve in this course, but we don't need to because it's a very large, global problem. A fix would require every single email server out there to comply with a brand new specfication which hasn't even been developed yet.
+=== Use Maildir and dovecot-lda ===
-== Generate encryption keys ==
+By default Postfix saves emails in the file '''/var/mail/username''' in the MBOX format. That means all the emails you get will be appended to this file. That works fine under normal circumstances but most people don't use MBOX for serious email servers. We'll configure Postfix to save messages in the Maildir format, under each user's home directory.
-You can't use the keys you've generated in [[OPS345_Lab_5 | lab 5]] because they were tied to the hostname youruserid.ops345.ca, and the email server has a different FQDN: email.youruserid.ops345.ca.
+We'll use dovecot-lda as a LDA. It's an application that comes with the Dovecot package.
-* Use certbot again on your workstation to generate a key pair for email.youruserid.ops345.ca. Look at the lab 5 notes if you need a reminder for how to do it. But don't overwrite your web server's keys by mistake.
+* Install dovecot, but don't start or enable the service. Leave the Dovecot configuration for the next lab.
-* Copy your new private key to ~yourusername/ops345/keys/certbot/'''email.yourusername.ops345.ca.key.pem''' on the workstation, for safekeeping.
+* Edit /etc/postfix/main.cf and add the following mailbox_command, to tell Postfix to deliver using dovecot-lda:<pre>mailbox_command = /usr/libexec/dovecot/dovecot-lda -f "$SENDER" -a "$RECIPIENT"</pre>
-* Copy your new certificate to ~yourusername/ops345/keys/certbot/'''email.yourusername.ops345.ca.cert.pem''' on the workstation, for safekeeping.
+* Restart postfix.
-** Make sure the keys are owned by your regular user, not root.
+* Edit /etc/dovecot/conf.d/10-mail.conf and add the following mail_location parameter, to tell dovecot-lda where to store emails it is asked to store and in which format:<pre>mail_location = maildir:~/Maildir</pre>
-* Copy the two keys into the appropriate directories (/etc/pki/tls/certs/ and /etc/pki/tls/private/) on your email server.
+* No need to restart any service after changing that, dovecot-lda is executed as an application for every email that Postfix wants delivered.
+* Send yourself another email and find where it ended up. The maildir format isn't meant for reading on the command-line, but you should be able to find a newly received message so that you can troubleshoot when it's not working properly.
-== Configure Postfix ==
+=== Aliases ===
-As you did with Apache: you need to tell the email server (Postfix) to use your keys for encryption. There are also some other options you need to specify. None of these are in main.cf by default, so you can copy-paste them from here:
+Sometimes you want to receive emails which are sent to different addressed. For example the system administrator typically receives emails sent to postmaster, webmaster, hostmaster, and root. Also dovecot-lda will not deliver email to root at all as a security measure.
-<pre># Settings to enable secure SMTP using my key and certificate:
+Create an alias to deliver emails meant for root to andrew instead:
-smtpd_tls_auth_only = no
-smtpd_use_tls = yes
-smtp_use_tls = yes
-smtpd_tls_key_file = /etc/pki/tls/private/email.asmith15.ops345.ca.key.pem
-smtpd_tls_cert_file = /etc/pki/tls/certs/email.asmith15.ops345.ca.cert.pem
-tls_random_source = dev:/dev/urandom
-smtpd_tls_loglevel = 1
-</pre>
-* After you restart Postfix and make sure it hasn't crashed due to a typo in the config file: test it. Use telnet to connect to it and look for 250-STARTTLS in its reply to your EHLO:
+* Edit /etc/aliases (it's a text file).
-[[File:AWSPostfix250Starttls.png|border|center]]
+* Uncomment the bottom line and change "mark" to your username.
+* Run "newaliases".
+* Send an email to root@yourusername.ops345.ca and make sure it ends up in your user's Maildir
-That implies that your Postfix server is capable of setting up encrypted connections. We will test this later in the lab using Thunderbird.
+= Submit evidence of your work =
-* dovecot installed in previous lab, it needs very little configuration for our simple setup
+For this lab, please submit screenshots that show you've completed the work, unless your professor has given you different instructions. As a minimum that's:
-* /etc/dovecot/dovecot.conf:
-* Modify the protocols option so that Dovecot will work with IMAP connections, no POP3 or LMTP.
-* 10-ssl.conf:
-<source>ssl_cert = </etc/pki/tls/certs/email.asmith15.ops345.ca.cert.pem
-ssl_key = </etc/pki/tls/private/email.asmith15.ops345.ca.key.pem</source>
-* ss should show port 993, no 995 or 110
-* there's no iptables running on email, so just open port 993 in ops345sgemail
-* set up thunderbird for myseneca
-* configure thunderbird:
-** IMAP: email.asmith15.ops345.ca, SSL 993, normal password, username without @domain
-** SMTP: email.asmith15.ops345.ca, STARTTLS 25, normal password, username without @domain
-* tail -f /var/log/maillog and send a message from thunderbird
-* The message is not sent, because postfix is not configured to authenticate users. encryption alone is not enough.
-* https://www.xmodulo.com/enable-user-authentication-postfix-smtp-server-sasl.html
-** except the SSL/TLS parameters part.
-* Send email again. Note the messages about connection timed out. We can't do anything about that in AWS Academy:
-** https://aws.amazon.com/premiumsupport/knowledge-center/ec2-port-25-throttle/
-* There's more to do anyway to set up a real sending server, as a minimum DKIM/SPF but reverse DNS for the domain also helps.
-* https://aws.amazon.com/premiumsupport/knowledge-center/route-53-reverse-dns/
-* http://littlesvr.ca/grumble/2014/01/26/setting-up-sendmail-on-a-dynamic-ip-part2-spf/
-* Eventually the email you tried to send will bounce because that's a temporary error
-*webmail
+* Your email VM details in the AWS console.
+* A dig query for your two new DNS records.
+* The output of the mount command.
+* The received emails in your Maildir.
 [[Category:OPS345]]

OPS345 Lab 7: Difference between revisions

Latest revision as of 08:14, 25 November 2024

Contents

Email services components

New VM: email

DNS records

Storage

SMTP server: Postfix

Test receiving email

Use Maildir and dovecot-lda

Aliases

Submit evidence of your work

Navigation menu

OPS345 Lab 7: Difference between revisions

Latest revision as of 08:14, 25 November 2024

Email services components

New VM: email

DNS records

Storage

SMTP server: Postfix

Test receiving email

Use Maildir and dovecot-lda

Aliases

Submit evidence of your work

Navigation menu

Search