Introducing davmail

or how to handle Uni central Exchange email

Aim, Steps
  1. All email must be handled on Exchange. All Maths email is to be forwarded to Exchange: not forwarded elsewhere, not kept at Maths.
  2. You learn to use your Exchange email service. You can keep using your existing email client or setup, but need to do this without forwarding: see Setup instructions below.
    Before you change your email client settings, log in to OWA (the webmail interface to Exchange at https://webmail.sydney.edu.au/) and ensure it does not have gazillion old emails in the Inbox, or your mail client would slurp them all in.
  3. You ensure that Exchange does not forward to Maths: ask the ICT HelpDesk to remove any forwarding.
  4. Then Maths is set to forward to Exchange: ask Paul to do.

Contents

Background

Please refer to Andrew Mathas's scnews item http://www.maths.usyd.edu.au/s/lscnitm/mathas-ICTAuditOfSchool noting (elsewhere) that the migration of staff to sydney.edu.au accounts is a nonnegotiable agreement with the Dean; also refer to the Dean's email to all staff on 11Apr2017.
Questions about when, why or whether, please direct to Andrew Mathas.
Questions about how, please send to Paul.

Our School will force forwarding of all email to @sydney addresses (the Exchange server); you must learn to use your Exchange email. We will need to set forwarding from Maths to @sydney addresses for everyone, ensure everyone only uses their Exchange accounts. All people with @sydney addresses are affected, who forward their Maths email to any non-Uni places and those whose Maths email "stays here".

Then ICT may also wish to migrate your email account: explain the use or benefits of Outlook or OWA and set you up with those, maybe grab all your existing/old mail and put into Exchange (whether from Maths or your laptop or even from gmail), help you to maybe set up your mobile devices (phone or tablet); or, maybe, they just leave you alone if you are already using Exchange successfully.
I do not quite know why ICT is involved, why they were contacted at all.
Should you want or need ICT help to migrate, then please let either Andrew or Paul know so we can organize.

You may use Exchange via an Outlook client (the "ICT preferred" mail software); or via OWA (Outlook Web Access, the Exchange web interface) at
https://webmail.sydney.edu.au/
or from "mobile devices" (phones, tablets).
For details or instructions on any of the above, go to
http://staff.ask.sydney.edu.au
and search for "email iphone" or similar.
Currently you may ask the ICT HelpDesk to set redirect forwarding from Exchange to any other mail services e.g. gmail, or can set it yourself:

but that is "not our problem", and is not recommended.

You may wish to use some other client, e.g. Thunderbird, Apple Mail, or even your gmail web interface, without any forwarding; this is supported by ICT, search staff.ask for IMAP to see.
However that is somewhat fallacious:

so in effect your client may not succeed in using those services.
Up until recently, SMTP was only accessible from Uni networks, fixed in Mar2017.
The fear of POP is related to "records management": though they strongly advise to select "keep on server", and IMAP or OWA users can delete just as well. There are no warnings about saving "sent" mail: it is a function of the email client, can be turned off in Outlook also.

You may use your favourite email client with our davmail server, instead.

Our davmail server

Our davmail server is host name

davmail.maths.usyd.edu.au
and it supports/accepts:
POP (pop3s) on port 995
IMAP (imaps) on port 993
SMTP (smtps) on either port 465 or port 587
with SSL/TLS encryption and "normal password" authentication, with the unikey (not mcs\unikey) name and password. Davmail can be used for any "IMAP" services e.g. Thunderbird or Apple Mail, or for the gmail web interface, from anywhere.

Our davmail server also supports/accepts:

LDAP on port 389
CalDAV on port 1080
without SSL/TLS (no encryption).

Our davmail server uses http://davmail.sourceforge.net/ software. The server accepts POP/IMAP/SMTP connections, and "translates" the requests into OWA (Exchange webmail) access: provides standard interfaces, using only supported OWA access to Exchange mail. Needed only because of the mentioned inadequacies of ICT IMAP services. Runs on a "virtual machine" using just some idle CPU cycles, for zero cost. This service might be used by the whole Uni community, not just Maths; not sure whether we could handle the network bandwidth if it became popular. Laptop users might instead run davmail themselves, locally.

Our davmail talks to the Uni Exchange server at webmail.sydney.edu.au, as set within its configuration; that choice is not part of the "conversation" with the client. Our davmail cannot be used to access any other Exchange servers; to access another, a different davmail service would need to be set up.

Davmail SMTP effectively sends via OWA, and that does not keep an original "Date:" header, but replaces it with UTC timezone and at the time the message is handled by Exchange. Some other SMTP headers are also added or deleted. Send yourself a message, look at the headers in the Exchange Sent and Inbox folders, and weep.

Seems that the Online Archive of Exchange cannot be accessed via davmail either: use OWA or Outlook. Do not store old or long-term messages on Exchange, but keep in "local" folders.

Curiously and amazingly, davmail is "faster" than ICT IMAP or SMTP.

Setup instructions

Thunderbird

See also the ICT instructions for Thunderbird setup.

In your Thunderbird go to

Edit or Settings/Preferences
Account Settings
Account Actions
Add Mail account
and there set:
Name: your name
Email address: your @sydney email address
Password: (none), un-check do not "remember password"
Incoming: IMAP, davmail.maths.usyd.edu.au, 993, SSL/TLS, normal password
Outgoing: SMTP, davmail.maths.usyd.edu.au, 465, SSL/TLS, normal password
Username: (both Incoming and Outgoing): your unikey
then Re-Test, Done.

To avoid duplicates in "Sent", still in

Edit or Settings/Preferences
Account Settings
your new @sydney account
Copies & Folders
Un-check (not select) the setting:
When sending messages, automatically:
    [ ] Place a copy in ... "Sent" Folder on ...
(since Exchange or davmail does pretty much the same anyway).

Click OK.

Go to Check your setup.

Apple Mail

Set things up as an Exchange2007 account.
(Seems cannot use IMAP or davmail... does not need to.)

In your Apple Mail go to

Preferences
Accounts
Add (the "+" sign under the list)
and there set:
Full Name: your name
Email address: your @sydney email address
Password: your matching unikey password
Continue, let it check (maybe change login name from first.surname to unikey?), then Create. Go to
Check your setup.

Gmail web interface

Currently you may set redirect forwarding from Exchange to gmail: ask the ICT HelpDesk to do, or see above; then Maths can forward to Exchange, and we are "all done". But this is not recommended: you may be "found out" because your replies will be seen to come from gmail, or by checking settings on Exchange.

On the gmail web interface, go to

Settings
Accounts and Import
Check mail from other accounts / Add a mail account
and add your "central" mail account via our davmail server:
your @sydney.edu.au email address
(choose Import emails ... POP3)
Username: your unikey (not your email), use matching password
change POP server to davmail.maths.usyd.edu.au on port 995
select "leave copy on server" (so the Uni keeps backing up your mail)
select "always use SSL" (leave selected).
Say "yes" to send mail as this new account, or in
Settings
Accounts and Import
Send mail as / Add another email address
un-select "treat as alias", then set:
SMTP server davmail.maths.usyd.edu.au on port 587
Username: your unikey (not your email), use matching password
select to use SSL (not TLS, not sure why TLS does not work)
then wait for the verification code to arrive in your email, add it.
Maybe also choose "Reply from the same address the message was sent to".

This setting "gives away" your unikey password to your email service. Not an issue if you trust them. Probably your laptop and phone also "remember" this password, already.

Go to Check your setup.

Mutt

Seems that in your ~/.mutt/muttrc file, you need to add lines like
(example for Paul Szabo, unikey psza5678, address paul.szabo@sydney.edu.au):
# IMAP settings
set imap_user = "psza5678"
set spoolfile = imaps://davmail.maths.usyd.edu.au:993/INBOX
set folder = "imaps://davmail.maths.usyd.edu.au:993/"
set imap_keepalive = 30
# SMTP settings
set smtp_url = "smtps://$imap_user@davmail.maths.usyd.edu.au:465"
set ssl_force_tls = yes
# davmail only accepts from the "right" sender
set realname = "Paul Szabo"
set from = "paul.szabo@sydney.edu.au"
set use_from = yes
set use_envelope_from = yes
# other settings
set header_cache = ~/.mutt/cache/headers
set message_cachedir = ~/.mutt/cache/bodies
set certificate_file = ~/.mutt/certificates
# keep cache clean
set message_cache_clean = yes

Go to Check your setup.

Alpine

See also the ICT instructions for Alpine setup.

Seems that in your ~/.pinerc file, you need to add lines like
(example for Paul Szabo, unikey psza5678, address paul.szabo@sydney.edu.au):

inbox-path={davmail.maths.usyd.edu.au:993/ssl/novalidate-cert/user=psza5678}inbox
smtp-server=davmail.maths.usyd.edu.au:587/ssl/novalidate-cert/user=psza5678
customized-hdrs=From: Paul Szabo <paul.szabo@sydney.edu.au>
disable-these-authenticators=PLAIN
In the alpine SETUP Config menu, you need to enable Expose Hidden Config (then exit and re-enter config) to set Disable These Authenticators.

One problem may(?) remain: a message sent by alpine then shown by it, may say:

[ The following text contains the unknown encoding type ]
[ "X-UNKNOWN". ]
[ Some or all of the text may be displayed incorrectly. ]
I do not know what causes this.

Go to Check your setup.

Other mail clients

Seems the gmail app on phones can use your Exchange account to be added, more directly (or it could use IMAP). That alone would be enough if you only ever used that gmail app; not sure whether necessary (or would cause duplicates) once you have set gmail via the web interface; I did not yet test the phone app.

Other mail services may have "add account" features (similar to gmail). Succeeded on mail.com (its "mail collector" using IMAP to webmail.sydney on port 993, it could also send email as if it was from @sydney, no davmail at all).

Any other clients or any problems, please ask Paul.

Check your setup

After setting up your email client, check that email reception works: log in to OWA, copy some message into your Inbox, see it appear in your mail client.
Beware: gmail does not like duplicates, it may not show messages it has already seen (sent or forwarded). On OWA send a message to some new or non-existent user, copy that Sent or Draft into your Inbox.

After you are comfortable accessing your Exchange email, forwarding from Exchange to Maths should be turned off, no longer needed; please ask the ICT HelpDesk to do. We can then re-jiggle Maths forwarding to go to Exchange. Please let Paul know when Maths forwarding can be changed.

Notes, blurb

Substance is left unchanged but perception is OK with above changes.

Beware of Exchange archive settings. They move messages older than a year into some "Online Archive" that you can access with Outlook or OWA, but not with IMAP or Apple Mail; then your email client would not see them anymore. Do not store old or long-term messages on Exchange, but keep in "local" folders. With IMAP you can copy messages (in either direction) between the Exchange server and other folders: try to take advantage of the unlimited storage offered by Exchange.

Beware of unikey password changes. Currently there is an enforced yearly change, and if you change then you need to re-do the settings in gmail. (Or if you forget, then you may end up with your account locked after too many bad tries.) Best to leave your unikey password as it was: go through 5 or 10 changes, then back.

If your email account is a "student" email on @uni.sydney.edu.au (that you can access at https://outlook.office365.com/ outsourced to Office365), then you may set your mail client to get that (directly without davmail) as per POP and IMAP settings for Office 365 so using settings:

IMAP outlook.office365.com 993
POP outlook.office365.com 995
SMTP smtp.office365.com 587
(or you could set forwarding).

The justification in Andrew's scnews is wrong. The real reason is that ... the migration of staff to sydney.edu.au accounts is a nonnegotiable agreement with the Dean.
The Dean's reasons are inscrutable. Cannot be "confidentiality of data" as that would only require a ban of forwarding to external services (and anyway ICT does not do that); cannot be "record-keeping" as a simple forward of each sent and received email to some recorder@sydney address would suffice (and anyway ICT does not do that, they rely on backups of the email store and on the email client to save sent mail in the first place, so may have no record if you delete your sent or received message before the monthly?! backup).
A rule that emails must not be handled on School systems is wrong: emails are handled on laptops and the Maths machines are not lesser. When sending email from Maths machines (with "local" clients), they were already sent with your "branded" @sydney.edu.au address; not exactly from "central" services but it is unlikely anyone would notice they were not sent from Exchange.

In email sent to gmail etc users on 14Mar2017, Andrew said: By ringing the ICT help desk ... you can have your exchange account forward your email to an external email address. Why don't we just arrange so for all users, without any annoyance? If Exchange is permitted to forward to gmail, why cannot we do same, and why cannot Exchange forward to Maths?
This is contradicted by email sent by the Dean to all staff on 11Apr2017: you should not forward emails related to your work at the University to private email accounts, and indeed such action is considered a breach of the University's Privacy Policy.
(Such prohibition should be Uni-wide not just Science; as yet ICT does not have bans on forwarding.)
The Dean does not give any reasons why School servers could not be used to send or receive email.
Some justification in the Dean's email seems wrong: he is concerned about your email provider being compromised, but not about the arguably greater likelihood of your laptop or phone being hacked or stolen. His concern about loss of data is plain wrong, a redirect from Exchange keeps copy of messages.
The Dean does not provide pointers: see the Privacy Policy 2013 and the Privacy Management Plan 2013.
Curious to see in the PMP: Personal information must not be stored in any "cloud computing" solution: seems to ban Cloudstor and Office365.

In email sent to @uni.sydney users on 14Mar2017, Andrew said: ... your student account with a uni.sydney.edu.au address, which is part of the exchange system .... That is wrong: @uni.sydney is outsourced to Office365, Exchange is a service within Uni. Normally for postgrads, Exchange accounts are set with a forward to the @uni.sydney address. Beware that the @sydney address will be deleted after graduation, whereas the @uni.sydney account is for life. (Note: email sent from OWA to postgrads may bounce with some bogus error.)

We could find who the gmail users were, only because we still have our email server. Wonder if Exchange admins could extract such info, and whether @uni.sydney people would be willing to. With the above changes, we will never be able to tell who uses gmail. (Unless we hack the davmail server to log some un-crypted traffic...)

One advantage of doing things without forwarding, other than to be seen to be complying, is to keep work and private emails separate. I was told of a case where a person's email was subpoenaed (by HR of his institution) and he had to hand over all private emails also. Of course this does not help when the one-and-only email address you use is your work email.

The Uni wants to store data only on servers under trusted jurisdictions, and gmail/google has servers in some Asian countries. The Uni trusts Microsoft (@uni.sydney is really Office365, and soon @sydney will also be), Mimecast (our spam filter), trusted Symantec (previous spam filter), and say Cloudstor; so far the Uni does not seem to worry about Dropbox, not even Google Drive. There is a push to have mobile devices (their data, and the passwords they remember) encrypted but that does not seem monitored or enforced.

Beware also of the Uni Mimecast spam filter, noting that all messages sent or received by Exchange go through it. The Mimecast filter is known to sometimes alter the encoding of messages: dislikes "Content-Transfer-Encoding: 7bit" or 8bit, prefers quoted-printable, may leave base64; specifically for "Content-Type: multipart/xxx" messages, does not put in any (useless?) Content-Transfer-Encoding lines into the email header; and corrects the capitalization of those headers. These actions usually wreck the DKIM signatures, and may result in messages being rejected.

Does not belong here at all... but curious: the Uni knew that two-factor authentication and complex passwords are good for security, and says that a sufficiently strong password has a minimum length of 13 ... or ... 10 characters but still Unikey passwords were restricted to exactly 8 characters, limit lifted from 31May2017, see Staff News.

Apologies for the verbiage.


Paul Szabo psz@maths.usyd.edu.au 9 Aug 17