Note about OAuth2

This page is obsolete after the "upgrade" of Office365 to OAuth2 authentication: see the current version instead.

This page retained for history only, about the initial transition to Office365.
See also the even older version about the transition to the Uni Exchange server.

 


 

Re-Introducing davmail

for Uni outsourced Office365

In June2018 our emails were migrated from the USyd internal Exchange server, to the cloud-based, outsourced Office365 service. Though Office365 supports IMAP/SMTP "natively", those are woefully slow and "clunky". Using davmail to convert IMAP/SMTP to EWS (Exchange Web Services) to that Office365 server, allows emails to be processed much faster, despite the intervening conversions and extra network traffic.

You want davmail if you use IMAP (e.g. Thunderbird) to access your email. You do not need to know about it if you use Outlook or AppleMail, or the web interface, or most mobile phone apps.

Contents

Native Office365 settings

Emails on Office365 can be accessed via its web interface, at either:
   
sydney.edu.au/office365
   sydney.edu.au/email
   http://outlook.com/owa/unisyd.onmicrosoft.com     or even
   outlook.office365.com
or by using Outlook or MacMail or many mobile phone apps (as Exchange mail).

For other email clients, Office365 supports IMAP/POP and SMTP, as per
   POP and IMAP settings for Office 365     or
   Uni sharepoint O365 FAQ
so using settings:
   proto   host   port   security
   IMAP    outlook.office365.com   993    SSL/TLS
   POP    outlook.office365.com   995    SSL/TLS
   SMTP    smtp.office365.com    587    STARTTLS
   Username   your @sydney email address
   Password   your unikey password

(or you could set forwarding).

Access to the Online Archive is possible with the web interface or Outlook, but not most other clients and not via IMAP, as per Microsoft documentation.

The IMAP/SMTP response of Office365 is woefully slow. Maybe this is as mentioned in Microsoft documentation:

  Note
  Each time a person accesses a POP-based or IMAP-based email program
  to open his or her Microsoft 365 or Office 365 email, that user will
  experience a delay of several seconds. The delay results from using a
  proxy server ...
... or may be just a sneaky way for Microsoft to promote Outlook and discourage other email clients.

Our davmail server

Our davmail server has host name

davmail.maths.usyd.edu.au
and it supports/accepts:
POP (pop3s) on port 995
IMAP (imaps) on port 993
SMTP (smtps) on either port 465 or port 587
with SSL/TLS encryption and "normal password" authentication, with the @sydney email address as username (not unikey), and unikey password. Davmail can be used for any "IMAP" services e.g. Thunderbird or Apple Mail, or for the gmail web interface, from anywhere.

Our davmail server could also support/accept:

LDAP (ldaps) on port 636
CalDAV on port 1080
both with SSL/TLS encryption, but it does not: LDAP not because it would be blocked by the ICT border router on some bogus security grounds, and CalDAV not because it does not seem needed or wanted.

Our davmail server uses http://davmail.sourceforge.net/ software. The server accepts POP/IMAP/SMTP connections, and "translates" the requests into EWS (Exchange Web Services) access: provides standard interfaces, using only supported EWS access to Office365 mail. Our server talks to the Office365 Exchange server at outlook.office365.com/EWS/Exchange.asmx, as set within its configuration; that choice is not part of the "conversation" with the client; it cannot be used to access any other Exchange servers: to access another, a different davmail service would need to be set up. It could be used "as is" for any other Office365 clients or login schemes, e.g. it should work for "student" email @uni.sydney.edu.au accounts.

Our davmail server runs on a "virtual machine" using just some idle CPU cycles, for zero cost. This service might be used by the whole Uni community (or even worldwide?), not just Maths. It would not be able to handle the network bandwidth if it became popular. Laptop users might instead run davmail themselves, locally.

Davmail SMTP effectively sends via EWS, and that does not keep an original "Date:" header, but replaces it with UTC timezone and at the time the message is handled by Office365. Some other SMTP headers are also added or deleted. Send yourself a message, then look at the headers in the Office365 Sent and Inbox folders, and weep.

The Office365 Online Archive can be accessed via davmail, at least with Thunderbird.

Curiously and amazingly, davmail is faster than Office365 IMAP or SMTP, e.g. "send" is in the blink of an eye, no 10-second wait.

Setup instructions

Thunderbird

In your Thunderbird go to

Edit or Settings/Preferences
Account Settings
Account Actions
Add Mail account
and there set:
Name: your name
Email address: your @sydney email address
Password: (none), un-check do not "remember password"
Incoming: IMAP, davmail.maths.usyd.edu.au, 993, SSL/TLS, normal password
Outgoing: SMTP, davmail.maths.usyd.edu.au, 465, SSL/TLS, normal password
Username: (both Incoming and Outgoing): your @sydney email address
then Re-Test, Done. (When asked for a password, use the matching unikey password.)

To avoid duplicates in "Sent", still in

Edit or Settings/Preferences
Account Settings
your new @sydney account
Copies & Folders
Un-check (not select) the setting:
When sending messages, automatically:
    [ ] Place a copy in ... "Sent" Folder on ...
(since Office365 or davmail does pretty much the same anyway).

Click OK.

To access the Online Archive follow the instructions (or my rip-off):

Edit or Settings/Preferences
Account Settings
your new @sydney account
Server Settings
Advanced
Set
Personal namespace: (blank)
Public (shared): /archive
and maybe un-check (not select) the setting:
[ ] Show only subscribed folders
Click OK, then re-start Thunderbird.

Go to Check your setup.

Apple Mail

You do not need (cannot use?) IMAP or davmail... so just for the record.
Set things up as an Exchange account:

In your Apple Mail go to

Preferences
Accounts
Add (the "+" sign under the list)
and there set:
Full Name: your name
Email address: your @sydney email address
Password: your matching unikey password
Continue, let it check, then Create. Go to
Check your setup.

Gmail web interface

You may (instead?) set redirect forwarding from Office365 to gmail, see set forwarding as mentioned above.

On the gmail web interface, go to

Settings
Accounts and Import
Check mail from other accounts / Add a mail account
and add your "central" mail account via our davmail server:
your @sydney.edu.au email address
(choose Import emails ... POP3)
Username: your @sydney email address, use matching unikey password
change POP server to davmail.maths.usyd.edu.au on port 995
select "leave copy on server" (so the Uni keeps backing up your mail)
select "always use SSL" (leave selected).
Say "yes" to send mail as this new account, or in
Settings
Accounts and Import
Send mail as / Add another email address
un-select "treat as alias", then set:
SMTP server davmail.maths.usyd.edu.au on port 587
Username: your @sydney email address, use matching unikey password
select to use SSL (not TLS, not sure why TLS does not work)
then wait for the verification code to arrive in your email, add it.
Maybe also choose "Reply from the same address the message was sent to".

This setting "gives away" your unikey password to your email service. Not an issue if you trust them. (Probably your laptop and phone also "remember" this password, anyway.)

Two or three Gmail oddities to BEWARE of.

Go to Check your setup.

Mutt

Seems that in your ~/.mutt/muttrc file, you need to add lines like
(example for Paul Szabo, address paul.szabo@sydney.edu.au):
# IMAP settings
set imap_user = "paul.szabo@sydney.edu.au"
set spoolfile = imaps://davmail.maths.usyd.edu.au:993/INBOX
set folder = "imaps://davmail.maths.usyd.edu.au:993/"
set imap_keepalive = 30
# SMTP settings
set smtp_url = "smtps://$imap_user@davmail.maths.usyd.edu.au:465"
set ssl_force_tls = yes
# davmail only accepts from the "right" sender
set realname = "Paul Szabo"
set from = "paul.szabo@sydney.edu.au"
set use_from = yes
set use_envelope_from = yes
# other settings
set header_cache = ~/.mutt/cache/headers
set message_cachedir = ~/.mutt/cache/bodies
set certificate_file = ~/.mutt/certificates
# keep cache clean
set message_cache_clean = yes

Go to Check your setup.

Alpine

Seems that in your ~/.pinerc file, you need to add lines like
(example for Paul Szabo, address paul.szabo@sydney.edu.au):

inbox-path={davmail.maths.usyd.edu.au:993/ssl/novalidate-cert/user=paul.szabo@sydney.edu.au}inbox
smtp-server=davmail.maths.usyd.edu.au:587/ssl/novalidate-cert/user=paul.szabo@sydney.edu.au
customized-hdrs=From: Paul Szabo <paul.szabo@sydney.edu.au>
disable-these-authenticators=PLAIN
In the alpine SETUP Config menu, you need to enable Expose Hidden Config (then exit and re-enter config) to set Disable These Authenticators.

One problem may(?) remain: a message sent by alpine then shown by it, may say:

[ The following text contains the unknown encoding type ]
[ "X-UNKNOWN". ]
[ Some or all of the text may be displayed incorrectly. ]
I do not know what causes this.

Go to Check your setup.

Other mail clients

Seems the gmail app on phones can use your Exchange account to be added, more directly (or it could use IMAP). That alone would be enough if you only ever used that gmail app; not sure whether necessary (or would cause duplicates) once you have set gmail via the web interface; I did not yet test the phone app.

Other mail services may have "add account" features (similar to gmail). Succeeded on mail.com (its "mail collector" using IMAP to webmail.sydney on port 993, it could also send email as if it was from @sydney, no davmail at all).

Any other clients or any problems, please ask Paul.

Check your setup

After setting up your email client, check that email reception works: log in to Office365, copy some message into your Inbox, see it appear in your mail client.

Notes, blurb

See also older version about the transition to the Uni Exchange server.

Some of the blurb below is non-original, been (wrongly!) updated since the change to OAuth2.

Do not store old, long-term, or important messages on Office365, but keep in "local" folders.
BEWARE that when you leave the Uni, ICT will disable your unikey and you will lose access to Uni email.
BEWARE of Office365 Online Archive settings: they move messages older than some time into some "Online Archive". You can access this with Outlook or web interface, or via davmail and with Thunderbird (but maybe not other clients?), and not with IMAP as per Microsoft documentation, and not with Apple Mail. Maybe, change the archiving policy using the web interface: right-click on (each) email folder and choose Assign Policy > Achive Policy : Personal never move to archive (Never).
BEWARE of the Outlook recall function: messages recalled and still in your Inbox (or other Office365 folders?), will disappear.
BEWARE that ICT will sometimes delete some (bad? virus?) messages from your mail folders.

BEWARE of unikey password changes. Currently there is an enforced yearly change, and if you change then you may need to re-do the settings in your mail client (gmail or thunderbird or phone etc). (Or if you forget, then you may end up with your account locked after too many bad tries.) Best to leave your unikey password as it was: go through 5 or 10 changes, then back.

Note how "student" email on @uni.sydney.edu.au is outsourced to the same Office365 cloud, though with a different login scheme.

Note that with IMAP you can copy messages (in either direction) between Office365 and other folders: try to take advantage of the unlimited storage offered by Office365.

The Uni wants to store data only on servers under trusted jurisdictions, and gmail/google has servers in some Asian countries. The Uni trusts Microsoft (both @sydney and @uni.sydney are really Office365), Mimecast (our spam filter), trusted Symantec (previous spam filter), and say Cloudstor and Dropbox; so far the Uni does not seem to worry about Google Drive. There is a push to have mobile devices (their data, and the passwords they remember) encrypted but that does not seem monitored or enforced.

BEWARE of the Uni Mimecast spam filter, noting that all @maths and @sydney messages received, and any sent by Office365, go through it.

Apologies for the verbiage.


Paul Szabo psz@maths.usyd.edu.au 17 Nov 23