expect this commit to be force-replaced periodically

[nmh] / docs / imap / DESIGN

#+TITLE: IMAP Synchronization for nmh

* Status

An EXPERIMENTAL, almost-entirely-happy-path-only partial implementation
exists.  Most of the nmh commands that are within the scope of the project
have not yet been updated.

All functionality behind --enable-experimental-imap-folder-manager
configuration flag.

* Rationale

The world is now mobile-first, and many of are are mobile-only, or nearly so.
When my smart phone took over my life, two things happened to my nmh usage:

- At work, I was frustrated constantly by messages read under nmh being unread
  on my phone and vice versa.
- In my personal life, that same frustration drove me away from nmh completely.
  Whenever I need to reply to email from my computer, I find my self
  incorporating months of messages.

We can use IMAP to make nmh a first-class citizen in the modern mail world.

* Goals

- Treat the IMAP server as the source of truth.
- Minimize startup penalty to nmh commands.
- Minimize the impact to the nmh code-base.
- Minimize round-trips to the server.
- Enhance =inc= to incorporate mail from a hierarchy of IMAP mailboxes.
- Enhance =inc= to delete mail locally that has been deleted on the server.
- Enhance all commands interpreting the user's =Unseen-Sequence= (i.e. all those
  taking message specifications (=mh-sequence(5)=), e.g. =mark=, =pick=, =scan=,
  =show=) to interpret references to the =Unseen-Sequence= as all messages not
  flagged as =\Seen= on the server.
- Enhance =scan= output to reflect live =\Seen= status when it is asked to
  display whether messages appear in the user's =Unseen-Sequence=.
- Enhance =show= to set the =\Seen= flag on messages shown.
- Enhance =mark= to set or clear the =\Seen= flag for messages removed from or
  added to the user's =Unseen-Sequence=, respectively.
- Enhance =rmm= to delete messages from the server.
- Enhance =refile= to move messages on the server.
- Enhance =comp= and =repl= to allow usage of a server-side draft folder.

** Deferred goals

- Off-line mode might be useful
- Synchronizing flags other than =\Seen=
- Storing sequences on the server (possibly the same problem as the previous)
- IMAP defines a message UID as an unsigned 32-bit integer, allowing for
  4,294,967,295 messages, but nmh uses SIGNED =int=, which is 32 bits on
  amd64, allowing for only 2,147,483,647 messages.  It's not clear any file
  system offers reasonable performance with directories of that size, so we
  may need to redesign nmh folders before the =int= becomes a problem.
  Assuming any of this is a practical concern...

* Approach

** Architecture

*** Managed folders

"Minimize startup penalty" and "Minimize round-trips" suggest we start a
separate process which can keep state between nmh commands.  Each nmh command
does not have to can pay the relatively expensive TLS connection setup cost.

"Minimize impact to the nmh code-base" means continuing to represent folders
 as directories and messages as numbered files within.

To solve those, we introduce managed folders, which are folders in the nmh
mail store managed by a folder manager.  When the user requests an operation
on a managed folder, we start by informing the folder manager of the operation
to be taken.  In some cases, the folder does nearly all the work.  In other
cases, it simply saves some information from the server where the rest of nmh
can easily find it and then gets out of the way.

For example, =scan= hands the user-specified message sequences to the folder
manager, which consults the IMAP server and stores the results in a special
sequence, and then the rest of =scan= as we all know and love it takes over,
blissfully unaware the folder is special.

*** Source of truth

"Treat the IMAP server as the source of truth" gives us a simple and
robust model.

If a user =rmm= a message, we first delete it from the server.  If that fails,
we report the failure to the user and do not proceed.  The message remains in
both the nmh folder and on the IMAP server.  If deletion from the server
succeeds, but we fail to delete locally, we still report an error, but the
message is now treated as though some other client had deleted it from the
server: we will delete it on the next =inc=.

** Language

I've programmed in C and C++ for most of my career and I've just about had it
with unsafe languages.  The thought of implementing IMAP synchronization in C
discouraged me from even starting for years.

In my last position at Google, my fellow tech lead and I would often joke
about dividing the team in two, with one team continuing to chase down a
memory corruption error we'd failed for months to find, and the other
rewriting the system in Rust.  It was a joke, but the idea stuck with me.

Since Rust supports the C ABI, we can seamlessly embed Rust into nmh programs.
Why not give it a shot?

In this case it's even easier:  write the folder manager in Rust.

* Implementation

** Managed folders

*** New profile parameters

- =Managed-Folders= is a space-separated list of folders to be managed.
  Each folder must have a corresponding =[folder]-manager= component naming the
  folder manager.
- =[folder]-manager= where =folder= is the name of the folder manager to start.
- [folder]-socket= is the path of the UNIX domain socket.

*** Starting the folder manager

1. Try to connect to running manager through =[folder]-socket=.  If that works, done!
2. If error was not ~ENOENT~, ~unlink(2)~ the stale socket.
3. ~pipe(2)~
4. ~fork~
5. child  - folder-manager; prints to stdout if successfully started (e.g. connected to IMAP)
5. parent - read from pipe to child
6. parent - if error, die
7. parent - else, time to connect again!  this time, die if error

*** Protocol

The protocol isn't the best, and we might want to replace it at some point, but
I have no plans to do that until finishing the first feature set.

Before refining this protocol, we should look into IPC.

nmh commands and the folder manager communicate over a UNIX domain socket
configured in the MH profile.  An nmh command opens a connection to the folder
manager and sends it a message encoded as a standard mail message header.
The folder manager replies in the same format.

Unlike in mail messages, these messages MUST start with a =len= component whose
value is the total length of the messgae including the =len= component itself.

**** Common messages

The following common message types are defined:

***** Folder

This message contains only a single folder name.

#+BEGIN_EXAMPLE
folder:...
#+END_EXAMPLE

***** Mark

This message contains a folder in which to operate, specifications of which
messages to pick and sequences to add to or remove from.

#+BEGIN_EXAMPLE
folder:...
msgs:...
seqs:...
#+END_EXAMPLE

***** Empty response

Many operations have nothing else to respond with but status and optional error message.

#+BEGIN_EXAMPLE
status:OK | NOTOK
error:error message present only if status:NOTOK
#+END_EXAMPLE

**** inc

=inc= sends the request to the folder manager, which opens the local database to
see which which messages exist, sends IMAP =FETCH= requests (up to
=max-fetch-messages-in-flight= at a time), stores the messages, and sends one
response back to =inc= per message, so =inc= can scan we go.

***** request

The =inc= request is =op:inc= followed by the folder message:

#+BEGIN_EXAMPLE
op:inc
folder:...
#+END_EXAMPLE

***** response

THe inc response is a series of messages.

First, one inc message response per message:

#+BEGIN_EXAMPLE
status:INC_FOLDER
folder:...
msg:N
unseen:N
#+END_EXAMPLE

Finally a status:OK empty response indicates completion.

**** mark-add / mark-del

These operations are inverse of each other, so request and response are the same.

***** request

#+BEGIN_EXAMPLE
op:mark-(add / del)
folder:...
msgs:...
seqs:...;
#+END_EXAMPLE

***** response

The empty response.

**** TODO the rest

** IMAP

*** New profile parameters

- [folder]-host, [folder]-port - to connect
- [folder]-tls TLS configuration
- [folder]-user: username
- [folder]-auth: PREAUTH | SASL - how to authenticate
- [folder]-authservice: for XOAUTH2
- [folder]-saslmech: force a SASL mechanism in case nmh chooses the wrong default
- [folder]-db: path to the database
- [folder]-mailbox-root: only look for mailboxes under this root

*** Folder store

imap-folder-manager needs to store the following per folder (mailbox):

- forward and reverse map of message UID and local message number
- mailbox UIDNEXT
- mailbox UIDVALIDITY

We store UIDNEXT and UIDVALIDITY in plain text files.  For the UID maps, we
need something fast.

LMDB is a simple and fast embedded memory-mapped database library with ACID
semantics, and seems like a reasonable choice.  It might be worth trying out
some alternatives, but this is a fine place to start.

lmdb-rkv is a Rust wrapper from Mozilla which seemed like a decent choice in 2020,
but that was before the Great Rust Layoff, and it now seems to be abandoned.
It might be worth looking into taking it over, or finding an alternative, or just
building our own simple binding only exposing what we need.

*** mhimap - a new nmh command for manipulating the imap-folder-manager

Currently, the only facilities this offers are for the tests.  They are:

- start
- shutdown
- noop
- check

=shutdown= may be useful to users who don't want to leave the authenticated
process running indefinitely.

=check= verifies the integrity of the synchronization database.

*** Code organization

**** /sbr/rust - crate nmh

=nmh= is the base crate.  It offers Rusty wrappers around the core subroutines
in the =sbr= directory (=libmh.a=).

**** /managedfolders

The =managed_folders= crate implements the folder-manager protocol.
Nothing specific to IMAP is here; if anyone wanted to implement a different
folder manager, they could use this.

**** /imap/types

Exposes types common to the 4 IMAP crates.

**** /imap/store

This crate implements Folder store described above.

**** /imap/folder-manager

This is =imap-folder-manager= itself.

**** /imap/mhimap

This is the =mhimap= command described above.

** Dependencies

*** Direct
libc = 0.2.78
lmdb-rkv = 0.14
lmdb-rkv-sys = 0.11
log = 0.4

*** Indirect
bitflags = 1.2.1
byteorder = 1.3
cc = 1.0.60
cfg-if = 0.1.10
pkg-config = 0.3.18

*** Unstable
cfg-if = "0.1.10"
libc = "0.2.78"
lmdb-rkv = "0.14.0"
lmdb-rkv-sys = "0.11.0"
log = "0.4.13"
pkg-config = "0.3.18"

*** License
pkg-config      MIT/Apache-2.0
bitflags        MIT/Apache-2.0
byteorder       MIT
cc              MIT/Apache-2.0
cfg-if          MIT/Apache-2.0
libc            MIT OR Apache-2.0
lmdb-rkv        Apache-2.0
lmdb-rkv-sys    Apache-2.0
log             MIT OR Apache-2.0

* Testing

Look in the =testing= directory beside this file for more information.

- =tests/imap/test-live= exercises as much of the IMAP functionality as possible
  against any supported server.  Because we run it against all servers, any test
  that can live here should live here.

- New tests using the common nmh test suite, run under =make check=.
  These tests use the classic Panda IMAP server.  =make check= also includes
  =tests/imap/test-live=, against Panda only.  Add test cases at this level only
  if they can't be written for =test-live=.

- Rust "integration tests", which go in the =tests= directory of each crate.
  Add test cases at this level only if they can't be written for the common nmh
  test suite.

- Rust "unit tests", which go in =tests= sub-modules under the =src= directory
  of each crate.  Add test cases at this level only if they can't be written as
  Rust "integration tests".

* Supported servers somewhere

** Free implementations

- Cyrus
- Dovecot
- Panda

** Service providers

- Gandi (they seem to run Dovecot)
- GMail