MAIL-SYNC(1) BSD General Commands Manual MAIL-SYNC(1)

NAME

MAIL-SYNC — Synchronise mails mail dirctories on different machines

SYNOPSIS

mail-sync [--config CONFIG] [--inital-log] COMMAND

DESCRIPTION

Achieve synchronisation of maildirs by locally observing changes and forwarding them to the peers (typically accounts of the same human user on different machines) in the network. It is a requirement that the network of peers form a connected acyclic graph.

The usual way of setting up such a network is to start from a single machine, e.g., the server where mails are received, and successively adding peers to the network (see SETUP COMMANDS section for details). On each machine, mail-sync is then regularly asked to inspect a single maildir folder (e.g., by the mail-reader after switching directories, mail delivery to a folder) or all maildirs (e.g., leaving the mail reader, cronjob) for local changes. Whenever a machine is online and it is convenient, mail-sync is then asked to inform one or all of its callable peers about local changes and receive updates from them. See REGULAR COMMANDS section for more detail and FILES section on how to tell mail-sync how to call a peer.

OPTIONS
-c CONFIG | --config CONFIG

Specify the location of the config file. Defaults to ~/.mail-syncrc.

--initial-log

Also log (to stdout) before the log file, in mail-sync’s status directory is determined.

REGULAR COMMANDS

mail-sync syncOne MAILBOX
mail-sync syncAllMailBoxes
mail-sync uux PEER-ID cmd [arg ...]
mail-sync update PEER
mail-sync updateAll
mail-sync purge

The commands syncOne and syncAllMailBoxes ask mail-sync to insepct for local changes a given maildir, or all maildirs, respectively. Changes are stored mail-sync’s global state description.

The uux command asks mail-sync to schedule the execution of a specific command on the peer specified by its UUID; the peer as to have that command whitelisted in its configration (otherwise the command will be rejected at update).

The update and updateAll commands ask mail-sync to call the given peer, or all callable peers, respectively. The information on how to call a given peer (typically by invocing ssh with appropriate arguments) is taken form the configuration. During such a call, both peers inform each other about their local changes. Received changes are applied and stored to be forwarded to the other peers of that node, if any. By the property of the network of being connected and acyclic each change is in this way eventually received by every peer exactly once.

Finally, the purge command asks mail-sync to drop all information from its internal state that is no longer needed. Typically, these are changes to be forwarded to other peers where the peer has already confirmed good receipt of that update.

SETUP COMMANDS

mail-sync setup init NAME
mail-sync setup printUUID
mail-sync setup syncAllMailBoxesInit
mail-sync setup initialSyncFor PEER-ID NAME
mail-sync setup initialSyncFrom
mail-sync setup listPeers
mail-sync setup dropPeer PEER-ID
mail-sync setup defaultConfig

Each peer starts with the init command to set up an empty local state. This command also generates a UUID for that peer that is later used to identify that peer in the network. It can be read by the printUUID command. The given name is also stored and used by some commands as a more human-friendly way of identifying a peer. As mail-sync needs to know where to put the local state, the init command already needs a configuration file; by default, that file is asusmed to be located in .mailsyncrc of the user’s home directory. To simplify creation of such a file, the defaultConfig command prints a sample config file on stdout. That command does not need a config file and ignores any config file specified with the --config option.

The founding peer, and only the founding peer, before peering with anyone, may, and usually does, call syncAllMailBoxesInit to inspect all present mail boxes in order to obtain an up-to-date initial state, without generating any update information yet.

A peer already in the network can be asked to accept a new peer with the initialSyncFor command; the command outputs on standard out (in haskell’s binary format) all information the peer needs to know, including the contents of all tracked mailboxes. This output has to be given to the new peer on stdin of the initialSyncFrom command. Given the size of this description, this data is typically streamed, over a good network, via ssh, but store and forward works equally well. As soon as the peer in the network has completed the initialSyncFor command, it will consider the new peer as present in the network and store updates for them in its state, for later consumption.

Finally, the dropPeer command tells mail-sync to forget about the specified peer. This peer (and the whole subtree reachable via them) will no longer be part of the network and can be discarded; all non-forwarded local changes of that subtree will be lost. The ids of all peers known can be obtained using the listPeers command.

PEER-TO-PEER COMMANDS

mail-sync peerToPeerProtocol requestFrom PEER-ID
mail-sync peerToPeerProtocol sendSince PEER-ID <change-message number>
mail-sync peerToPeerProtocol applyMany PEER-ID

These commands are used by the update commands in the call to the other peer. Usually, they are not called directly.

The requestFrom command outputs a command-line (calling sendSince) asking the peer to send it all the information they do not know yet. The sendSince command outputs on standard out (in haskell’s binary format) the sequence of updates since the given change; as a side effect, the peer executing sendSince remembers that the given peer already knows everything up to the given change number. The output of sendSince has to be given to standard in of the applyMay command on the receiving peer, to update their mailbox and state accordingly.

DEBUG COMMANDS

mail-sync debugState showGlobalState
mail-sync debugState encodeGlobalState
mail-sync debugState showOutGoing

These commands are mainly for developers to debug. The showGlobalState outputs the state of that peer in non-binary format as given by haskell’s Show instance. The encodeGlobalState reads a non-binary format of a global state on standard in and outputs its binary representation on standard out; it does not modify the state of that peer.

The showOutGoing command lists all patches in the global state that might still have to be sent to some peer. While this information is also part of the global state, having only the outgoing patches is a lot shorter and hence more easy to process.

CONSISTENCY AND CRASH RECOVERY

During normal operation, i.e., after having successfully completed the initial sync, all operations on mail dirs are performed in an atomic durable way. In other words, after a crash, an operation on a mail dir has happened either completely and correctly, or not at all. Also, updates to the global state are written in an atomic durable way and only after the corresponding operation on the mail dir has been persisted. Therefore, after a crash, the global state is always consistent. The only possible difference between state of record and state of world is that changes might be applied to the mail dir that are not yet recorded in the global state. To mail-sync they look as if the have originated simultaneously, but independently by the local peer. For the two idempotent operations flag change and deletion, this perceived independent change will effectively be a no-op. For mail addition, the mail will have a new base name and be assigned a new global identifier. Therefore, this duplication of mail will propagate consistently over the network. As same mail user agents have an easy way to deduplicate mails based on the message identifier, this is merely an annoyance, but no serious harm is done; the duplicate can be removed anywhere in the network (but do make sure this happens only at one place before propagating the change to avoid the two copies being removed independently at different nodes).

Based on this observation, the global state is written out only at the end of a peerToPeerProtocol applyMany operation (usually called indirectly via update or updateAll), as well as after a configurable number of new-mail-generation operations, as specified by the attribute stateWritePeriod in the configuration.

During setup initialSyncFrom, on the other hand, the main focus is speed. A sync(2) is only performed at the very end of that operation. Therefore, after a crash during an initial sync, the only reliable way to get back to a consistent state is to throw away everything on the new peer, do a setup dropPeer on the old peer, and start the initial sync again.

FILES
~/.mail-syncrc

The default location of the configuration file for mail-sync. It contains the Show representation of the configuration data. It, in particular, specifies the mail-boxes that mail-sync should consider existing by include/exclude patters, and the instructions on how to call each peer, if possible.

~/.mail-sync

The default location of the directory where mail-sync stores all it files, including the log file called log. It is the user’s responsibility to rotate this file.

EXIT CODES

Returns 0 on success and a non-zero value otherwise. For the non-succesfull invocations, a set bit with value 64 indicates a prerquisite error, indicating that no changes on the file system happened. Exit codes in the range 1 to 15 indicate internal errors in the run time and the state left on the file system is unknown in this case.

The exit codes generated by mail-sync itself are the following.

Success.

During application of incoming changes, a parse failure occured. Some changes might have already be applied, but the global state should remain in a state consistent with the changes applied to the mail-dirs.

An error occurred during the processing of the initial setup message. In this case, the newly generated local copy of the mailboxes can be discarded and a new peering can be attempted (after dropping the peer on the remote side, if not tried with the same initial message).

The global state could not be read.

Refusing to execute init again, as a global state is already present.

Refusing to perform setup operations that can only done before accepting peers, as mail-sync has peers already.

Refusing to accept an initial message as the mail root is already present.

Refusing to accept an already known peer again.

Failed to parse the command line.

Failed to read and parse the config file.

AUTHORS

Klaus Aehlig <aehlig@linta.de>, Christian Luetkte-Stetzkamp <christian@lkamp.de>

Apr 15, 2021