How Nylas syncs mail, contacts, and calendars
When an account is connected to Nylas, the sync engine starts pulling in all mail for the account, prioritizing recent messages first (though mail isn’t strictly added in reverse chronological order). As new email arrives in the user’s mailbox, it is also added in parallel. Click to see an overview of
How Nylas Works.
For contacts and calendar syncing, events and contacts are downloaded from newest to oldest. New events or contacts added after the start of the sync are not downloaded until the initial sync is complete. Since the volume of data for contacts and calendar is typically not large, this isn’t generally a problem.
Nylas continues to keep its data up-to-date with the backend provider through a number of mechanisms: keeping an IMAP IDLE connection open, using Exchange ActiveSync’s “ping” notifications, using webhooks from calendar providers, or, in the case that no better mechanism is available, simply polling the provider.
Syncing Strategies
Overview
There are two main ways to pull in email information from an account:
- The Threads and Messages endpoints allows you retrieve all messages and threads, or filter for a certain subset of them based on various constraints.
- Deltas allow you to process new data quickly without having to fetch an index of the user’s mailbox or perform a large number of API calls.
You shouldn’t make any assumptions about the sync progress of an account. We recommend using both mechanisms to ensure you’re able to quickly provide data to a user when the account starts syncing (through the threads endpoint), and to quickly update the user when new mail comes through (through deltas). There are other caveats you should keep in mind that are discussed below.
Threads and Messages
In some cases, it’s easiest to request data from the Nylas API as it’s needed for display, without ever needing to store it locally. The
Threads and
Messages endpoints are great for this use case.
For example, if your application provides a custom “Inbox” view, you could fetch data from the /threads endpoint and use it to render HTML or provide data to a single page JavaScript application. Making /threads requests on-demand rather than caching the data in a local database makes it easier to ensure the data is always current, and it’s easy to
filter messages too.
🚧Important Note
When you first connect an account and fetch data from the /threads endpoint you won’t receive all of a user’s email if Nylas is still syncing historical mail. You can’t make any assumptions about whether an account has finished syncing on Nylas’ servers, and it can take anywhere from a few minutes to several days depending on the account size.
📘Sync Tip
You could use the delta streaming purely as a notification method. New delta comes in? Just request the thread endpoint again for new messages. (Don’t forget to stay within our
rate limits!)
Deltas
The Nylas Sync Engine builds a transaction log that records every change as it synchronizes users’ mailboxes. Your application can use this transaction log, exposed through the Delta APIs, to build email applications that process incoming changes quickly, without fetching an index of the user’s mailbox, polling, or making a large number of API requests.
If your application wants to ingest emails as they are processed by the Nylas Sync Engine, the delta stream is an efficient solution.
Deltas + Threads
By themselves, the Threads and Deltas APIs aren’t sufficient to build a full, realtime view of a user’s mailbox, since:
- Threads only return mail that has already been processed by Nylas. If a user has just connected their account, a request to /threads might only return a few Thread objects that Nylas has processed. A few hours later, /threads might return many thousand objects since more mail has been processed.
- Deltas only return mail as it’s being processed by Nylas. Listening to deltas tells your application that mail is available which was was not previously synced or has been modified.
To build a robust application that maintains an up-to-date cache of a user’s entire mailbox, you should leverage both Threads and Deltas. Be sure to:
- Begin listening to deltas (changes to the mailbox) at the same time you start paginating /threads (existing data in the mailbox.)
- Always “upsert” when you retrieve objects. A delta about a thread should create or update that thread in your cache.
If your application is only interested in caching a subset of mail—for example, mail in the last month, or mail in a particular folder—you should:
- Limit pagination using threads filter parameters
- Listen for deltas, but ignore deltas about mail you do not care about.
In general, you should obtain the cursor first before paginating through /threads to ensure you don’t miss any emails.
- Authorize an account
- Obtain a cursor
- Start listening for deltas
- Paginate /threads (remember to account for duplicates)
Managing multiple account syncs
The
Delta API documentation goes into detail on when to use long polling versus streaming to call the delta endpoint. Depending on what strategy you use and your software stack, the number of parallel processes or open connections when syncing a large number of accounts may be a concern.
In our experience, clients have been successful using the Delta API with large numbers of accounts using the following asynchronous frameworks:
