Memento Database API Adaptor

[laszcz]

Request

We want to build an adaptor to integrate with the Memento Database REST API, which is a cloud-based database service that allows users to create custom databases and mobile apps. The API has strict rate limits of 10 requests per minute, and maintaining precise sync progress is critical for reliable incremental data synchronization.

To start, this adaptor should:

1. Handle API key authentication
2. Retrieve library data by revision - Priority feature to get all entries from a library after or at a specific revision number for incremental syncing
3. Implement automatic rate limiting (API requests currently limited at 10 per minute) and pagination accordingly
4. Track exact progress so operations can resume from failure points and provide resumable sync functions for incremental data collection

Credentials

• Login credentials: API key from LP under Memento Database API
• Base URL: https://api.mementodatabase.com/v1/
• Test record(s): One library exist with data samples (first retrieve library id to query records)
• Rate limit: 10 requests per minute (enforced by API)

Methods needed

See the different available methods here

API endpoints needing wrappers:

• GET a library by id GET https://api.mementodatabase.com/v1/libraries/libraryId
• GET all entries in a library by library id GET https://api.mementodatabase.com/v1/libraries/libraryId/entries and use startRevision and pagination
• GET an individual entry by entry id GET https://api.mementodatabase.com/v1/libraries/libraryId/entries/entryId

Example of suggested core functions

getEntriesByRevision('libraryId', { 
  revision: 1000,
  limit: 100,
  pageToken: 'eyJvZmZzZXQiOjEwMH0' // optional, if not the first page, use the next page token to retrieve subsequent records,
  autoThrottle: true // Respects 10 req/min limit,
  trackLastRevisionRetrieved : true // returns the last full revision number completely retrieved (moved to another one higher up?) 
});

• Retrieves all entries modified after specified revision
• Critical for incremental synchronization
• Parameters: revision, pageSize limit, next pageToken, autoThrottle (?)
• Automatically handles pagination within rate limits
• Returns: { entries: [], nextPageToken: string|null, hasMore: boolean }

get(path, options) - Generic GET request with rate limiting

Vision and use cases

Our use case: we want to retrieve periodically all entries in a library (new and updated ones thanks to revision number) to then upsert into an external database. However we will face API limit restrictions (10 per minutes) so we need to factor that in our approach. We could use collections to store current revision number (workflow 2 described below).

Workflow 1: Library Metadata Sync (Daily)
├── Job: Sync library definitions and field mappings
└── Store in OpenFn collections

Workflow 2: Incremental Data Sync (Every 10-15 mins)
├── Job 1: Fetch entries from Memento (using adaptor)
├── Job 2: Transform and process entries  
├── Job 3: Send to destination system(s)
└── Job 4: Update revision tracker

Workflow 3: Outbound Sync (Every 5 mins)
├── Job 1: Get pending changes from source
├── Job 2: Transform for Memento format
└── Job 3: Send to Memento (using adaptor)

Resources

Main API Documentation

Logo Links

Logo link

[mtuchi]

The pageToken in getEntriesByRevision is a red flag to me. I think the adaptor should auto fetch records and provide a boolean option if you don't want to auto-fetch.

I think we should start with a generic get(), post(), request() functions and cut a release then build a new getEntriesByRevision function based on how get() function will be used.

This

getEntriesByRevision('libraryId', { 
  revision: 1000,
  limit: 100,
  pageToken: 'eyJvZmZzZXQiOjEwMH0' // optional, if not the first page, use the next page token to retrieve subsequent records,
  autoThrottle: true // Respects 10 req/min limit,
  trackLastRevisionRetrieved : true // returns the last full revision number completely retrieved (moved to another one higher up?) 
});

Can be this

get("library/:libraryId", {
  revision: 1000,
  limit: 100,
  autoThrottle: true,
  trackLastRevisionRetrieved: true,
});

And still make sense

[josephjclark]

The problem with infinite auto pagination @mtuchi is that it'll just fetch items forever and you'll end up with a massive state object that can't be processed. So a lot depends on load. How many items will be downloaded every 10 minutes?

What we really need is automatic pagination with a limit, but also to preserve pageToken for manual pagination downstream. You'd have to use cron or keep calling a webhook to page over multiple workflow runs.

Let's include a response object with { pageToken, lastRevisionRetrieved }

autothrottle feels like a good idea when autopaging, and should not be an option - we should just do it. But this will only work if a single workflow is running at a time. If there's concurrency (multiple runs in parallel) then there isn't a good way to limit rates across the runs.

One approach might be to just fetch "naturally", but if a rate limit error comes back from the server, we wait 30 seconds or so and then try again. So no need for a counter - we just respond to the server directly.

Can I recommend getEntries(libraryId, { entryId?, ...otherOptions }). This would let you get one or more entries from a given library. You probably then also want a getLibrary() if you want library (meta?) data.

[josephjclark]

Ditching entryId because we recently decided that getEntry() and getEntries() is a better structure for these APIs!