OSDU provides an open source technology-agnostic data platform for the energy industry. OSDU platform support most datatypes found in the energy industry. To learn more about OSDU visit here. To know more about OSDU™ Data Platform on AWS visit here.
Amazon AppFlow is a fully-managed integration service that enables you to securely exchange data between software as a service (SaaS) applications, such as Salesforce, and AWS services, such as Amazon Simple Storage Service (Amazon S3) and Amazon Redshift. For example, you can ingest contact records from Salesforce to Amazon Redshift or pull support tickets from Zendesk to an Amazon S3 bucket. The following diagram illustrates how it works:
AppFlow can create data flows for services that it does not natively support by leveraging custom connectors. With custom connectors, you can interact with these services from the AppFlow console just like any natively supported connector. To build a custom connector, use the Custom Connector SDK (Python SDK or Java SDK) to implement an AWS Lambda function. This Lambda function will serve as the custom connector, and will be compliant with Lambda's quotas and other requirements. You can learn more about the Custom Connector SDK here.
This custom example OSDU Appflow connector supports both source and destination modes. OSDU custom connector runs search query provided by the user on a source OSDU instance and exports/ingests the resulting OSDU records to Destination services like Amazon S3 or another OSDU instance.
The process for deploying and using the Appflow custom connector is as follows:
- Implement and deploy Lambda functions to be used as custom connectors.
- Registering Custom Connector using AppFlow Console or AWS CLI
- Creating one or more connections using the registered connector.
- Creating one or more flow using registered connector and connections
It is necessary to implement the following three Classes and methods within each Class in Custom Connector Lambda Function.
describe_connector_configuration defines the capability of the connector such as supported modes, supported Auth types, scheduling frequencies, runtime settings of different scopes(source mode, destination mode, both and connector profile) etc., This information is fetched during connector registration process and stored in AppFlow connector registry. Using this information Amazon AppFlow UI renders the connector and navigate the user accordingly.
A successful registration of a connector enables user to create connector profile according to the supported Auth types and flows according to supported modes. A successful connector profile includes validating the user credentials and connector profile runtime settings. During this process Amazon AppFlow invokes validate_credentials to validate the user credentials and validate_connector_runtime_settings with connector_profile scope to validate the runtime settings.
Screenshot from Amazon AppFlow Custom Demo Connector for OSDU
Amazon AppFlow deals with metadata of the object to create proper query for fetching the data from source and apply the transformations given in the flow tasks. AppFlow restricts users to use only the entities defined the listEntities for both source and destination modes in the flows. Also restricts users to use only the fields and its properties defined in the describeEntity in tasks in order to apply the transformations such as mapping, masking, filtering etc., during data processing step of the flow execution. The properties of the entity such as datatype, queryable, updatable etc., helps customers to define the filter condition, mapping tasks and flow triggers correctly.
list_entities is called during flow creation and returns a list of schema definitions (Entity). This Entity is defined for each supported API.
describe_entity is called in data field mapping when creating a flow and returns the field definition.
Screenshot from Amazon AppFlow Custom Demo Connector for OSDU
AppFlow’s Flow execution for a custom connector mostly depends on the implementation of of this RecordHandler interface. This is where we expect customers to implement the core functionality of making requests source/destination application in order to get/post data.
query_data is called when the flow is run to get the fields defined in the flow. In this OSDU custom connector example, we make a call to OSDU search API in query_data function, get the record Id from the result and then get the storage record and return the results.
write_data is called when the the flow is run to write the fields to the supported destination connector. If a flow has destination connector as another OSDU instance or other supported destination connector, write_data ingests storage records into destination connector.
Screenshot from Amazon AppFlow Custom Demo Connector for OSDU
- AWS CLI already configured with Administrator permission
- Docker Desktop running locally
- NodeJS installed
- AWS CDK installed
- Clone the repository
git clone https://github.com/srihariph/osdu-appflow-custom-connector.git
- Navigate to Project directory
cd osdu-appflow-custom-connector
- Install Dependencies
npm install
- Deploy AppFlow Custom Demo Connector for OSDU
cdk deploy --all