This sub module contains upload source connector of RADAR-base platform. This application periodically pools data stored in radar_upload_backend for processing then uploading it to Kafka. The processing of the data depends on the metadata of each record read from the database and eventually converted into to avro records and published to Kafka.
Some files such as wearable camera data (images) and raw binary data from Physilog5 are directly uploaded to the target destination.
Each record of data belongs to a specific source-type and each source-type should have a matching ConverterFactory implemented in radar-upload-source-connector.
Currently supported ConverterFactory classes are listed below:
- org.radarbase.connect.upload.converter.altoida.AltoidaConverterFactory
- org.radarbase.connect.upload.converter.axivity.AxivityConverterFactory
- org.radarbase.connect.upload.converter.oxford.WearableCameraConverterFactory
- org.radarbase.connect.upload.converter.gaitup.Physilog5ConverterFactory
To add support to process data from new source-types, a compatible ConverterFactory should be implemented according to the data structure of the data.
In addition to Zookeeper and Kafka brokers, [radar-upload-backend] and [managment-portal] applications should be running to use radar-upload-source-connector
To run a all dependent services of radar-upload-source-connect in your localhost, please refer to the docker-compose
First, create OAuth Clients for radar-upload-source-connector and radar-upload-backend in Management Portal
Copy etc/source-upload.properties.template to docker/source-upload.properties and enter
the client ID and client secret of radar-upload-source-connector.
By default all ConverterFactory classes are enabled. You can enable a subset of them by specifying a list of ConverterFactory classes in comma-separated-value for property upload.source.record.converter.classes
The following tables shows the all properties.
| Name | Description | Type | Default | Valid Values | Importance |
|---|---|---|---|---|---|
| upload.source.client.id | OAuth Client id of radar-upload-source-connector in Management Portal. | string | radar-upload-connector-client | high | |
| upload.source.client.secret | OAuth Client secret of radar-upload-source-connector in Management Portal. | string | high | ||
| upload.source.client.tokenUrl | OAuth Token url of Managment Portal. | string | http://managementportal-app:8080/managementportal/oauth/token | Resolvable URL of Managmemnt portal plus /oauth/token | low |
| upload.source.backend.baseUrl | Base URL of radar-upload-backend | string | http://radar-upload-connect-backend:8085/radar-upload/ | low | |
| upload.source.poll.interval.ms | How often the connector should poll new data from radar-upload-backend to process in millseconds | long | 60000 | high | |
| upload.source.queue.size | Capacity of the records queue. | int | 1000 | high | |
| upload.source.record.converter.classes | List of `ConverterFactory` classes to be enabled, separated by commas. | list | high | ||
If any of the ConverterFactory with direct file upload is enabled, the following properties must be configured.
| Name | Description | s3 | sftp | local |
|---|---|---|---|---|
| upload.source.file.uploader.type | What type of file uploader should be used for direct file uploads. Specify a value from `s3`, `sftp` or `local`. | s3 | sftp | |
| upload.source.file.uploader.target.endpoint | Advertised URL Endpoint of the file upload target. For sftp, specify advertised url in sftp://host:port format. If port is not specified default port 22 will be used. | http://minio:9000/ | sftp://hostname:port/ | file:// |
| upload.source.file.uploader.target.root.directory | Target root directory or a s3 bucket name where files should be uploaded to. | radar-output-storage | /output | |
| upload.source.file.uploader.username | Username to upload files to the target. Specify the access-key if s3 is chosen. | access-key | sftp username | |
| upload.source.file.uploader.password | Username to upload files to the target. Specify the secret-key if s3 is chosen. | secret-key | sftp password | |
| upload.source.file.uploader.sftp.private.key.file | Path of private-key file if using private key for uploading files using sftp. | /etc/upload-upload-source-connect/ssh-privatekey | ||
| upload.source.file.uploader.sftp.passphrase | Passphrase of the private-key file if using private key for uploading files using sftp. | passphrase for private-key | ||
Code should be formatted using the Google Java Code Style Guide. If you want to contribute a feature or fix browse our issues, and please make a pull request.