Microservice implementing the Global Alliance for Genomics and Health (GA4GH) Data Repository Service (DRS) API specification.
To build locally and push to docker hub, complete the following steps (todo automate this):
docker login
docker build . -t pacificanalytics/pa-drs-crypt4gh-poc:x.x.x
docker push pacificanalytics/pa-drs-crypt4gh-poc:x.x.x(1) Create a Python environment (as a virtual environment, via Conda, etc) and install the server application and its dependencies. From the root of the repo, run:
pip install -r requirements.txt
pip install -e . -v(2) Create a public and a private key for the server:
crypt4gh-keygen --sk server-sk.key --pk server-pk.key- Build the container
docker build . -t crypt- Run the container
docker run \
-e MONGO_DBNAME=drsstore \
-e MONGO_HOST=localhost \
-e MONGO_USERNAME=admin \
-e MONGO_PASSWORD=password123 \
-e ACCESS_KEY=123 \
-e SECRET_KEY=456 \
-e STORAGE_BUCKET=mybucket \
-e STORAGE_SECURE=false
cryptTo deploy into an existing kubernetes cluster, the cluster will require some dependencies to already be installed such as the nginx ingress, cert-manager and mongodb community operator.
- Ensure docker registry secret exists (you can create the token in via dockerhub):
kubectl create secret docker-registry dockerhub --docker-username=%username% --docker-password=%token%- Ensure the mongodb pass exists (replace with real values)
kubectl create secret generic pa-drs-crypt4gh-poc-secrets \
--from-literal=MONGOPASS=password123 \
--from-literal=ACCESS_KEY=access-key \
--from-literal=SECRET_KEY=secret-key \
--from-literal=STORAGE_BUCKET=staging-pa-drs-crypt4gh-poc \
--from-literal=STORAGE_HOST=s3.ap-southeast-2.amazonaws.com \
--from-file=PUB_KEY=key.pub \
--from-file=SEC_KEY=key- Install this helm chart:
helm upgrade -i crypt4gh-poc deployment- Run the following (edit to taste):
export MONGO_URI="mongodb://admin:password@db:27017/drsStore?authSource=admin"
export SEC_KEY="-----BEGIN CRYPT4GH PRIVATE KEY-----
YzRnaC12MQAEbm9uZQAEbm9uZQAg5eYgf1QUl1cFyquP6OgMz2faF2uSc4s8OXf0L4MLRQM=
-----END CRYPT4GH PRIVATE KEY-----"
export PUB_KEY="-----BEGIN CRYPT4GH PUBLIC KEY-----
dq/9iq2WMYpYQqnxVpfd0pwRp2PToAccVWldr+kynCI=
-----END CRYPT4GH PUBLIC KEY-----"
export STORAGE_HOST=https://s3.ap-southeast-2.amazonaws.com
export STORAGE_BUCKET=staging-pa-drs-crypt4gh-poc
export STORAGE_SECURE=true
export ACCESS_KEY=key
export SECRET_KEY=secret
docker-compose up --buildLogs are sent to elasticsearch via fluentd or failing that you can access the logs directly using kubectl by completing the following steps:
- Install awscli and ensure you have it authed with
aws configure - Install kubectl and run the aws command the configures your kubectl to work with eks
aws eks update-kubeconfig --region ap-southeast-2 --name staging-pa-drs-kubernetes - View the current pods running in the cluster
kubectl get pods - View the logs for the pod you are interested, in our case it would be
kubectl logs -f pa-drs-crypt4gh-poc-684d56f666-c6z9c(-f being follow which is similar to tail).
The source code repository is mounted as a volume inside the Docker container for the server. That means that you can edit the code in this repo, and the server will automatically restart when any changes are detected. There is no need to rebuild or even restart the Docker container.
To run the unit tests, it is necessary to activate the Python environment set up above. To run the unit test suite, run the following command:
pytest testsIf pytest cannot be found, install the testing requirements via pip install -r requirements-test.txt.
The unit test suite does not require the server to be running.
The server comes with a number of basic integration or "smoke" tests that test basic end-to-end functionality of the entire server stack. The goal is not to exhaustively test the server and make manual QA superfluous, but rather to detect simple breakage that cannot be detected at the level of the unit tests early on.
Currently the following scenarios are tested:
- Querying the service-info endpoint and comparing the output with what is expected.
To run the integration test suite, bring up the server as described above, and then run (from within the development environment, and with the same environment variables set as were used to bring up the server):
pytest integrationThe integration test suite is automatically run under GitHub actions as well, for every PR.
The file config.yaml in the drs_filer can be used to set various configuration options for the server. The Crypt4GH-related options and some others can also be passed in as environment variables. Currently the following options are supported:
STORAGE_HOST: The FQDN of the storage host (e.g.https://s3.eu-west-1.amazonaws.comfor AWS orhttp://localhost:9000for Minio).STORAGE_BUCKET: The name of the bucket.STORAGE_SECURE: Whether or not to check the TLS certificate of the storage host.
Note that the server must have the following environment variables set as well, as detailed above:
ACCESS_KEY: Access key ID providing access to the storage bucket.SECRET_KEY: Secret access key providing access to the storage bucket.
This project is a community effort and lives off your contributions, be it in the form of bug reports, feature requests, discussions, or fixes and other code changes. Please refer to our organization's contributing guidelines if you are interested to contribute. Please mind the code of conduct for all interactions with the community.
The project adopts the semantic versioning scheme for versioning. Currently the service is in beta stage, so the API may change without further notice.
This project is covered by the Apache License 2.0 also shipped with this repository.
The project is a collaborative effort under the umbrella of ELIXIR Cloud & AAI. Follow the link to get in touch with us via chat or email. Please mention the name of this service for any inquiry, proposal, question etc.