Skip to content

Commit

Permalink
Implement a Persistent Job Scheduler
Browse files Browse the repository at this point in the history
Closes: #1486
  • Loading branch information
alexanderkiel committed Mar 26, 2024
1 parent ad42b37 commit 499846f
Show file tree
Hide file tree
Showing 131 changed files with 3,477 additions and 727 deletions.
1 change: 1 addition & 0 deletions .clj-kondo/root/config.edn
Original file line number Diff line number Diff line change
Expand Up @@ -71,6 +71,7 @@
blaze.executors ex
blaze.fhir.spec.references fsr
blaze.fhir.structure-definition-repo sdr
blaze.job-scheduler js
blaze.middleware.fhir.db db
blaze.rest-api.header header
blaze.scheduler sched
Expand Down
6 changes: 6 additions & 0 deletions .github/scripts/jobs/re-index/create-invalid-job.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
#!/bin/bash -e

SCRIPT_DIR="$(dirname "$(readlink -f "$0")")"
. "$SCRIPT_DIR/../../util.sh"

BASE="http://localhost:8080/fhir"
8 changes: 8 additions & 0 deletions .github/scripts/util.sh
Original file line number Diff line number Diff line change
Expand Up @@ -25,3 +25,11 @@ create() {
update() {
curl -XPUT -s -H 'Accept: application/fhir+json' -H "Content-Type: application/fhir+json" -d @- -o /dev/null "$1"
}

create() {
curl -s -H 'Accept: application/fhir+json' -H "Content-Type: application/fhir+json" -d @- "$1"
}

update() {
curl -XPUT -s -H 'Accept: application/fhir+json' -H "Content-Type: application/fhir+json" -d @- -o /dev/null "$1"
}
15 changes: 14 additions & 1 deletion Dockerfile
Original file line number Diff line number Diff line change
@@ -1,4 +1,9 @@
# Update the SHA by calling crane digest eclipse-temurin:17-jre-jammy
FROM mcr.microsoft.com/dotnet/sdk:6.0 as fhir-packages

RUN dotnet tool install -g firely.terminal
RUN /root/.dotnet/tools/fhir install hl7.fhir.r4.core 4.0.1
RUN /root/.dotnet/tools/fhir install hl7.fhir.xver-extensions 0.1.0

FROM eclipse-temurin:17-jre-jammy@sha256:c1dd17f6b753f0af572069fb80dcacdc687f5d1ae3e05c98d5f699761b84a10a

RUN apt-get update && apt-get upgrade -y && \
Expand All @@ -7,8 +12,13 @@ RUN apt-get update && apt-get upgrade -y && \
apt-get autoremove -y && apt-get clean && \
rm -rf /var/lib/apt/lists/

RUN groupadd -g 1001 blaze
RUN useradd -u 1001 -g 1001 --create-home blaze

RUN mkdir -p /app/data && chown 1001:1001 /app/data
COPY target/blaze-0.25.0-standalone.jar /app/
COPY --from=fhir-packages /root/.fhir/packages /home/blaze/.fhir/packages/
RUN chown -R 1001:1001 /home/blaze/.fhir

WORKDIR /app
USER 1001
Expand All @@ -18,5 +28,8 @@ ENV STORAGE="standalone"
ENV INDEX_DB_DIR="/app/data/index"
ENV TRANSACTION_DB_DIR="/app/data/transaction"
ENV RESOURCE_DB_DIR="/app/data/resource"
ENV ADMIN_INDEX_DB_DIR="/app/data/admin-index"
ENV ADMIN_TRANSACTION_DB_DIR="/app/data/admin-transaction"
ENV ADMIN_RESOURCE_DB_DIR="/app/data/admin-resource"

CMD ["java", "-jar", "blaze-0.25.0-standalone.jar"]
1 change: 1 addition & 0 deletions docs/implementation/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,3 +2,4 @@

* [Database](database.md)
* [FHIR Data Model](fhir-data-model.md)
* [Frontend](frontend.md)
17 changes: 17 additions & 0 deletions docs/implementation/frontend.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,17 @@
# Frontend Implementation

## Authentication / Authorization

The frontend uses the [Auth.js][1] [@auth/sveltekit][2] library for for authentication and authorization.

* a single Keycloak provider is used
* the env vars `AUTH_CLIENT_ID`, `AUTH_CLIENT_SECRET`, `AUTH_ISSUER` and `AUTH_SECRET` are used as config
* the authorization code flow is used
* at sign-in the access token and refresh token are stored in a secure, HTTP only, encrypted JWT session cookie
* nobody can access the tokens in the session cookie, because it is encrypted and only the server-side of the frontend has the secret
* the session cookie is transferred for every request (the frontend is stateless)
* the access token will be refreshed via the refresh token if possible
* the session will expire at the same time as the last successful refreshed access token will expire

[1]: <https://authjs.dev>
[2]: <https://www.npmjs.com/package/@auth/sveltekit>
5 changes: 5 additions & 0 deletions job-ig/.gitignore
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
/fsh-generated
/input-cache
/output
/temp
/template
3 changes: 3 additions & 0 deletions job-ig/ig.ini
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
[IG]
ig = fsh-generated/resources/ImplementationGuide-fhir.example.json
template = fhir.base.template#current
27 changes: 27 additions & 0 deletions job-ig/input/fsh/compact-job.fsh
Original file line number Diff line number Diff line change
@@ -0,0 +1,27 @@
Alias: $JT = https://samply.github.io/blaze/fhir/CodeSystem/JobType
Alias: $CJP = https://samply.github.io/blaze/fhir/CodeSystem/CompactJobParameter

CodeSystem: CompactJobParameter
Id: CompactJobParameter
* #column-family-name "Column Family Name"

Profile: CompactJob
Parent: Task
* code 1..1
* code = $JT#compact "Compact Database Column Families"
* input ^slicing.discriminator.type = #pattern
* input ^slicing.discriminator.path = "type"
* input ^slicing.rules = #closed
* input contains columnFamilyName 1..1
* input[columnFamilyName] ^short = "Column Family Name"
* input[columnFamilyName] ^definition = "The name of the column family to compact."
* input[columnFamilyName].type = $CJP#column-family-name
* input[columnFamilyName].value[x] only string

Instance: CompactJobExample
InstanceOf: CompactJob
* status = #ready
* intent = #order
* code = $JT#compact "Compact Database Column Families"
* input[columnFamilyName].type = $CJP#column-family-name
* input[columnFamilyName].valueString = "SearchParamValueIndex"
5 changes: 5 additions & 0 deletions job-ig/input/fsh/job-type.fsh
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
CodeSystem: JobType
Id: JobType
Title: "Job Type"
* #re-index "(Re)Index a Search Parameter"
* #compact "Compact Database Column Families"
27 changes: 27 additions & 0 deletions job-ig/input/fsh/re-index-job.fsh
Original file line number Diff line number Diff line change
@@ -0,0 +1,27 @@
Alias: $JT = https://samply.github.io/blaze/fhir/CodeSystem/JobType
Alias: $RJP = https://samply.github.io/blaze/fhir/CodeSystem/ReIndexJobParameter

CodeSystem: ReIndexJobParameter
Id: ReIndexJobParameter
* #search-param-url "Search Param URL"

Profile: ReIndexJob
Parent: Task
* code 1..1
* code = $JT#re-index "(Re)Index a Search Parameter"
* input ^slicing.discriminator.type = #pattern
* input ^slicing.discriminator.path = "type"
* input ^slicing.rules = #closed
* input contains searchParamUrl 1..1
* input[searchParamUrl] ^short = "Search Param URL"
* input[searchParamUrl] ^definition = "The URL of the Search Parameter to (re)index."
* input[searchParamUrl].type = $RJP#search-param-url
* input[searchParamUrl].value[x] only canonical

Instance: ReIndexJobExample
InstanceOf: ReIndexJob
* status = #ready
* intent = #order
* code = $JT#re-index "(Re)Index a Search Parameter"
* input[searchParamUrl].type = $RJP#search-param-url
* input[searchParamUrl].valueCanonical = "http://hl7.org/fhir/SearchParameter/Resource-profile"
5 changes: 5 additions & 0 deletions job-ig/input/ignoreWarnings.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
== Suppressed Messages ==

# Add warning and/or information messages here after you've confirmed that they aren't really a problem
# (And include comments like this justifying why)
# See https://github.com/FHIR/sample-ig/blob/master/input/ignoreWarnings.txt for examples
3 changes: 3 additions & 0 deletions job-ig/input/pagecontent/index.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
# job-ig

Feel free to modify this index page with your own awesome content!
213 changes: 213 additions & 0 deletions job-ig/sushi-config.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,213 @@
# ╭─────────────────────────Commonly Used ImplementationGuide Properties───────────────────────────╮
# │ The properties below are used to create the ImplementationGuide resource. The most commonly │
# │ used properties are included. For a list of all supported properties and their functions, │
# │ see: https://fshschool.org/docs/sushi/configuration/. │
# ╰────────────────────────────────────────────────────────────────────────────────────────────────╯
id: fhir.example
canonical: https://samply.github.io/blaze/fhir
name: job-ig
# title: Example Title
# description: Example Implementation Guide for getting started with SUSHI
status: draft # draft | active | retired | unknown
version: 0.1.0
fhirVersion: 4.0.1 # https://www.hl7.org/fhir/valueset-FHIR-version.html
copyrightYear: 2024+
releaseLabel: ci-build # ci-build | draft | qa-preview | ballot | trial-use | release | update | normative+trial-use
# license: CC0-1.0 # https://www.hl7.org/fhir/valueset-spdx-license.html
# jurisdiction: urn:iso:std:iso:3166#US "United States of America" # https://www.hl7.org/fhir/valueset-jurisdiction.html
publisher:
name: The Samply Community
url: http://example.org/example-publisher
# email: [email protected]

# The dependencies property corresponds to IG.dependsOn. The key is the
# package id and the value is the version (or dev/current). For advanced
# use cases, the value can be an object with keys for id, uri, and version.
#
# dependencies:
# hl7.fhir.us.core: 3.1.0
# hl7.fhir.us.mcode:
# id: mcode
# uri: http://hl7.org/fhir/us/mcode/ImplementationGuide/hl7.fhir.us.mcode
# version: 1.0.0
#
#
# The pages property corresponds to IG.definition.page. SUSHI can
# auto-generate the page list, but if the author includes pages in
# this file, it is assumed that the author will fully manage the
# pages section and SUSHI will not generate any page entries.
# The page file name is used as the key. If title is not provided,
# then the title will be generated from the file name. If a
# generation value is not provided, it will be inferred from the
# file name extension. Any subproperties that are valid filenames
# with supported extensions (e.g., .md/.xml) will be treated as
# sub-pages.
#
# pages:
# index.md:
# title: Example Home
# implementation.xml:
# examples.xml:
# title: Examples Overview
# simpleExamples.xml:
# complexExamples.xml:
#
#
# The parameters property represents IG.definition.parameter. Rather
# than a list of code/value pairs (as in the ImplementationGuide
# resource), the code is the YAML key. If a parameter allows repeating
# values, the value in the YAML should be a sequence/array.
# For parameters defined by core FHIR see:
# http://build.fhir.org/codesystem-guide-parameter-code.html
# For parameters defined by the FHIR Tools IG see:
# http://build.fhir.org/ig/FHIR/fhir-tools-ig/branches/master/CodeSystem-ig-parameters.html
#
# parameters:
# excludettl: true
# validation: [allow-any-extensions, no-broken-links]
#
# ╭────────────────────────────────────────────menu.xml────────────────────────────────────────────╮
# │ The menu property will be used to generate the input/menu.xml file. The menu is represented │
# │ as a simple structure where the YAML key is the menu item name and the value is the URL. │
# │ The IG publisher currently only supports one level deep on sub-menus. To provide a │
# │ custom menu.xml file, do not include this property and include a `menu.xml` file in │
# │ input/includes. To use a provided input/includes/menu.xml file, delete the "menu" │
# │ property below. │
# ╰────────────────────────────────────────────────────────────────────────────────────────────────╯
menu:
Home: index.html
Artifacts: artifacts.html

# ╭───────────────────────────Less Common Implementation Guide Properties──────────────────────────╮
# │ Uncomment the properties below to configure additional properties on the ImplementationGuide │
# │ resource. These properties are less commonly needed than those above. │
# ╰────────────────────────────────────────────────────────────────────────────────────────────────╯
#
# Those who need more control or want to add additional details to the contact values can use
# contact directly and follow the format outlined in the ImplementationGuide resource and
# ContactDetail.
#
# contact:
# - name: Bob Smith
# telecom:
# - system: email # phone | fax | email | pager | url | sms | other
# value: [email protected]
# use: work
#
#
# The global property corresponds to the IG.global property, but it
# uses the type as the YAML key and the profile as its value. Since
# FHIR does not explicitly disallow more than one profile per type,
# neither do we; the value can be a single profile URL or an array
# of profile URLs. If a value is an id or name, SUSHI will replace
# it with the correct canonical when generating the IG JSON.
#
# global:
# Patient: http://example.org/fhir/StructureDefinition/my-patient-profile
# Encounter: http://example.org/fhir/StructureDefinition/my-encounter-profile
#
#
# The resources property corresponds to IG.definition.resource.
# SUSHI can auto-generate all of the resource entries based on
# the FSH definitions and/or information in any user-provided
# JSON or XML resource files. If the generated entries are not
# sufficient or complete, however, the author can add entries
# here. If the reference matches a generated entry, it will
# replace the generated entry. If it doesn't match any generated
# entries, it will be added to the generated entries. The format
# follows IG.definition.resource with the following differences:
# * use IG.definition.resource.reference.reference as the YAML key.
# * if the key is an id or name, SUSHI will replace it with the
# correct URL when generating the IG JSON.
# * specify "omit" to omit a FSH-generated resource from the
# resource list.
# * if the exampleCanonical is an id or name, SUSHI will replace
# it with the correct canonical when generating the IG JSON.
# * groupingId can be used, but top-level groups syntax may be a
# better option (see below).
# The following are simple examples to demonstrate what this might
# look like:
#
# resources:
# Patient/my-example-patient:
# name: My Example Patient
# description: An example Patient
# exampleBoolean: true
# Patient/bad-example: omit
#
#
# Groups can control certain aspects of the IG generation. The IG
# documentation recommends that authors use the default groups that
# are provided by the templating framework, but if authors want to
# use their own instead, they can use the mechanism below. This will
# create IG.definition.grouping entries and associate the individual
# resource entries with the corresponding groupIds. If a resource
# is specified by id or name, SUSHI will replace it with the correct
# URL when generating the IG JSON.
#
# groups:
# GroupA:
# name: Group A
# description: The Alpha Group
# resources:
# - StructureDefinition/animal-patient
# - StructureDefinition/arm-procedure
# GroupB:
# name: Group B
# description: The Beta Group
# resources:
# - StructureDefinition/bark-control
# - StructureDefinition/bee-sting
#
#
# The ImplementationGuide resource defines several other properties
# not represented above. These properties can be used as-is and
# should follow the format defined in ImplementationGuide:
# * date
# * meta
# * implicitRules
# * language
# * text
# * contained
# * extension
# * modifierExtension
# * experimental
# * useContext
# * copyright
# * packageId
#
#
# ╭──────────────────────────────────────────SUSHI flags───────────────────────────────────────────╮
# │ The flags below configure aspects of how SUSHI processes FSH. │
# ╰────────────────────────────────────────────────────────────────────────────────────────────────╯
# The FSHOnly flag indicates if only FSH resources should be exported.
# If set to true, no IG related content will be generated.
# The default value for this property is false.
#
# FSHOnly: false
#
#
# When set to true, the "short" and "definition" field on the root element of an Extension will
# be set to the "Title" and "Description" of that Extension. Default is true.
#
# applyExtensionMetadataToRoot: true
#
#
# The instanceOptions property is used to configure certain aspects of how SUSHI processes instances.
# See the individual option definitions below for more detail.
#
instanceOptions:
# When set to true, slices must be referred to by name and not only by a numeric index in order to be used
# in an Instance's assignment rule. All slices appear in the order in which they are specified in FSH rules.
# While SUSHI defaults to false for legacy reasons, manualSliceOrding is recommended for new projects.
manualSliceOrdering: true # true | false
# Determines for which types of Instances SUSHI will automatically set meta.profile
# if InstanceOf references a profile:
#
# setMetaProfile: always # always | never | inline-only | standalone-only
#
#
# Determines for which types of Instances SUSHI will automatically set id
# if InstanceOf references a profile:
#
# setId: always # always | standalone-only
4 changes: 3 additions & 1 deletion modules/admin-api/.clj-kondo/config.edn
Original file line number Diff line number Diff line change
@@ -1,6 +1,8 @@
{:config-paths
["../../../.clj-kondo/root"
"../../anomaly/resources/clj-kondo.exports/blaze/anomaly"]
"../../anomaly/resources/clj-kondo.exports/blaze/anomaly"
"../../async/resources/clj-kondo.exports/blaze/async"
"../../module-test-util/resources/clj-kondo.exports/blaze/module-test-util"]

:lint-as
{blaze.admin-api-test/with-handler clojure.core/fn}}
Loading

0 comments on commit 499846f

Please sign in to comment.