Skip to content

US922349: integrating with latest dev build of worker-framework #124

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
wants to merge 21 commits into
base: develop
Choose a base branch
from
Draft

US922349: integrating with latest dev build of worker-framework #124

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
21 commits
Select commit Hold shift + click to select a range
eecd5d7
intergating with latest dev build of worker-framework
DavidGerald-Milligan Sep 4, 2024
7d1f42a
cfg update
DavidGerald-Milligan Sep 4, 2024
6563258
Added MessageSystemConfiguration
DavidGerald-Milligan Sep 5, 2024
c69fcad
Added cfg
DavidGerald-Milligan Sep 6, 2024
4d53324
Corrected cfg
DavidGerald-Milligan Sep 6, 2024
e6d92a3
Corrected cfg
DavidGerald-Milligan Sep 6, 2024
c221e00
Updated cfg
DavidGerald-Milligan Sep 6, 2024
35340f5
Updated cfg
DavidGerald-Milligan Sep 9, 2024
77bf1ac
Updated cfg
DavidGerald-Milligan Sep 11, 2024
33590f6
Update to allow sqs testing
DavidGerald-Milligan Sep 13, 2024
ccc688d
Run tests for both sqs and rabbit
DavidGerald-Milligan Sep 16, 2024
0a8ce0a
removed target dir
DavidGerald-Milligan Sep 16, 2024
52cee14
Moved verify
DavidGerald-Milligan Sep 17, 2024
fcede5a
Added test container vars for jenkins
DavidGerald-Milligan Sep 17, 2024
9ce9e3b
Added test container vars for jenkins
DavidGerald-Milligan Sep 17, 2024
e2555f2
update cfg for test framework
DavidGerald-Milligan Sep 17, 2024
61c785e
merging test reports
DavidGerald-Milligan Sep 17, 2024
d0c7146
removed sqs hostname cfg
DavidGerald-Milligan Sep 17, 2024
d0956dd
removed redundant sys vars in pom
DavidGerald-Milligan Sep 17, 2024
69aade7
Added publisher wait timeout
DavidGerald-Milligan Sep 17, 2024
4166f7b
Updated test-config
DavidGerald-Milligan Sep 18, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
35 changes: 23 additions & 12 deletions pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -62,6 +62,8 @@
<dockerCafDataProcessingOrg>${dockerImagePrefix}${dockerHubOrganization}${dockerOrgSeperator}</dockerCafDataProcessingOrg>
<dockerProjectVersion>${dockerVersionSeperator}${project.version}</dockerProjectVersion>
<projectDockerRegistry>worker-languagedetection-${project.version}.project-registries.local</projectDockerRegistry>
<worker.framework.version>8.2.0-US922349-SNAPSHOT</worker.framework.version>
<worker.framework.testing.version>3.1.0-US922349-SNAPSHOT</worker.framework.testing.version>
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@DavidGerald-Milligan revert

</properties>

<dependencyManagement>
Expand Down Expand Up @@ -285,62 +287,67 @@
<groupId>com.github.workerframework</groupId>
<artifactId>standard-worker-container</artifactId>
<type>pom</type>
<version>8.1.0-1251</version>
<version>${worker.framework.version}</version>
</dependency>
<dependency>
<groupId>com.github.workerframework</groupId>
<artifactId>util-rabbitmq</artifactId>
<version>8.1.0-1251</version>
<version>${worker.framework.version}</version>
</dependency>
<dependency>
<groupId>com.github.workerframework</groupId>
<artifactId>worker-api</artifactId>
<version>8.1.0-1251</version>
<version>${worker.framework.version}</version>
</dependency>
<dependency>
<groupId>com.github.workerframework</groupId>
<artifactId>worker-configs</artifactId>
<version>8.1.0-1251</version>
<version>${worker.framework.version}</version>
</dependency>
<dependency>
<groupId>com.github.workerframework</groupId>
<artifactId>worker-core</artifactId>
<version>8.1.0-1251</version>
<version>${worker.framework.version}</version>
</dependency>
<dependency>
<groupId>com.github.workerframework</groupId>
<artifactId>worker-default-configs</artifactId>
<version>8.1.0-1251</version>
<version>${worker.framework.version}</version>
</dependency>
<dependency>
<groupId>com.github.workerframework</groupId>
<artifactId>worker-queue-rabbit</artifactId>
<version>8.1.0-1251</version>
<version>${worker.framework.version}</version>
</dependency>
<dependency>
<groupId>com.github.workerframework</groupId>
<artifactId>worker-queue-sqs</artifactId>
<version>${worker.framework.version}</version>
</dependency>
<dependency>
<groupId>com.github.workerframework</groupId>
<artifactId>worker-store-fs</artifactId>
<version>8.1.0-1251</version>
<version>${worker.framework.version}</version>
</dependency>
<dependency>
<groupId>com.github.workerframework</groupId>
<artifactId>worker-store-http</artifactId>
<version>8.1.0-1251</version>
<version>${worker.framework.version}</version>
</dependency>
<dependency>
<groupId>com.github.workerframework</groupId>
<artifactId>worker-tracking-report</artifactId>
<version>8.1.0-1251</version>
<version>${worker.framework.version}</version>
</dependency>
<dependency>
<groupId>com.github.workerframework.testing</groupId>
<artifactId>workerframework-testing-integration</artifactId>
<version>3.0.0-205</version>
<version>${worker.framework.testing.version}</version>
</dependency>
<dependency>
<groupId>com.github.workerframework.testing</groupId>
<artifactId>workerframework-testing-util</artifactId>
<version>3.0.0-205</version>
<version>${worker.framework.testing.version}</version>
</dependency>
<dependency>
<groupId>com.google.code.findbugs</groupId>
Expand Down Expand Up @@ -925,6 +932,10 @@
<repository>${dockerHubPublic}/library/rabbitmq</repository>
<tag>3-management</tag>
</image>
<image>
<repository>${dockerHubPublic}/localstack/localstack</repository>
<tag>latest</tag>
</image>
</imageManagement>
</configuration>
</plugin>
Expand Down
114 changes: 106 additions & 8 deletions worker-languagedetection-container/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -145,33 +145,64 @@
<artifactId>maven-failsafe-plugin</artifactId>
<executions>
<execution>
<id>sqs-implementation-testing</id>
<goals>
<goal>integration-test</goal>
<goal>verify</goal>
</goals>
<configuration>
<skip>false</skip>
<systemPropertyVariables>
<!-- for test-framework-->
<sqs.ctrl.port>${sqs.ctrl.port}</sqs.ctrl.port>
<messaging.implementation>sqs</messaging.implementation>
</systemPropertyVariables>
<environmentVariables>
<CAF_APPNAME>test/sqs-worker-languagedetection</CAF_APPNAME>
</environmentVariables>
<reportNameSuffix>sqs</reportNameSuffix>
</configuration>
</execution>
<execution>
<id>rabbitmq-implementation-testing</id>
<goals>
<goal>integration-test</goal>
<goal>verify</goal>
</goals>
<configuration>
<skip>false</skip>
<systemPropertyVariables>
<!-- for test-framework-->
<worker.restport>${worker.restport}</worker.restport>
<worker.adminport>${worker.adminport}</worker.adminport>
<rabbitmq.node.port>${rabbitmq.node.port}</rabbitmq.node.port>
<rabbitmq.ctrl.port>${rabbitmq.ctrl.port}</rabbitmq.ctrl.port>
<messaging.implementation>rabbitmq</messaging.implementation>
</systemPropertyVariables>
<environmentVariables>
<CAF_APPNAME>test/rabbitmq-worker-languagedetection</CAF_APPNAME>
</environmentVariables>
<reportNameSuffix>rabbitmq</reportNameSuffix>
</configuration>
</execution>
</executions>
<configuration>
<systemPropertyVariables>
<docker.host.address>${docker.host.address}</docker.host.address>
<worker.restport>${worker.restport}</worker.restport>
<worker.adminport>${worker.adminport}</worker.adminport>
<rabbitmq.node.port>${rabbitmq.node.port}</rabbitmq.node.port>
<rabbitmq.ctrl.port>${rabbitmq.ctrl.port}</rabbitmq.ctrl.port>
<input.folder>${test.data}</input.folder>
<expected.folder>${test.data}</expected.folder>
<datastore.enabled>false</datastore.enabled>
<worker.testing.usehttpdatastore>true</worker.testing.usehttpdatastore>
</systemPropertyVariables>
<environmentVariables>
<CAF_APPNAME>test/worker-languagedetection</CAF_APPNAME>
<CAF_CONFIG_PATH>${test.configs}</CAF_CONFIG_PATH>
<CAF_INTEGRATIONTESTS_FS_PATH>\\${docker.host.address}@${webdav.apache.port}\webdav</CAF_INTEGRATIONTESTS_FS_PATH>
<CAF_WORKER_HTTP_DATASTORE_URL>http://${docker.host.address}:${webdav.apache.port}/webdav</CAF_WORKER_HTTP_DATASTORE_URL>
</environmentVariables>
<!--<debugForkedProcess>true</debugForkedProcess>-->
</configuration>
</plugin>

<!--
Builds the Docker image for the LanguageDetectionWorker, starts RabbitMQ and assembles the configuration files
for running the integration tests.
Expand Down Expand Up @@ -274,6 +305,29 @@
</run>
</image>

<!-- Run the Localstack image. -->
<image>
<alias>sqs</alias>
<name>${projectDockerRegistry}/localstack/localstack</name>
<run>
<ports>
<port>${sqs.ctrl.port}:4566</port>
</ports>
<env>
<!--https://docs.localstack.cloud/references/logging/-->
<LS_LOG>warn</LS_LOG>
</env>
<wait>
<log>Ready.</log>
<time>120000</time>
<shutdown>500</shutdown>
</wait>
<log>
<enabled>true</enabled>
</log>
</run>
</image>

<!-- Assemble the config files from the test-configs folder to an image in /config on the host. -->
<image>
<alias>config</alias>
Expand All @@ -298,9 +352,7 @@
</build>
</image>

<!-- Configuration for the worker-languagedetection image -->
<image>
<alias>worker-languagedetection</alias>
<name>${dockerCafDataProcessingOrg}worker-languagedetection${dockerProjectVersion}</name>
<build>
<from>${projectDockerRegistry}/cafapi/oraclelinux-jre21</from>
Expand Down Expand Up @@ -353,14 +405,21 @@
</inline>
</assembly>
</build>
</image>

<!-- Configuration for the rabbitmq-worker-languagedetection image -->
<image>
<alias>rabbitmq-worker-languagedetection</alias>
<name>${dockerCafDataProcessingOrg}worker-languagedetection${dockerProjectVersion}</name>
<run>
<ports>
<port>${worker.restport}:8080</port>
<port>${worker.adminport}:8081</port>
</ports>
<env>
<CAF_APPNAME>test/worker-languagedetection</CAF_APPNAME>
<CAF_APPNAME>test/rabbitmq-worker-languagedetection</CAF_APPNAME>
<CAF_CONFIG_PATH>/config</CAF_CONFIG_PATH>
<CAF_MESSAGING_IMPLEMENTATION>rabbitmq</CAF_MESSAGING_IMPLEMENTATION>
</env>
<volumes>
<from>
Expand All @@ -384,6 +443,42 @@
</run>
</image>

<!-- Configuration for the sqs-worker-languagedetection image -->
<image>
<alias>sqs-worker-languagedetection</alias>
<name>${dockerCafDataProcessingOrg}worker-languagedetection${dockerProjectVersion}</name>
<run>
<ports>
<port>${sqs.worker.restport}:8080</port>
<port>${sqs.worker.adminport}:8081</port>
</ports>
<env>
<CAF_APPNAME>test/sqs-worker-languagedetection</CAF_APPNAME>
<CAF_CONFIG_PATH>/config</CAF_CONFIG_PATH>
<CAF_MESSAGING_IMPLEMENTATION>sqs</CAF_MESSAGING_IMPLEMENTATION>
<CAF_SQS_PUBLISHER_WAIT_TIMEOUT>0</CAF_SQS_PUBLISHER_WAIT_TIMEOUT>
</env>
<volumes>
<from>
<image>config</image>
<image>webdav</image>
</from>
</volumes>
<links>
<link>sqs</link>
</links>
<log>
<enabled>true</enabled>
</log>
<wait>
<http>
<url>http://${docker.host.address}:${sqs.worker.restport}/health-check?type=READY</url>
</http>
<time>120000</time>
<shutdown>500</shutdown>
</wait>
</run>
</image>
</images>
</configuration>
</plugin>
Expand Down Expand Up @@ -425,6 +520,9 @@
<webdav.apache.port>28080</webdav.apache.port>
<worker.restport>8080</worker.restport>
<worker.adminport>8081</worker.adminport>
<sqs.worker.restport>8082</sqs.worker.restport>
<sqs.worker.adminport>8083</sqs.worker.adminport>
<sqs.ctrl.port>14566</sqs.ctrl.port>
</properties>
</profile>
<profile>
Expand Down
0 ...uagedetection~DocumentWorkerConfiguration → ...uagedetection~DocumentWorkerConfiguration
View file
Open in desktop
File renamed without changes.
0 ...etection~FileSystemDataStoreConfiguration → ...etection~FileSystemDataStoreConfiguration
View file
Open in desktop
File renamed without changes.
0 ...ker-languagedetection~HealthConfiguration → ...ker-languagedetection~HealthConfiguration
View file
Open in desktop
File renamed without changes.
0 ...guagedetection~HttpDataStoreConfiguration → ...guagedetection~HttpDataStoreConfiguration
View file
Open in desktop
File renamed without changes.
0 ...tion~LanguageDetectionWorkerConfiguration → ...tion~LanguageDetectionWorkerConfiguration
View file
Open in desktop
File renamed without changes.
3 changes: 3 additions & 0 deletions ...tainer/test-configs/cfg~test~rabbitmq-worker-languagedetection~MessageSystemConfiguration
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
{
"implementation": "${CAF_MESSAGING_IMPLEMENTATION:-this_should_get_ignored}"
}
0 ...ker-languagedetection~RabbitConfiguration → ...ker-languagedetection~RabbitConfiguration
View file
Open in desktop
File renamed without changes.
0 ...edetection~RabbitWorkerQueueConfiguration → ...edetection~RabbitWorkerQueueConfiguration
View file
Open in desktop
File renamed without changes.
8 changes: 8 additions & 0 deletions ...-container/test-configs/cfg~test~sqs-worker-languagedetection~DocumentWorkerConfiguration
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
{
"workerName": "LanguageDetectionWorker",
"workerVersion": "2.3.1",
"outputQueue": "LanguageDetectionWorker-output-1",
"threads": 1,
"maxBatchSize": 2,
"maxBatchTime": 5000
}
3 changes: 3 additions & 0 deletions ...ainer/test-configs/cfg~test~sqs-worker-languagedetection~FileSystemDataStoreConfiguration
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
{
"dataDir": "${CAF_INTEGRATIONTESTS_FS_PATH:-/srv/common/webdav}"
}
12 changes: 12 additions & 0 deletions ...etection-container/test-configs/cfg~test~sqs-worker-languagedetection~HealthConfiguration
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
{
"livenessInitialDelaySeconds": 15,
"livenessCheckIntervalSeconds": 60,
"livenessDowntimeIntervalSeconds": 60,
"livenessSuccessAttempts": 1,
"livenessFailureAttempts": 3,
"readinessInitialDelaySeconds": 15,
"readinessCheckIntervalSeconds": 60,
"readinessDowntimeIntervalSeconds": 60,
"readinessSuccessAttempts": 1,
"readinessFailureAttempts": 3
}
3 changes: 3 additions & 0 deletions ...n-container/test-configs/cfg~test~sqs-worker-languagedetection~HttpDataStoreConfiguration
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
{
"url": "${CAF_WORKER_HTTP_DATASTORE_URL}"
}
3 changes: 3 additions & 0 deletions ...r/test-configs/cfg~test~sqs-worker-languagedetection~LanguageDetectionWorkerConfiguration
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
{
"resultFormat": "SIMPLE"
}
3 changes: 3 additions & 0 deletions ...n-container/test-configs/cfg~test~sqs-worker-languagedetection~MessageSystemConfiguration
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
{
"implementation": "${CAF_MESSAGING_IMPLEMENTATION:-this_should_get_ignored}"
}
8 changes: 8 additions & 0 deletions ...gedetection-container/test-configs/cfg~test~sqs-worker-languagedetection~SQSConfiguration
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
{
"sqsProtocol": "http",
"sqsHost": "sqs",
"sqsPort": 4566,
"sqsRegion": "us-east-1",
"sqsAccessKey": "x",
"sqsSecretAccessKey": "x"
}
13 changes: 13 additions & 0 deletions ...-container/test-configs/cfg~test~sqs-worker-languagedetection~SQSWorkerQueueConfiguration
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,13 @@
{
"inputQueue": "LanguageDetectionWorker-input-1",
"retryQueue": "LanguageDetectionWorker-input-1",
"pausedQueue": "LanguageDetectionWorker-paused-1",
"rejectedQueue": "worker-rejected",
"longPollInterval": 20,
"maxNumberOfMessages": 10,
"visibilityTimeout": 300,
"messageRetentionPeriod": 1000,
"maxDeliveries": 1000,
"maxInflightMessages": 120000,
"publisherWaitTimeout": 0
}