[.github/actions/release] upgrade base image to gen25

[Z4ck404]

Jira ticket

https://spotinst.atlassian.net/browse/BGD-6109

Description

Quartly patching => update to latest ofas sparkbase images.

Demo

Please add a recording/screenshot of the feature/bug fix in work

Checklist

• I have added a Jira ticket link
• I have filled in the test plan
• I have executed the tests and filled in the test results
• I have updated/created relevant documentation

How to test

• Run the dev image on a dev DP.
• Run the integration tests on the DP with this spark operator image.

Test plan and results

1- image is built successfully
public.ecr.aws/n8e8v3t5/spark-operator:BGD-6109-scan-of-2024-11-04-fix-vulnerabilities-discovered-in-spark-operator and running on the dev DP aws-infra-dev-dp

2- Applications failed because of missing permissions (spark operator service account was missing the permission to patch the services in the namespace spark-apps). Adding the permission to the spark operator cluster role fixed the problem and application was running (see spotinst/bigdata-charts#259)

3:57PM INF Checking cluster availability clusterId=osc-eb6dd28b
3:57PM INF Cluster is available, tests are ready to run clusterId=osc-eb6dd28b
3:57PM INF Start running tests
3:57PM INF Optimizer Url: http://optimizer-public.dev.spotinst.com:8081
3:57PM INF accountId: act-27419163
3:57PM INF clusterId: osc-eb6dd28b
3:57PM INF sparkProps: [{Version:3.5.0}]
3:57PM INF 20 tests selected: [list_applications, kill_application_pending_3.5.0, kill_application_running_3.5.0, kill_application_completed_3.5.0, run_spark_connect_application-3.5.0, run_spark_connect_application_code-3.5.0, run_notebook_application-Python, run_notebook_application-Scala, list_jobs, get_job, put_job, list_clusters, get_config_template, create_valid_config_template, create_invalid_config_template, list_config_templates, put_config_templates, delete_config_template, run_workspace, run_application-3.5.0]
3:57PM INF Starting suite=run_application-3.5.0
3:57PM INF Starting suite=delete_config_template
3:57PM INF Testing Spark submit with properties {3.5.0} suite=run_application-3.5.0
3:57PM INF Starting suite=run_workspace
3:57PM INF Starting suite=create_valid_config_template
3:57PM INF Starting suite=create_invalid_config_template
3:57PM INF Starting suite=kill_application_pending_3.5.0
3:57PM INF Trying to create invalid config template integration-tests-config-template-create-umauud suite=create_invalid_config_template
3:57PM INF Starting suite=list_applications
3:57PM INF Starting suite=put_config_templates
3:57PM INF Starting suite=run_notebook_application-Scala
3:57PM INF Workspace creation test suite=run_workspace
3:57PM INF Starting suite=list_clusters
3:57PM INF Starting suite=kill_application_running_3.5.0
3:57PM INF Starting suite=put_job
3:57PM INF Starting suite=get_config_template
3:57PM INF Starting suite=list_config_templates
3:57PM INF Trying to create config template integration-tests-config-template-create-uoxffk suite=create_valid_config_template
3:57PM INF Starting suite=get_job
3:57PM INF Creating config Scala template. suite=run_notebook_application-Scala
3:57PM INF Starting suite=list_jobs
3:57PM INF Starting suite=kill_application_completed_3.5.0
3:57PM INF Starting suite=run_spark_connect_application_code-3.5.0
3:57PM INF Testing spark connect app submit with properties {3.5.0} suite=run_spark_connect_application_code-3.5.0
3:57PM INF Starting suite=run_notebook_application-Python
3:57PM INF Starting suite=run_spark_connect_application-3.5.0
3:57PM INF Creating config Python template. suite=run_notebook_application-Python
3:57PM INF Testing spark connect app submit with properties {3.5.0} suite=run_spark_connect_application-3.5.0
3:57PM INF Success suite=list_clusters
3:57PM INF Created config template integration-tests-config-template-create-uoxffk suite=create_valid_config_template
3:57PM INF Trying to create config template with same name integration-tests-config-template-create-uoxffk suite=create_valid_config_template
3:57PM INF Testing Spark submit with properties Python suite=run_notebook_application-Python
3:57PM INF Config template created successfully suite=delete_config_template
3:57PM INF Config template created successfully suite=list_config_templates
3:57PM INF Submitted spark connect app suite=run_spark_connect_application-3.5.0
3:57PM INF Submitted app integration-tests-504aa-coded suite=kill_application_pending_3.5.0
3:57PM INF Submitted app integration-tests-job-get-jsezvr-c670e-nulls suite=get_job
3:57PM INF Successfully received unprocessable entity response for integration-tests-config-template-create-umauud suite=create_invalid_config_template
3:57PM INF Success suite=create_invalid_config_template
3:57PM INF Workspace created: wksp-5272-a1c6606e suite=run_workspace
3:57PM INF Config template created successfully suite=put_config_templates
3:57PM INF Success suite=list_applications
3:57PM INF Config template created successfully suite=get_config_template
3:57PM INF Success suite=list_jobs
3:57PM INF Submitted app integration-tests-job-put-oxrams-f6a07-incas suite=put_job
3:57PM INF Job fetched successfully suite=get_job
3:57PM INF Got a correct job suite=get_job
3:57PM INF Successfully received bad request response for integration-tests-config-template-create-uoxffk suite=create_valid_config_template
3:57PM INF Submitted app integration-tests-9ef9d-scarp suite=kill_application_running_3.5.0
3:57PM INF App is processed suite=kill_application_pending_3.5.0
3:57PM INF Config template successfully updated suite=put_config_templates
3:57PM INF Job config has updated suite=put_job
3:57PM INF Updated job labels are correct suite=put_job
3:57PM INF Submitted spark connect app suite=run_spark_connect_application_code-3.5.0
3:57PM INF Trying to list the config templates suite=list_config_templates
3:57PM INF Testing Spark submit with properties Scala suite=run_notebook_application-Scala
3:57PM INF Success suite=create_valid_config_template
3:57PM INF App is processed suite=kill_application_running_3.5.0
3:57PM INF App killed suite=kill_application_pending_3.5.0
3:57PM INF Submitted app suite=run_application-3.5.0
3:57PM INF Submitted app integration-tests-7cdf6-brine suite=kill_application_completed_3.5.0
3:57PM INF App is processed suite=run_application-3.5.0
3:57PM INF App is processed suite=kill_application_completed_3.5.0
3:57PM INF Config template successfully deleted suite=delete_config_template
3:57PM INF Success suite=delete_config_template
3:57PM INF Config template fetched successfully suite=get_config_template
3:57PM INF Got a correct config template suite=get_config_template
3:57PM INF Config template fetched successfully suite=list_config_templates
3:57PM INF Got a correct config template list suite=list_config_templates
3:57PM INF Success suite=get_config_template
3:57PM INF Success suite=list_config_templates
3:57PM INF Config template fetched successfully suite=put_config_templates
3:57PM INF Got a correct config template suite=put_config_templates
3:57PM INF Success suite=kill_application_pending_3.5.0
3:57PM INF Success suite=put_config_templates
3:57PM INF Live kubernetes logs are available suite=run_application-3.5.0
3:58PM WRN Error while submitting notebook application error="Got unexpected status 502 - 502 Bad Gateway" suite=run_notebook_application-Scala
3:58PM INF Cleaning up config templates. suite=run_notebook_application-Scala
3:58PM ERR Failure error="Got unexpected status 502 - 502 Bad Gateway" suite=run_notebook_application-Scala
3:58PM INF Workspace is running: wksp-5272-a1c6606e suite=run_workspace
3:58PM INF Workspace killed: wksp-5272-a1c6606e suite=run_workspace
3:58PM INF Success suite=run_workspace
3:59PM WRN Error while submitting notebook application error="Got unexpected status 504 - 504 Gateway Time-out" suite=run_notebook_application-Python
3:59PM INF Kernel Id not returned for Python notebook suite=run_notebook_application-Python
4:00PM INF App is now running. suite=run_application-3.5.0
4:00PM INF Success suite=get_job
4:00PM INF App is now running. suite=kill_application_running_3.5.0
4:00PM INF Live driver logs are available suite=run_application-3.5.0
4:00PM INF App killed suite=kill_application_running_3.5.0
4:00PM INF Success suite=kill_application_running_3.5.0
4:00PM INF Kernel Id found for Python notebook suite=run_notebook_application-Python
4:00PM INF Listing applications for Python test suite=run_notebook_application-Python
4:00PM INF Submitted app suite=run_notebook_application-Python
4:00PM INF App is processed suite=run_notebook_application-Python
4:00PM INF App is now running. suite=run_notebook_application-Python
4:00PM INF Get App status. suite=run_notebook_application-Python
4:00PM INF Spark connect app is now running suite=run_spark_connect_application_code-3.5.0
4:01PM INF Cleaning up config templates. suite=run_notebook_application-Python
4:01PM INF Success suite=run_notebook_application-Python
4:01PM INF Success suite=put_job
4:01PM INF Spark connect app is now running suite=run_spark_connect_application-3.5.0
4:01PM INF Spark connect app killed suite=run_spark_connect_application-3.5.0
4:01PM INF Success suite=run_spark_connect_application-3.5.0
4:03PM INF App is completed suite=kill_application_completed_3.5.0
4:03PM INF App is completed suite=run_application-3.5.0
4:03PM INF Success suite=kill_application_completed_3.5.0
4:03PM INF Spark connect app executed sql suite=run_spark_connect_application_code-3.5.0
4:03PM INF Spark connect app killed suite=run_spark_connect_application_code-3.5.0
4:03PM INF Success suite=run_spark_connect_application_code-3.5.0
4:07PM INF Logs are processed suite=run_application-3.5.0
4:07PM INF Driver log archive is available suite=run_application-3.5.0
4:07PM INF Success suite=run_application-3.5.0

Test	Description	Result	Notes
1	Run the spark operator image on a dev DP	✅
2	Run integration tests on dev DP	✅	⚠️ notebook failing with 504 errors with the integration tests but works when tested manually