Spark On K8S 0.11.0

What's Changed

Significant Changes

• chore: remove support for Python 3.8 by @hussein-awala in #96

Bug Fixes:

• fix(chart): add missing config to spark-history deployment by @hussein-awala in #104
• fix(doc): fix failed mkdocs and remove warnings by @hussein-awala in #111
• fix(webserver): fix broken apps endpoint by @hussein-awala in #117
• fix(python-client): truncate driver pod name and fix app name/id truncation by @hussein-awala in #119

New Features:

• feat(airflow): add on_finish_action to decide what to do when the job finishes by @hussein-awala in #100
• feat(python-client): add more config for dynamic allocation by @hussein-awala in #102
• feat(cli): add more config for dynamic allocation by @hussein-awala in #103
• feat(airflow): create SparkSqlOnK8SOperator to execute Spark SQL commands by @hussein-awala in #109
• feat(python-client): Support for init containers in driver pod by @hussein-awala in #121
• feat(python-client): Support for host aliases in driver pod by @hussein-awala in #122

Misc

• chore(airflow): create an enum for app_waiter by @hussein-awala in #101
• chore(spark-history): update spark-history image to jdk 17 and spark 3.5.3 by @hussein-awala in #105
• chore: update spark version in java example by @hussein-awala in #107
• chore: user StrEnum instead of str, Enum by @hussein-awala in #108
• chore(doc): add some examples for SparkSqlOnK8SOperator operator by @hussein-awala in #112
• chore: add sqlfluff to lint spark sql queries by @hussein-awala in #113
• chore(doc): add some examples for driver configuration by @hussein-awala in #125

Full Changelog: 0.10.1...0.11.0

Spark On K8S 0.10.1

What's Changed

• fix(airflow): load namespace and pod_name from trigger event by @hussein-awala in #89

Full Changelog: 0.10.0...0.10.1

Spark On K8S 0.10.0

What's Changed

• security: bump zipp to avoid CVE-2024-5569 by @hussein-awala in #81
• security: bump apache-airflow to avoid CVE-2024-39863 and CVE-2024-39877 by @hussein-awala in #82
• security: update setuptools to avoid CVE-2024-6345 by @hussein-awala in #83
• fix: propagate service account + test by @rivamarco in #88
• feat(python-client, airflow): support startup_timeout for pending state waiting by @hussein-awala in #84

New Contributors

• @rivamarco made their first contribution in #88

Full Changelog: 0.9.1...0.10.0

Spark On K8S 0.9.1

What's Changed

• fix(airflow): fix xcom pull in _try_to_adopt_job by @hussein-awala in #80

Full Changelog: 0.9.0...0.9.1

Spark On K8S 0.9.0

What's Changed

• feat(api): support pagination on list_apps route by @hussein-awala in #77
• feat: add support for packages param by @hussein-awala in #79

Full Changelog: 0.8.0...0.9.0

Spark On K8S 0.8.0

What's Changed

• feat(python-client): support creating and mounting an ephemeral configmap to the driver by @hussein-awala in #74
• feat(airflow): support driver_ephemeral_configmaps_volumes by @hussein-awala in #75
• security: update urllib3 to avoid CVE-2024-37891 by @hussein-awala in #69
• security: bump apache-airflow to avoid CVE-2024-32077 by @hussein-awala in #70
• security: update requests to avoid CVE-2024-35195 by @hussein-awala in #71
• security: bump certifi to avoid CVE-2024-39689 by @hussein-awala in #76
• docs: update python example to explain how to use custom modules by @hussein-awala in #73

Full Changelog: 0.7.2...0.8.0

Spark On K8S 0.7.2

What's Changed

• fix(airflow): fix spark_history_ui_link by removing -driver suffix by @hussein-awala in #68

Full Changelog: 0.7.1...0.7.2

Spark On K8S 0.7.1

What's Changed

• fix(airflow): configure provider metadata to make the extra-link discoverable by @hussein-awala in #60
• security: bump Jinja2 to avoid CVE-2024-34064 by @hussein-awala in #61

Full Changelog: 0.7.0...0.7.1

Spark On K8S 0.7.0

What's Changed

• dev: Create the doc with mkdocs and deploy to gh page by @hussein-awala in #52
• feat(airflow): Implement an operator link for Spark app and history UIs by @hussein-awala in #49
• test(airflow): add tests for SparkOnK8SOperatorLink by @hussein-awala in #54
• doc: improve examples doc by @hussein-awala in #56
• security: Stop returning the exception details to the API users by @hussein-awala in #57
• chore(helm): Improve the chart doc and migrate to SemVer versioning by @hussein-awala in #58
• docs(helm): add helm chart doc to the gh page by @hussein-awala in #59

Full Changelog: 0.6.0...0.7.0

Spark On K8S 0.6.0

What's Changed

• feat(airflow): add app_id_suffix argument to Airflow operator by @hussein-awala in #41
• dev: add code of conduct to the project by @hussein-awala in #43
• dev: add a security policy to the project by @hussein-awala in #44
• dev: add a contributing doc to the project by @hussein-awala in #45
• dev: setup sponsorship by adding funding.yml file by @hussein-awala in #46
• feat(airflow): try adopt running job in case of unexpected failures by @hussein-awala in #42

Full Changelog: 0.5.2...0.6.0