Skip to content

Commit ba7f927

Browse files
wardlicanwardliturboFeizhoujinsongliaoyt
authored
[Subtask]: Use a new configuration item to control whether master & slave mode is enabled. (#3846)
* [Subtask]: Use a new configuration item to control whether master & slave mode is enabled. #3845 * [Subtask]: Use a new configuration item to control whether master & slave mode is enabled. #3845 * [AMORO-3850] Fix openapi sdk build and refine the package name (#3847) * rename the sdk pkg * version and artifactId * [HotFix] Change the uri configuration name in terminal (#3844) Change the uri configuration in terminal * [AMORO-3851] Separate spark-3.3 and spark-3.5 modules (#3848) * separate * java17 * build * GA * jdk17 tests * fix conflicts * <java.source.version>17</java.source.version> * JDK8 + Spark-3.5 * revert JDK17 * [AMORO-3857] Make the metadata file name conform to the Iceberg spec (#3858) Each version of table metadata is stored in a metadata folder under the table’s base location using a naming scheme that includes a version and UUID: <V>-<random-uuid>.metadata.json. * [AMORO-3850][FOLLOWUP] Add GA for OpenAPI SDK (#3870) [AMORO-3850][FOLLOWUP] Add GA for openAPI SDK * [AMORO-3864] Bump spark 3.5 version to 3.5.7 (#3860) Bump spark 3.5 version to 3.5.7 * [AMORO-3852][BUILD] Enhance build system (#3849) * Build project with fixed version check-latest: false GA * docs * shell * [AMORO-3851][FOLLOWUP] Using scala.binary.version for Spark dependency artifactId (#3861) Using scala.binary.version for dependency artifactId * [AMORO-3851][FOLLOWUP] Fix and simplify spark versions management (#3874) * Fix iceberg-spark artifactId in amoro-mixed-spark-3-common * fix ut conflicts * [AMORO-3863] Add script to reformat code (#3862) * format * spark profiles * [AMORO-3852] Using Maven Wrapper (#3879) * save * 3.9.9 * mvnw * slash * maven 3.9.11 * maven wrapper 3.3.4 * [AMORO-3880] Make amoro-openapi-sdk standalone (#3881) * [AMORO-3880] Make amoro-openapi-sdk standalone * save * [AMORO-3891] Bump netty version to 4.1.128.Final to fix CVE (#3892) Bump netty version to 4.1.128.Final to fix CVE * [AMORO-3890]Bump flink versions from 1.16.2/1.17.1/1.20.0 to 1.16.3/1.17.2/1.20.3 (#3889) bump * [AMORO-3880][FOLLOWUP] Fix pull request paths for OpenAPI SDK CI (#3903) * Bump orc-core version from 1.8.3 to 1.9.7 (#3900) * [AMORO-3883] Bump Paimon 1.1.1 to 1.2.0 (#3884) * Bump Paimon 1.1.1 to 1.2.0 * [AMORO-3883] Bump Paimon 1.1.1 to 1.2.0 --------- Co-authored-by: Xu Bai <xuba@apache.org> * [AMORO-3863][FOLLOWUP] Support to reformat pom (#3882) Support to reformat pom format trino * [AMORO-3872] Support to customize basic authentication implementation (#3871) * support to customize basic auth" * save * docs * common * address comments * nit --------- Co-authored-by: Xu Bai <xuba@apache.org> * [AMORO-3907] Bump kubernetes-client version to 6.13.5 (#3908) Bump kubernetes-client version to 6.13.5 * [AMORO-3875] Support to build on Spark Scala-2.13 (#3878) * scala binary version save bin profile GA revert paimon ams api compatiblity fix flink scala style save save rewrite by scala save conflicts dependency save save scala paimon ignore paimon idea * nit * nit * save * revert ci change * [AMORO-3875][FOLLOWUP] Fix optimizer-spark docker image build due to artifactId change (#3912) * fix save * test * Revert "test" This reverts commit 6c38dbd. * [AMORO-3863][FOLLOWUP] Enable format-mixed-format-trino profile on JDK17+ automatically (#3906) reformat * [AMORO-3848][FOLLOWUP] Prompt spark profile in docker build (#3904) Prompt spark profile in docker build * [AMORO-3918][INFRA] Add dependencies check CI (#3917) * [AMORO-3931] Exclude curator and zookeeper deps (#3932) * save deps exclude jline * nit * [AMORO-3945][DOCS] Add docs for REST API (#3944) docs combine * [AMORO-3873] Support Bearer/JWT authentication (#3905) * save * save * save * save * Save * asve * rename package * comments * basic * remove token from log * [AMORO-3938] change file_content_b64 column type to CLOB type (#3939) * [AMORO-3933] Fix Playground demo failure due `PartitionExpressionForMetastore class not found` (#3935) * Fix Playground demo failure due PartitionExpressionForMetastore class not found * Revert "Fix Playground demo failure due PartitionExpressionForMetastore class not found" This reverts commit fea6ddc. * add for runtime hive-exec --------- Co-authored-by: Xu Bai <xuba@apache.org> * Save the last completion time for each cleanup operation performed on each optimization table. (#3802) * Save the last completion time for each cleanup operation performed on each optimization table. # Conflicts: # amoro-ams/src/main/resources/mysql/upgrade.sql * Store the execution time of each cleanup operation for the optimization table in the table_runtime_state table. * fixup style --------- Co-authored-by: 张文领 <zhangwl9@chinatelecom.cn> * nit: remove duplicate code (#3957) NIT: Remove duplicate code * [AMORO-3873][FOLLOWUP] Rename rest auth bearer type to JWT (#3953) * [AMORO-3968] Update the thrift api compile command to use amoro shaded thrift (#3967) * relocate generated thrift code to use amoro shaded thrift * docs * save * profile * [AMORO-3961] Filter null key and value for Configurations::toMap (#3962) filter null key value * nit: remove unused code (#3959) remove unused code * [AMORO-3972][Core] Upgrade default Spark version from 3.3 to 3.5 (#3975) * [AMORO-3972][Core] Upgrade default Spark version from 3.3 to 3.5 This PR upgrades the default Spark version from 3.3 to 3.5 in the Amoro project. Changes: - pom.xml: Update spark.version from 3.3.4 to 3.5.7 and spark.major.version from 3.3 to 3.5 - pom.xml (hadoop2 profile): Update spark.version from 3.3.4 to 3.5.7 and spark.major.version from 3.3 to 3.5 - docker/build.sh: Update SPARK_VERSION from 3.3.3 to 3.5.7 - docker/optimizer-spark/Dockerfile: Update ARG SPARK_VERSION from 3.3.3 to 3.5.7 - .github/workflows/docker-images.yml: Update Spark optimizer matrix from 3.3.3 to 3.5.7 - README.md: Update Spark optimizer default version documentation Closes #3972 * Keep Spark 3.3 as default for hadoop2 profile Address review feedback from @turboFei: For hadoop2 profile, keep using the legacy Spark 3.3 by default for better compatibility with Hadoop 2.x. * Fix Spark version parameter name in README.md Address review feedback from @turboFei: Change -Dspark-optimizer.spark-version to -Dspark.version This was missed in PR #3874. * [AMORO-3966][Helm] Support custom volumes and volumeMounts (#3965) [Improvement][Helm] Support custom volumes and volumeMounts * [AMORO-3970] Update README.md with correct Spark versions for Mixed format (#3978) * [AMORO-3977] Combine amoro-site to prevent docs loose sync (#3979) * clone site * remove invalid soft links * docs soft link * Update site/README.md with clear structure documentation - Reorganized documentation to show the versioned and non-versioned content structure - Provided a clear directory tree representation of the site structure - Updated instructions for running the documentation site locally - Added section on testing both sites together 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com> * Update site/README.md with clear structure documentation - Fixed references to make it clear this is part of the main repository - Updated section titles to match the new organization - Clarified paths for versioned and non-versioned content - Simplified local development instructions 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com> * github action * ignore * rat exclude site * test * test please * Revert "test please" This reverts commit 7c536fe. * label --------- Co-authored-by: Claude <noreply@anthropic.com> * [Improvement]: Disable verification for helm-unittest plugin installation (#3993) Disable verification for helm-unittest plugin installation * [AMORO_3990] Add package chart instructions to README (#3991) * [Feature] Introduce a new framework that supports scheduling for Formats and Processes (#3924) * process service poc * Refactor table process framework * Optimize and refactor the process service framework & support framework test case --------- Co-authored-by: dailiang <dailiang@bytedance.com> Co-authored-by: zhangyongxiang.alpha <zhangyongxiang.alpha@bytedance.com> Co-authored-by: baiyangtx <xiangnebula@163.com> * [AMORO-3981] Automatic generate amoro configuration docs (#3982) * auto generate ams config * add auto generate notice * junit 5 * convertToString * Update CONTRIBUTING.md --------- Co-authored-by: big face cat <731030576@qq.com> * [AMORO-3981][FOLLOWUP] Refine the duration to string format (#3987) * [AMORO-3981][FOLLOWUP] Refine duration type parse and convert to string * Revert "[AMORO-3981][FOLLOWUP] Refine duration type parse and convert to string" This reverts commit 877bcb9. * format duration with TimeUtils.formatWithHighestUnit * largest unit * [AMORO-1258] Support Zookeeper kerberos authentication (#3980) * [AMORO-1258] Support Zookeeper kerberos authentication * remove minkdc * Revert "remove minkdc" This reverts commit 466d632. * docs * [AMORO-3934] Manage com.fasterxml.jackson dependencies in dependencyManagement (#3936) * jackson version * deps * fasterxml.jackson.version --------- Co-authored-by: ZhouJinsong <zhoujinsong0505@163.com> * [AMORO-3866] Using shaded hadoop-client-api and hadoop-client-runtime for hadoop3 (#3983) hadoop client api and hadoop client runtime remove guava test replace exclude remove runtime * [AMORO-3632] Refine data expiration literal calculation for date type (#3964) * [Feature]: data-retention, add support for partition column type Date #3632 * [AMORO-3632]: data-retention, add support for partition column type Date * [AMORO-3632]: data-retention, add support for partition column type Date * [AMORO-3632]: data-retention, add support for partition column type Date * [AMORO-3632] Fix #3665: Prevent long overflow in Date expiration calculation and fix related tests --------- * [AMORO-3885] fix missing oss.endpoint for internal mixed_iceberg catalog (#3886) * fix missing oss.endpoint for internal mixed_iceberg catalog * fix missing oss.endpoint for internal mixed_iceberg catalog --------- Co-authored-by: ZhouJinsong <zhoujinsong0505@163.com> Co-authored-by: ConradJam <jam.gzczy@gmail.com> * [AMORO-2635] Enhance table partition files list performance (#4003) * [AMORO-2635] * UT * UT for fallback * address comments * TODO * [AMORO-3628] Add user logo wall on the home page. (#4010) * [AMORO-3628] Add users logo wall in the home page * update Co-authored-by: Claude * [AMORO-3804] Skip RUNTIME_CONTEXT_CACHE for TableMetaStore with local configuration (#4005) * [AMORO-3804] Skip RUNTIME_CONTEXT_CACHE for TableMetaStore with local configuration * return * test * configuration * create runtime context directly * remove unneeded tests * [AMORO-4011] Fix JUnit 4 tests skipped (#4017) * add back * test * revert test * deps * [AMORO-3775] Add support for metric-based refresh event trigger in TableRuntimeRefreshExecutor (#3776) [AMORO-3775] Add support for metadata-based refresh event in TableRuntimeRefreshExecutor * [AMORO-3998]support DATABASE HA SERVICE (#3997) support DATABASE HA SERVICE Co-authored-by: dailiang <dailiang@bytedance.com> * [Hotfix] Fix the deploy site GitHub workflow (#4020) * Fix deploy site GitHub workflow * Add publish information in the .asf.yaml * [Hotfix] Enable manual triggering for site deployment workflow (#4021) * [Hotfix]Refactor the build command in the `Publish Docker Image` GitHub workflow to reduce image size (#4023) Refactor the build command in the Publish DOcker Image GitHub workflow to reduce image size * [Hotfix] Fix typos (#4026) * [AMORO-3531] Drop support for java8 (#3899) drop support jdk8 Co-authored-by: Xu Bai <xuba@apache.org> Co-authored-by: ConradJam <jam.gzczy@gmail.com> Co-authored-by: ZhouJinsong <zhoujinsong0505@163.com> * [AMORO-3940] Flink config load will cause ClassCastException when dir… (#3941) [AMORO-3940] Flink config load will cause ClassCastException when directly return result of Yaml load Co-authored-by: ConradJam <jam.gzczy@gmail.com> * [AMORO-3943] Shade all third party classes for spark runtime (#3942) * relocate * filter * [AMORO-4027] Bump shade plugin to fix compilation failure on JDK 17 (#4028) (cherry picked from commit 83e5272) * [AMORO-3973] Support spark3.4 for mixed format (#4013) * Copy the code form spark3.5 * Optimize code * Maintain consistency in the references to scala-compiler and scala-library across Spark 3.5, Spark 3.4, and Spark 3.3 * fixup --------- Co-authored-by: 张文领 <zhangwl9@chinatelecom.cn> Co-authored-by: Xu Bai <tocreationbai@gmail.com> * [Hotfix] Try to fix site-deploy permission problem (#4025) * [AMORO-4022] [Improvement]: AMS Iceberg maintainer moved to the amoro-iceberg module (#4024) [AMORO-4022] AMS Iceberg maintainer moved to the amoro-iceberg module (#4022) * [AMORO-3994] Support Exposing AMS High Availability (HA) Status (#3996) [amoro-3994] Support Exposing AMS High Availability (HA) Status Co-authored-by: davedwwang <davedwwang@tencent.com> * [Feature]: Support LDAP Authentication for Dashboard Login (#4009) Co-authored-by: davedwwang <davedwwang@tencent.com> * [HotFix] Remove duplicate references to scala-library and scala-compiler (#4033) Remove duplicate references to scala-library and scala-compiler Co-authored-by: 张文领 <zhangwl9@chinatelecom.cn> * [HotFix] Shade third party classes for spark-3.4 runtime (#4032) Shade third party classes for spark runtime 3.4 Co-authored-by: 张文领 <zhangwl9@chinatelecom.cn> * [Subtask]: Use a new configuration item to control whether master & slave mode is enabled. #3845 * [Subtask]: Use a new configuration item to control whether master & slave mode is enabled. #3845 * [Subtask]: Use a new configuration item to control whether master & slave mode is enabled. #3845 * [Subtask]: Use a new configuration item to control whether master & slave mode is enabled. #3845 * [Subtask]: Use a new configuration item to control whether master & slave mode is enabled. #3845 * [AMORO-3998]support DATABASE HA SERVICE (#3997) support DATABASE HA SERVICE Co-authored-by: dailiang <dailiang@bytedance.com> * [AMORO-3994] Support Exposing AMS High Availability (HA) Status (#3996) [amoro-3994] Support Exposing AMS High Availability (HA) Status Co-authored-by: davedwwang <davedwwang@tencent.com> * [Subtask]: Use a new configuration item to control whether master & slave mode is enabled. #3845 * [Subtask]: Use a new configuration item to control whether master & slave mode is enabled. #3845 * [Subtask]: Optimize the description information for use-master-slave-mode. --------- Co-authored-by: wardli <wardli@tencent.com> Co-authored-by: Fei Wang <fwang12@ebay.com> Co-authored-by: ZhouJinsong <zhoujinsong0505@163.com> Co-authored-by: yeatsliao <liaoyt66066@gmail.com> Co-authored-by: Xu Bai <xuba@apache.org> Co-authored-by: xuzifu666 <xuzifu666@gmail.com> Co-authored-by: simonsssu <barley0806@gmail.com> Co-authored-by: zhangwl9 <1298877813@qq.com> Co-authored-by: 张文领 <zhangwl9@chinatelecom.cn> Co-authored-by: zhan7236 <76658920+zhan7236@users.noreply.github.com> Co-authored-by: Abhishek Pathania <theabhishekpathania@gmail.com> Co-authored-by: xykera <akrai9554@gmail.com> Co-authored-by: Claude <noreply@anthropic.com> Co-authored-by: tcodehuber <tcodehuber@gmail.com> Co-authored-by: LiangDai-Mars <dd574852610@gmail.com> Co-authored-by: dailiang <dailiang@bytedance.com> Co-authored-by: zhangyongxiang.alpha <zhangyongxiang.alpha@bytedance.com> Co-authored-by: baiyangtx <xiangnebula@163.com> Co-authored-by: big face cat <731030576@qq.com> Co-authored-by: davedwwang <iverson89w@163.com> Co-authored-by: ConradJam <jam.gzczy@gmail.com> Co-authored-by: Xu Bai <tocreationbai@gmail.com> Co-authored-by: Jzjsnow <snow.jiangzj@gmail.com> Co-authored-by: leosanqing <liurongtong001@qq.com> Co-authored-by: leosanqing <stormleo@qq.com> Co-authored-by: Sebb <sebbASF@users.noreply.github.com> Co-authored-by: davedwwang <davedwwang@tencent.com>
1 parent c69eff4 commit ba7f927

File tree

8 files changed

+63
-11
lines changed

8 files changed

+63
-11
lines changed

amoro-ams/src/main/java/org/apache/amoro/server/AmoroManagementConf.java

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -53,6 +53,15 @@ public class AmoroManagementConf {
5353
.defaultValue("admin")
5454
.withDescription("The administrator password");
5555

56+
/** Enable master & slave mode, which supports horizontal scaling of AMS. */
57+
public static final ConfigOption<Boolean> USE_MASTER_SLAVE_MODE =
58+
ConfigOptions.key("use-master-slave-mode")
59+
.booleanType()
60+
.defaultValue(false)
61+
.withDescription(
62+
"This setting controls whether to enable the AMS horizontal scaling feature, "
63+
+ "which is currently under development and testing.");
64+
5665
public static final ConfigOption<Duration> CATALOG_META_CACHE_EXPIRATION_INTERVAL =
5766
ConfigOptions.key("catalog-meta-cache.expiration-interval")
5867
.durationType()

amoro-ams/src/main/java/org/apache/amoro/server/AmoroServiceContainer.java

Lines changed: 26 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -18,6 +18,8 @@
1818

1919
package org.apache.amoro.server;
2020

21+
import static org.apache.amoro.server.AmoroManagementConf.USE_MASTER_SLAVE_MODE;
22+
2123
import io.javalin.Javalin;
2224
import io.javalin.http.HttpCode;
2325
import io.javalin.http.staticfiles.Location;
@@ -99,6 +101,7 @@ public class AmoroServiceContainer {
99101
public static final Logger LOG = LoggerFactory.getLogger(AmoroServiceContainer.class);
100102

101103
public static final String SERVER_CONFIG_FILENAME = "config.yaml";
104+
private static boolean IS_MASTER_SLAVE_MODE = false;
102105

103106
private final HighAvailabilityContainer haContainer;
104107
private DataSource dataSource;
@@ -133,17 +136,24 @@ public static void main(String[] args) {
133136
LOG.info("AMS service has been shut down");
134137
}));
135138
service.startRestServices();
136-
while (true) {
137-
try {
138-
// Used to block AMS instances that have not acquired leadership
139-
service.waitLeaderShip();
140-
service.transitionToLeader();
141-
// Used to block AMS instances that have acquired leadership
142-
service.waitFollowerShip();
143-
} catch (Exception e) {
144-
LOG.error("AMS start error", e);
145-
} finally {
146-
service.transitionToFollower();
139+
if (IS_MASTER_SLAVE_MODE) {
140+
// Even if one does not become the master, it cannot block the subsequent logic.
141+
service.registAndElect();
142+
// Regardless of whether tp becomes the master, the service needs to be activated.
143+
service.startOptimizingService();
144+
} else {
145+
while (true) {
146+
try {
147+
// Used to block AMS instances that have not acquired leadership
148+
service.waitLeaderShip();
149+
service.transitionToLeader();
150+
// Used to block AMS instances that have acquired leadership
151+
service.waitFollowerShip();
152+
} catch (Exception e) {
153+
LOG.error("AMS start error", e);
154+
} finally {
155+
service.transitionToFollower();
156+
}
147157
}
148158
}
149159
} catch (Throwable t) {
@@ -152,6 +162,10 @@ public static void main(String[] args) {
152162
}
153163
}
154164

165+
public void registAndElect() throws Exception {
166+
haContainer.registAndElect();
167+
}
168+
155169
public enum HAState {
156170
INITIALIZING(0),
157171
FOLLOWER(1),
@@ -306,6 +320,7 @@ public void dispose() {
306320
private void initConfig() throws Exception {
307321
LOG.info("initializing configurations...");
308322
new ConfigurationHelper().init();
323+
IS_MASTER_SLAVE_MODE = serviceConfig.getBoolean(USE_MASTER_SLAVE_MODE);
309324
}
310325

311326
public Configurations getServiceConfig() {

amoro-ams/src/main/java/org/apache/amoro/server/ha/DataBaseHighAvailabilityContainer.java

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -147,6 +147,9 @@ public void close() {
147147
}
148148
}
149149

150+
@Override
151+
public void registAndElect() throws Exception {}
152+
150153
private class HeartbeatRunnable implements Runnable {
151154
@Override
152155
public void run() {

amoro-ams/src/main/java/org/apache/amoro/server/ha/HighAvailabilityContainer.java

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -42,4 +42,12 @@ public interface HighAvailabilityContainer {
4242

4343
/** Closes the container and releases resources. */
4444
void close();
45+
46+
/**
47+
* In master-slave mode, this is used for AMS nodes to register and participate in the master
48+
* election process.
49+
*
50+
* @throws Exception If registration fails or participation in the primary election fails.
51+
*/
52+
void registAndElect() throws Exception;
4553
}

amoro-ams/src/main/java/org/apache/amoro/server/ha/NoopHighAvailabilityContainer.java

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -46,4 +46,7 @@ public void waitFollowerShip() throws InterruptedException {
4646
public void close() {
4747
LOG.info("Noop HA: closed");
4848
}
49+
50+
@Override
51+
public void registAndElect() throws Exception {}
4952
}

amoro-ams/src/main/java/org/apache/amoro/server/ha/ZkHighAvailabilityContainer.java

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -140,6 +140,11 @@ public void waitLeaderShip() throws Exception {
140140
LOG.info("Became the leader of AMS");
141141
}
142142

143+
@Override
144+
public void registAndElect() throws Exception {
145+
// TODO Here you can register for AMS and participate in the election.
146+
}
147+
143148
@Override
144149
public void waitFollowerShip() throws Exception {
145150
LOG.info("Waiting to become the follower of AMS");

amoro-ams/src/test/java/org/apache/amoro/server/TestInternalMixedCatalogService.java

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -189,6 +189,14 @@ public void before() {
189189
@AfterEach
190190
public void after() {
191191
LOG.info("Test finished.");
192+
try {
193+
// explicitly clean up possible residual table runtime records
194+
if (catalog.tableExists(tableIdentifier)) {
195+
catalog.dropTable(tableIdentifier, true);
196+
}
197+
} catch (Exception e) {
198+
LOG.warn("Failed to drop table during cleanup", e);
199+
}
192200
catalog.dropDatabase(database);
193201
}
194202

docs/configuration/ams-config.md

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -125,6 +125,7 @@ table td:last-child, table th:last-child { width: 40%; word-break: break-all; }
125125
| thrift-server.selector-thread-count | 2 | The number of selector threads for the Thrift server. |
126126
| thrift-server.table-service.bind-port | 1260 | Port that the table service thrift server is bound to. |
127127
| thrift-server.table-service.worker-thread-count | 20 | The number of worker threads for the Thrift server. |
128+
| use-master-slave-mode | false | This setting controls whether to enable the AMS horizontal scaling feature, which is currently under development and testing. |
128129

129130

130131
## Shade Utils Configuration

0 commit comments

Comments
 (0)