Skip to content

Conversation

@yinbenrong
Copy link

Purpose of this pull request

#7995
Using the zeta engine, data was written to iceberg via mysqlcdc. The checkpoint failed to save after 24 hours, and the error was that the kerberos authentication of hdfs had expired and failed

Does this PR introduce any user-facing change?

yes

How was this patch tested?

The seatunnel.yaml configuration file has added an expiration time for kerberos. Within one expiration period, kerberos authentication will be retweeted.

Add a new configuration “seatunnel.hadoop.dfs.kerberos.ticket.lifetime” in seatunnel.yaml
Sample:

seatunnel:
  engine:
    history-job-expire-minutes: 10080
    backup-count: 1
    queue-type: blockingqueue
    print-execution-info-interval: 60000
    print-job-metrics-info-interval: 60000
    slot-service:
      dynamic-slot: false
      slot-num: 10
    checkpoint:
      interval: 600000
      timeout: 300000
      storage:
        type: hdfs
        max-retained: 1
        plugin-config:
          namespace: /tmp/seatunnel/checkpoint_snapshot
          storage.type: hdfs
          fs.defaultFS: hdfs://[NAMESERVICE]
          #fs.defaultFS: file:///
          seatunnel.hadoop.dfs.nameservices: [NAMESERVICE]
          seatunnel.hadoop.dfs.ha.namenodes.[NAMESERVICE]: [NAMENODE1],[NAMENODE2]
          seatunnel.hadoop.dfs.namenode.rpc-address.[NAMESERVICE].[NAMENODE1]: [NAMENODE1_HOST]:8020
          seatunnel.hadoop.dfs.namenode.rpc-address.[NAMESERVICE].[NAMENODE2]: [NAMENODE2_HOST]:8020
          seatunnel.hadoop.dfs.client.failover.proxy.provider.[NAMESERVICE]: org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider
          seatunnel.hadoop.dfs.namenode.kerberos.principal: ENC(加密后的字符串)
          seatunnel.hadoop.dfs.datanode.kerberos.principal: ENC(加密后的字符串)
          seatunnel.hadoop.dfs.kerberos.ticket.lifetime: 86400
          seatunnel.hadoop.rpc.protection: authentication
          kerberosPrincipal: ENC(加密后的字符串)
          kerberosKeytabFilePath: ENC(加密后的字符串)
    telemetry:
      metric:
        enabled: false
      logs:
        scheduled-deletion-enable: true
    http:
      enable-http: true
      port: 8080
      enable-dynamic-port: false
      # Uncomment the following lines to enable basic authentication for web UI
      # enable-basic-auth: true
      # basic-auth-username: admin
      # basic-auth-password: admin

Check list

private static final String STORAGE_TMP_SUFFIX = "tmp";
private static final String STORAGE_TYPE_KEY = "storage.type";

private final Map<String, String> initConfiguration; // 初始配置
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we should use English for comments in the code.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1

@dybyte
Copy link
Contributor

dybyte commented Oct 16, 2025

Please enable CI by following these instructions, and also add an e2e test case.

Comment on lines +60 to +65
private Integer ticketLifetime = 900; //kerberos票据过期时间

public HdfsStorage(Map<String, String> configuration) throws CheckpointStorageException {
this.initStorage(configuration);
// 深拷贝初始配置
this.initConfiguration = deepCopy(configuration);
this.ticketLifetime=Integer.valueOf(StringUtils.defaultString(configuration.get("seatunnel.hadoop.dfs.kerberos.ticket.lifetime"),"900"));
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why is the default value 900? Additionally, the assignment logic is duplicated.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants