Skip to content

Conversation

@gnehil
Copy link
Contributor

@gnehil gnehil commented Nov 18, 2025

Proposed changes

Issue Number: close #xxx

Problem Summary:

When writing column names containing Chinese or other Unicode characters, stream load will return the following exception information:

{
    "TxnId": 2060,
    "Label": "spark-doris-0-0-0-1763547930966",
    "Comment": "",
    "TwoPhaseCommit": "false",
    "Status": "Fail",
    "Message": "[ANALYSIS_ERROR]TStatus: errCode = 2, detailMessage = Duplicate column: ??",
    "NumberTotalRows": 0,
    "NumberLoadedRows": 0,
    "NumberFilteredRows": 0,
    "NumberUnselectedRows": 0,
    "LoadBytes": 0,
    "LoadTimeMs": 0,
    "BeginTxnTimeMs": 0,
    "StreamLoadPutTimeMs": 1,
    "ReadDataTimeMs": 0,
    "WriteDataTimeMs": 0,
    "ReceiveDataTimeMs": 0,
    "CommitAndPublishTimeMs": 0
}

You can now set doris.sink.http-utf8-charset to true to configure the HTTP client to use the UTF-8 character set, allowing Unicode characters to be included in the header.

Checklist(Required)

  1. Does it affect the original behavior: (Yes/No/I Don't know)
  2. Has unit tests been added: (Yes/No/No Need)
  3. Has document been added or modified: (Yes/No/No Need)
  4. Does it need to update dependencies: (Yes/No)
  5. Are there any changes that cannot be rolled back: (Yes/No)

Further comments

If this is a relatively large or complex change, kick off the discussion at [email protected] by explaining why you chose the solution you did and what alternatives you considered, etc...

@gnehil gnehil force-pushed the dev-ignore-columns branch 2 times, most recently from 98930d0 to f5492fc Compare November 19, 2025 07:09
@gnehil gnehil force-pushed the dev-ignore-columns branch from f8d8ada to 8c02efc Compare November 19, 2025 10:23
@gnehil gnehil changed the title [improve] Supports removing the forced addition of columns header during writes [improve] Supports http request use utf8 charset Nov 19, 2025
Copy link
Member

@JNSimba JNSimba left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants