Utf8 support #1948
Merged
Utf8 support #1948
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
jpwhite4
force-pushed
the
utf8_support
branch
3 times, most recently
from
08c6131 to
6085568
Compare
This was referenced Dec 16, 2024
Closed
jpwhite4
force-pushed
the
utf8_support
branch
from
fcdfaca to
6c86451
Compare
jpwhite4
requested review from
aaronweeden,
eiffel777 and
rvtovar
and removed request for
aaronweeden and
rvtovar
The order of data returned by the rawData endpoint is not deterministic so we don't need the test to check the ordering.
jpwhite4
requested review from
eiffel777
and removed request for
aaronweeden and
eiffel777
eiffel777
approved these changes
Oct 21, 2025
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
We've had problems with data encoding for a long time now. This should fix it properly by setting UTF8 as the only encoding supported by XDMoD.
Prior versions of Open XDMoD did not specify the character encoding and used the database
software default values for the database tables (which was latin1
for MariaDB 10). The ingestor and shredder pipelines used different character encoding
(some used UTF8 encoding, some the database connection defaults). This meant that non
ASCII characters could be incorrectly displayed in XDMoD.
This upgrade changes Open XDMoD database tables to support UTF8 character encoding and
updates the shredder and ingestor to process UTF8 encoded files. This ensures
that information in new logs loaded into Open XDMoD will display correctly.