Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Added support to write metadata files in parquet #5105

Merged
merged 50 commits into from
Apr 2, 2024
Merged
Changes from 1 commit
Commits
Show all changes
50 commits
Select commit Hold shift + click to select a range
f8ae4ef
Copied methods from inside parquet hadoop to write metadata files
malhotrashivam Jan 30, 2024
0ef7f79
Calling methods from inside ParquetHadoop for writing metadata files
malhotrashivam Jan 31, 2024
44d031f
Cleaned up the code and added a lot of TODOs for review
malhotrashivam Feb 13, 2024
6394bac
Merge branch 'main' into sm-pq-metadata
malhotrashivam Feb 13, 2024
6c6b075
Some more changes
malhotrashivam Feb 13, 2024
9263aa0
WIP commit
malhotrashivam Feb 14, 2024
8c887c4
Added a custom metadata file writer
malhotrashivam Feb 16, 2024
05400c5
Merge branch 'main' into sm-pq-metadata
malhotrashivam Feb 16, 2024
b696649
Minor fix
malhotrashivam Feb 16, 2024
797b3dc
Fixed failing test
malhotrashivam Feb 17, 2024
d987a06
Moved some code around
malhotrashivam Feb 20, 2024
d4da175
Minor change
malhotrashivam Feb 20, 2024
f33d712
Review comments
malhotrashivam Feb 22, 2024
7d66445
Merge branch 'main' into sm-pq-metadata
malhotrashivam Feb 26, 2024
238f5f5
Read offset index from column chunk on demand
malhotrashivam Feb 26, 2024
7a3652d
Fixed failing test
malhotrashivam Feb 26, 2024
1f41eef
Added support for partitioned parquet writing
malhotrashivam Feb 27, 2024
c8aa764
Added some more tests
malhotrashivam Feb 27, 2024
ff99a36
Added some more tests
malhotrashivam Feb 27, 2024
3e48937
Added a new API for writing a partitioned table directly
malhotrashivam Feb 28, 2024
e584343
Improved the tests
malhotrashivam Feb 29, 2024
f659f3c
Review with Ryan part 1
malhotrashivam Mar 4, 2024
3651e5f
Added more tests
malhotrashivam Mar 4, 2024
51eefe1
Iterating using chunked iterators
malhotrashivam Mar 4, 2024
60ee1f5
Removed some unnecessary includes
malhotrashivam Mar 4, 2024
5c7353f
Added support for {index} and {partition} in file basename
malhotrashivam Mar 4, 2024
c174452
Review comments
malhotrashivam Mar 5, 2024
ab73df0
Minor touchups
malhotrashivam Mar 5, 2024
1cb8b81
Added fix and tests for big decimals
malhotrashivam Mar 6, 2024
20e8204
Updated a comment
malhotrashivam Mar 6, 2024
9892d14
Review with Ryan part 1
malhotrashivam Mar 7, 2024
1d98927
Review with Ryan part 2
malhotrashivam Mar 11, 2024
5018feb
Minor touchups
malhotrashivam Mar 11, 2024
a529fc2
Fixed failing tests
malhotrashivam Mar 15, 2024
48906dd
Merge branch 'main' into sm-pq-metadata
malhotrashivam Mar 15, 2024
819a1b9
Added python APIs and improved comments
malhotrashivam Mar 16, 2024
10b7e0d
Added more fixes for python
malhotrashivam Mar 19, 2024
9f7c55e
Review with Ryan and Chip
malhotrashivam Mar 21, 2024
b62abb7
Merge branch 'main' into sm-pq-metadata
malhotrashivam Mar 21, 2024
0d7c62e
Review with Chip part 2
malhotrashivam Mar 22, 2024
5a3de8e
Review with Chip and Jianfeng Part 3
malhotrashivam Mar 22, 2024
f68551f
Review with Chip and Jianfeng continued
malhotrashivam Mar 22, 2024
b11ebd1
Added new APIs for managing indexes
malhotrashivam Mar 25, 2024
3bdc92b
Trigger CI jobs
malhotrashivam Mar 25, 2024
3dd6097
Review with Ryan
malhotrashivam Mar 26, 2024
d8bc0a5
Added python support for writing indexes
malhotrashivam Mar 27, 2024
f700883
Reordered comments
malhotrashivam Mar 27, 2024
6246289
Added more details to python comments
malhotrashivam Mar 27, 2024
fa2b0c5
Moved from list to sequence
malhotrashivam Mar 27, 2024
7909ea0
Added fixes and tests for windows
malhotrashivam Apr 1, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Review comments
malhotrashivam committed Feb 22, 2024
commit f33d712ec22e37ef51acf2029274ffb46321b14a
Original file line number Diff line number Diff line change
@@ -61,25 +61,24 @@ public class ParquetTableWriter {
/**
* Helper struct used to pass information about where to write the grouping files for each grouping column
*/
public static class GroupingColumnWritingInfo {
static class GroupingColumnWritingInfo {
/**
* Parquet name of this grouping column
*/
public final String parquetColumnName;
final String parquetColumnName;
/**
* File path to be added in the grouping metadata of main parquet file
*/
public final File metadataFilePath;
final File metadataFilePath;

/**
* Destination path for writing the grouping file. The two filenames can differ because we write grouping files
* to shadow file paths first and then place them at the final path once the write is complete. But the metadata
* should always hold the accurate path.
*/
public final File destFile;
final File destFile;

public GroupingColumnWritingInfo(final String parquetColumnName, final File metadataFilePath,
final File destFile) {
GroupingColumnWritingInfo(final String parquetColumnName, final File metadataFilePath, final File destFile) {
this.parquetColumnName = parquetColumnName;
this.metadataFilePath = metadataFilePath;
this.destFile = destFile;
@@ -97,6 +96,10 @@ public GroupingColumnWritingInfo(final String parquetColumnName, final File meta
* {@code destFile} if we are writing the parquet file to a shadow location first since the metadata should
* always hold the accurate path.
* @param incomingMeta A map of metadata values to be stores in the file footer
* @param groupingColumnsWritingInfoMap A map of grouping column names to their respective info used for writing
* grouping files
* @param metadataFileWriter The writer for the _metadata and _common_metadata files
*
* @throws SchemaMappingException Error creating a parquet table schema for the given table (likely due to
* unsupported types)
* @throws IOException For file writing related errors
@@ -161,6 +164,7 @@ public static void write(
* always hold the accurate path.
* @param tableMeta A map of metadata values to be stores in the file footer
* @param tableInfoBuilder A partially constructed builder for the metadata object
* @param metadataFileWriter The writer for the _metadata and _common_metadata files
malhotrashivam marked this conversation as resolved.
Show resolved Hide resolved
* @throws SchemaMappingException Error creating a parquet table schema for the given table (likely due to
* unsupported types)
* @throws IOException For file writing related errors
@@ -279,6 +283,7 @@ private static Table pretransformTable(@NotNull final Table table, @NotNull fina
* @param writeInstructions write instructions for the file
* @param tableMeta metadata to include in the parquet metadata
* @param tableInfoBuilder a builder for accumulating per-column information to construct the deephaven metadata
* @param metadataFileWriter The writer for the _metadata and _common_metadata files
*
* @return a new file writer
*/