v1.2.680

What's Changed

Exciting New Features ✨

• feat(query): optimize min/max aggregation use accurate column stats by @sundy-li in #16270
• feat: new table option data_retention_period_in_hours by @dantengsky in #16266
• feat(query): add config udf_server_allow_insecure by @everpcpc in #16290
• feat(query): Inverted index search function support options by @b41sh in #16256
• feat(geo): add geography data type by @forsaken628 in #16286
• feat(query): Support create/drop dictionary ddl. by @Winnie-Hong0927 in #16280
• feat: add endpoint /v1/session/logout and record client session ID in meta. by @youngsofun in #16281
• feat(query): ST_GEOGRAPHYFROMEWKT by @forsaken628 in #16302
• feat: support temporary table by @SkyFan2002 in #16250
• feat(query): Add jq set returning function for processing Variant data with jq filters by @maxjustus in #16288
• feat: Add KVPbApi::get_id_and_value() by @drmingdrmer in #16325
• feat: Check essential permissions when creating table with external location by @dantengsky in #16315
• feat(query): add is_local to admin /v1/tables info by @everpcpc in #16329
• feat(query): add is_external to /v1/tables info by @everpcpc in #16336
• feat(query): add plan cache by @sundy-li in #16333
• feat: login and renew return session_token_validity_in_secs too. by @youngsofun in #16313
• feat(query): add discovery nodes api by @zhang2014 in #16353
• feat: hilbert clustering by @zhyass in #16296
• feat: add error stack to ErrorCode by @andylokandy in #16343
• feat: support vacuum temporary table by @SkyFan2002 in #16364
• feat: http handler support temp table. by @youngsofun in #16375
• feat: batch size hint of stream consumption by @zhyass in #16372
• feat(query): make external server parallel by batch by @sundy-li in #16390
• feat: estimate selectivity by table sample by @xudong963 in #16362
• feat: introduce an intermediate version that is compatible with vacuum2 by @SkyFan2002 in #16354
• feat(query): Procedure (Part1) by @TCeason in #16348
• feat: add limit push down rule by @xudong963 in #16403
• feat: discovery_nodes not need auth. by @youngsofun in #16392
• feat(query): Support SHOW VARIABLES by @TCeason in #16409
• feat: compact after merge sort by @zhyass in #16401
• feat(query): Support access Redis data from dictionaries via the dict_get function. by @Winnie-Hong0927 in #16389
• feat: continue vacuum drop table on per-table cleanup failures by @SkyFan2002 in #16424
• feat: Load DATABEND_ENTERPRISE_LICENSE_EMBEDDED while building by @Xuanwo in #16440
• feat(query): Support access Mysql data from dictionaries via the dict_get function. by @Winnie-Hong0927 in #16444
• feat: Auto compact(re-cluster) for multiple table insertion statement by @SkyFan2002 in #16443
• feat(query): Procedure Part2 support arguments by @TCeason in #16453
• feat(query): refactor window spill by @Dousir9 in #16448
• feat(query): push rank limit into aggregate partial node by @sundy-li in #16466
• feat: add eliminate union optimizer rule by @xudong963 in #16478
• feat(query): parallel k way merge sort by @forsaken628 in #16340
• feat: add fast path for cross join to optimize tpcds q23 (~50x) by @xudong963 in #16504
• feat(catalog): ignore error when listing databases by @BohuTANG in #16514
• feat: limit stage name to plain_identifier by @youngsofun in #16522
• feat(query): Initial implementation of postgres-compatible number to_char function by @forsaken628 in #16507
• feat(query): Procedure support drop if exists and create or replace by @TCeason in #16500
• feat(query): DATE_ADD Functions to support week as a unit by @TracyZYJ in #16530
• feat: http handler add X-DATABEND-VERSION in each response. by @youngsofun in #16518
• feat: short sql length setting by @arkzuse in #16502
• feat: unset table options by @dantengsky in #16544
• feat(query): window partition by spill to disk by @forsaken628 in #16441
• feat(query): Simplified version Pattern S/L/D/G for number to_char by @forsaken628 in #16569
• feat(query): add json array lambda functions by @sundy-li in #16573
• feat(query): support describe user statement by @b41sh in #16579
• feat: copy option 'pattern' support variable. by @youngsofun in #16525
• feat(query): support json array functions by @b41sh in #16571
• feat(query): add date_diff function by @chagelo in #16560
• feat(query): Reduce serialisation memory usage when spilling to local disk by @forsaken628 in #16580
• feat(query): Support show dictionaries DQL. by @Winnie-Hong0927 in #16602
• feat: support sampling table with block and row level simultaneously by @xudong963 in #16613
• feat: add unload options overwrite, include_query_id and use_raw_path. by @youngsofun in #16614
• feat(function): Add mode Aggregate function. by @Freejww in #16627
• feat(query): Support use parquet format when spilling by @forsaken628 in #16612
• feat(query): support json_object_insert function by @b41sh in #16636
• feat: http handler support request forwarding. by @youngsofun in #16637
• feat: Implemented a new sql explain analyze graphical by @Maricaya in #16543
• feat: add deterministic block level sampling for small datasets by @xudong963 in #16670
• feat: support explain decorrelated plan by @xudong963 in #16681
• feat: add ARRAY_SIZE alias for ARRAY_LENGTH by @ding-young in #16685
• feat(query): support json_object_delete and json_object_pick function by @b41sh in #16682
• feat: Bump OpenDAL to 0.50.1 by @Xuanwo in #16661
• feat: support map lambda functions by @Dragonliu2018 in #16683
• feat(query): add api /v1/verify by @everpcpc in #16665
• feat: support calculating table storage size(including time travel) by @SkyFan2002 in #16698
• feat(query): add table function udf_echo && expose more metrics of udf client by @sundy-li in #16696
• feat(query): Settings clause by @TCeason in #16669
• feat: implement is_not_null selectivity based on null count in stats by @xudong963 in #16730
• feat: HTTP handler use cookie to pass session id to and from clients. by @youngsofun in #16735
• feat: impl opendal::HttpFetch for StorageHttpClient by @everpcpc in #16738
• feat(query): function about convert_timezone #16177 by @florann in #16181
• feat(functions): add new function: map_pick by @hanxuanliang in #15573
• feat(metrics): new storage_http_requests_count by @everpcpc in #16769
• feat(functions): add map_insert function by @hanxuanliang in #15567
• feat: filter null value before join by @xudong963 in #16722
• feat(query): Support rename dictionary ddl. by @Winnie-Hong0927 in #16754
• feat(query): add last_day, previous_day, next_day function by @Freejww in #16740
• feat: implement StringColumn using StringViewArray by @andylokandy in #16610
• feat: Implement WAL-based RaftLog storage by @drmingdrmer in #16776
• feat(query): expand trim/ltrim/rtrim by @TCeason in #16802
• feat: support pushdown predicate into iceberg engine by @sundy-li in #16650
• feat(query): TopN window operator by @forsaken628 in #16726
• feat(query): support SHOW DROP DATABASES by @TCeason in #16811
• feat: uuid function from v4 to v7 by @BohuTANG in #16827
• feat: refine the rule PushDownFilterWindowTopN in case top n is equal to 0. by @forsaken628 in #16830
• feat: add aws glue as an iceberg connection type by @Rowlandev in #16824
• feat(query): add distributed pruning settings by @zhang2014 in #16864
• feat: databend-meta adds more metrics about raft-log by @drmingdrmer in #16884
• feat(query): speed up fetching redis data from dictionaries via mget. by @Dragonliu2018 in #16766
• feat(query): add new function TO_TIMESTAMP<int, scale> by @TCeason in #16924
• feat: disable specifying copy options when create stage. by @youngsofun in #16925
• feat(doc): Introducing Databend Guru on Gurubase.io by @kursataktas in #16773
• feat(query): Support geometry relation functions by @b41sh in #16927
• feat(query): virtual column allow cast to other type by @b41sh in #16903
• feat: copy support option COLUMN_MATCH_MODE by @youngsofun in #16963
• feat(query): speed up fetching mysql data from dictionaries via batch processing. by @Dragonliu2018 in #16948
• feat: Support USE CATALOG syntax and current_catalog() function by @flashmouse in #16926
• feat(query): Support st_collect aggregate function by @b41sh in #16996
• feat(query): Support flight_connection_max_retry_times and flight_connection_retry_interval setting by @b41sh in #16856
• feat: show dropped table and data_retention_time_in_hours in fuse_tim… by @SkyFan2002 in #17015
• feat(rbac): add GrantObject::Warehouse by @TCeason in #17029
• feat(query): partition sort spill by @forsaken628 in #16987
• feat: support vacuum leaked table data by @SkyFan2002 in #17022
• feat(query): to_binary function support variant, bitmap, geometry, geography types by @b41sh in #17026
• feat(query): add interval type by @TCeason in #16990
• feat: add key prefix count assertions to transactions by @drmingdrmer in #17053
• feat(query): null type infer as Nullabel(String) by default in create as statement by @TCeason in #17052
• feat(query): support global network policy by @everpcpc in #17050
• feat(query): add config network_policy_whitelist by @everpcpc in #17065
• feat(query): pre-check cast type by @andylokandy in #16836
• feat(functions): add jaro_winkler string similarity function by @maxjustus in #16993
• feat(query): add config jwks_refresh_interval & jwks_refresh_timeout by @everpcpc in #17087
• feat: databend-meta transaction support generic bool-expression and else-if chain by @drmingdrmer in #17064
• feat(query): refactor vacuum temp files by @sundy-li in #17089
• refactor: Serialize iceberg table inside engine options to reduce catalog requests by @Xuanwo in #17049
• feat(query): support interval function by @TCeason in #17093
• feat: new settings stream_consume_batch_size_hint by @dantengsky in #17102
• feat(query): support javascript/python script User Defined Aggregate Function by @forsaken628 in #17054
• feat: databend-meta watch API provides initial_flush field by @drmingdrmer in #17098
• feat(query): support javascript/python script User Defined Aggregate Function by @forsaken628 in #17108
• feat(query): support interval 'string' by @TCeason in #17105
• feat(query): Add external_block_rows metrics by @b41sh in #17116
• feat: support vacuum leaked table data (#17022)-Revert by @dantengsky in #17123

Thoughtful Bug Fix 🔧

• fix: max_wait_time in http handler not work when result set is large … by @youngsofun in #16267
• fix(query): enable distribute query for system log table by @zhang2014 in #16300
• fix(query): fix child_has_window by @sundy-li in #16301
• fix: uri path used as both path and root. by @youngsofun in #16321
• fix: vacuum dry run should carry on when target objects are missing by @dantengsky in #16322
• fix(query): fix session leak in ctas by @zhang2014 in #16337
• fix: When Replacing column mask, old name->id->value should be completely deleted by @drmingdrmer in #16328
• fix: reduce sample possibility to avoid flaky test by @xudong963 in #16330
• fix(query): fix IFNULL by @sundy-li in #16339
• fix(query): fix wrong result for join system tables in cluster mode by @zhang2014 in #16342
• fix(query): fix try cast by @sundy-li in #16341
• fix(query): fix binary default string by @sundy-li in #16345
• fix: push filter down join with materialized cte by @xudong963 in #16349
• fix(query): fix aggregate panic in cluster mode by @zhang2014 in #16319
• fix(query): fix incorrect alias of select items by @sundy-li in #16368
• fix: separate sessions for temporary tables with memory engine by @SkyFan2002 in #16374
• fix(executor): enable backpressure for union all by @zhang2014 in #16388
• fix: hilbert_index function panic by @zhyass in #16393
• fix(iceberg): Handling options like "s3.region" by @Xuanwo in #16404
• fix: non-default catalog should not call get_table_function by @Xuanwo in #16410
• fix(query): fix stack overflow for optimizer by @zhang2014 in #16420
• fix: add custom (de)serialization methods for special float value by @dqhl76 in #16258
• fix: table lock flaky test by @zhyass in #16429
• fix(query): The columns permission of the system library is misjudged. by @TCeason in #16431
• fix: restore parser backrtace when map_res failed by @andylokandy in #16434
• fix(iceberg): Table desc should allow nested db_name by @Xuanwo in #16414
• fix: split if block too big during append by @zhyass in #16435
• fix: flaky test 09_0036_merge_into_without_dist by @dantengsky in #16457
• fix: wrong order by result by @xudong963 in #16422
• fix: Multi-thread recreate view/table gives error by @zhyass in #16465
• fix: log query detail and profile with identical format by @everpcpc in #16469
• fix: revert log format breakage from #16249 by @andylokandy in #16470
• fix: typo for query & profile logging config by @everpcpc in #16476
• chore: add extra bracket for and and or to make explain clear by @xudong963 in #16494
• fix(query): fix planner cache cause modify async function does not take effect by @b41sh in #16510
• fix: ignore_result doesn't have an effect on union by @xudong963 in #16515
• fix(query): fix copy into table with disable distributed copy into by @zhang2014 in #16529
• fix(query): fix column leaf_index by @Dousir9 in #16537
• fix: Only when all tables are included, the db can be dropped by @drmingdrmer in #16539
• fix: move TableInfo building out of SchemaApi by @drmingdrmer in #16548
• fix: identify failed vacuum db by id by @drmingdrmer in #16553
• fix(query): fix json object compare cause panic by @b41sh in #16555
• fix: direct io syscall not available on mac by @forsaken628 in #16574
• fix: window in scalar subquery returns wrong results by @xudong963 in #16567
• fix: avoid eliminating the left child of union in RuleEliminateUnion by @xudong963 in #16584
• fix(query): fix inverted index panic caused by null value by @b41sh in #16576
• fix(query): fix cannot obtain stack trace when segmentation fault by @zhang2014 in #16566
• fix: merge into panic by @zhyass in #16581
• fix(query): list_dictionaries should not ignore db_id by @TCeason in #16587
• fix(iceberg): Make sure iceberg table has been loaded by @Xuanwo in #16593
• fix: Disable prewhere for iceberg table for now by @Xuanwo in #16597
• fix(query): fix left outer join wrong result by @Dousir9 in #16601
• fix: subquery returns wrong results under cluster by @xudong963 in #16606
• fix(query): ntile frame should be unbounded by @TCeason in #16604
• fix: lock mgr may failed to release lock in time by @dantengsky in #16611
• fix(query): support subquery in pivot by @Dragonliu2018 in #16631
• fix(query): check_timestamp/date -> clamp_timestamp/date by @TCeason in #16634
• fix: deadlock in table lock by @zhyass in #16632
• fix(query): sort spilling use arrow file format painc by @forsaken628 in #16658
• fix(query): create table need fail if storage format is invalid by @TCeason in #16663
• fix(query): forbid explain explain statement by @TCeason in #16654
• fix(storage): fix refresh virtual column using async api by @b41sh in #16656
• fix(query): insert with long sql parse error by @sundy-li in #16678
• fix(ci): flaky test by @zhyass in #16664
• fix: insert long sql again by @sundy-li in #16684
• fix(query): sort spilling may hang by @forsaken628 in #16672
• fix: JEAllocator should properly handle nullptr returned by @dantengsky in #16692
• fix(metrics): use f64::MAX as histogram bound to display +Inf properly by @flaneur2020 in #16701
• fix(query): use custom connector for udf client by @everpcpc in #16697
• fix: MetaGrpcClient deadlock when drop by @drmingdrmer in #16727
• fix: date_add/sub need respect timezone by @TCeason in #16721
• fix(query): remove single quotes for system.columns comment by @TCeason in #16732
• fix: HTTP handle lost rows when result set is wide and large. by @youngsofun in #16748
• fix: keep column statistics of all NULL column by @dantengsky in #16753
• fix: Files::remove_file_in_batch should not swallow errors by @dantengsky in #16761
• fix(query): add tls config for udf client by @everpcpc in #16782
• fix: token and cookie session id miss match. by @youngsofun in #16786
• fix: clustering_information panic if string domain is none by @zhyass in #16792
• fix(query): fix broken file log layout by @zhang2014 in #16803
• fix: re-ensure dir before upgrading by @drmingdrmer in #16805
• fix: purge may not work on tables after a flash back operation by @SkyFan2002 in #16812
• fix(storage): recluster endless loop by @zhyass in #16831
• fix: incorrect table data disk cache key by @dantengsky in #16837
• fix(query): fix group by with alias column can't bind the column by @b41sh in #16804
• fix(cluster): fix the possibility of connection leak when the cluster state is broken. by @zhang2014 in #16842
• fix(query): fix aggregator_groups_builder to work with string view by @sundy-li in #16843
• fix(query): check storage request method with arg list-type by @everpcpc in #16849
• fix: replace risky dma_buffer_as_vec implementations by @forsaken628 in #16829
• fix(query): fix and check total_buffer_len and total_bytes_len by @sundy-li in #16854
• fix: raft-log wont write when exceeding 256M by @drmingdrmer in #16876
• fix(query): fix distinct set-returning function by @b41sh in #16883
• fix(query): fix incorrect total_bytes_len in string view by @sundy-li in #16877
• fix: StreamVersionMismatched when using CTAS in multi statement trans… by @SkyFan2002 in #16889
• fix(ci): flaky test by @zhyass in #16898
• fix(storge): write progress in compact hook by @zhyass in #16901
• fix: ensure atomicity in create_lock_revision by @zhyass in #16907
• fix: dropped temporary table should not be shown in system.temporary_tables by @SkyFan2002 in #16911
• fix(query): join predict use cast_expr_to_non_null_boolean by @sundy-li in #16937
• fix(ci): flaky test by @zhyass in #16933
• fix(query): add_hours function may panic if the argument is too big by @TCeason in #16929
• fix: lock attach table's snapshot location by @dantengsky in #16952
• fix(ci): fix release dbg upload path by @zhang2014 in #16958
• fix(query): make memory engine as non-local table by @sundy-li in #16955
• fix: solve some inconsistencies between code and comments by @YichiZhang0613 in #16974
• fix: fix segfault in list_domain by @forsaken628 in #16979
• fix(query): keep remaining_predicates when filtering grouping sets by @sundy-li in #16971
• fix(query): fix bug while mysql external dictionary table contains null values by @Dragonliu2018 in #16978
• fix(sqlparser): fix order by followed by scalar by @chagelo in #16967
• fix(query): group by item allow set returning functions by @b41sh in #16986
• fix: http handler set cookie only when cookie_enabled=true. by @youngsofun in #16992
• fix: need_sticky field was incorrectly set to true. by @youngsofun in #16997
• fix(ci): fix benchmark ci failure by @zhang2014 in #17005
• fix: insert and mutation progress by @zhyass in #17014
• fix(query): reuse connection to fix dictionary mysql flaky test by @b41sh in #17016
• fix(ci): flaky test by @zhyass in #17030
• fix: filter push down union in r cte by @xudong963 in #17031
• fix: drop channel if partition receiver finished by @dqhl76 in #17037
• fix: handle trailing zeros in Raft log WAL with EXT4 writeback mode by @drmingdrmer in #17042
• fix(query): fix read small parquet files in cluster mode by @sundy-li in #17063
• fix(query): fix thread leak by @zhang2014 in #17077
• fix(query): fix thread leak and remove all usage by @dqhl76 in #17081
• fix: change tracking stream failed by @zhyass in #17072
• fix(query): fix join on not null column cause panic by @b41sh in #17069
• fix: create as query respect ddl_column_type_nullable by @TCeason in #17071
• fix: check if the table option is valid according to the engine by @SkyFan2002 in #17076
• fix(ci): flaky test by @zhyass in #17090
• fix(query): double panic if broken state in block builder by @zhang2014 in #17091
• fix(query): Correct sign extension handling in months_days_micros struct by @TCeason in #17086
• fix(query): union all panic in mysql client by @zhyass in #17095
• fix: name resolution bug in case-sensitive mode by @zhyass in #17097
• fix(query): fix a number of bugs in the spill config. by @forsaken628 in #17103
• fix(query): uri path should not be percent-encoded by @youngsofun in #17109
• fix(query): fix default values of SpillConfig may not take effect by @forsaken628 in #17113
• fix(query): table field data type in show create table incorrect by @forsaken628 in #17112
• fix(query): fix udf script return type is arrow DataType::LargeList caused failed by @b41sh in #17127
• fix: ignore case when matching function name by @notauserx in #16912
• fix: analyze table contain keyword bug by @zhyass in #17131
• fix(query): Disabled the shrink_scalar optimization for cast expressions during the resolve procedure args by @TCeason in #17119
• fix(log): prefix_filter not working. by @youngsofun in #17136

Code Refactor 🎉

• refactor: replace absolute expiration time with relative TTL in schema API by @drmingdrmer in #16268
• refactor(query): remove lru cache meter by @zhang2014 in #16260
• refactor: Use global http client to share the connection pool by @Xuanwo in #16276
• refactor(query): refactor hash join spill by @Dousir9 in #15746
• refactor: change DatabaseInfo.meta to SeqV<DatabaseMeta> by @drmingdrmer in #16295
• refactor: change value of meta-service key DatabaseNameIdent from DatabaseId to Id<DatabaseId> by @drmingdrmer in #16299
• refactor: adopt get_pb() for get_background_task by @drmingdrmer in #16308
• refactor: Reduce parallelism when vacuuming dropped tables by @dantengsky in #16312
• refactor(query): add setting external_server_request_retry_times by @sundy-li in #16307
• refactor(query): refactor bind set returning functions by @b41sh in #16316
• refactor(query): add config enable_meta_data_upgrade_json_to_pb_from_v307 by @TCeason in #16306
• refactor: add associated type kvapi::Value::KeyType by @drmingdrmer in #16320
• refactor: kv-app-errors does not need serde by @drmingdrmer in #16324
• refactor(query): refactor bind async functions by @b41sh in #16331
• refactor: Catalog Manager API improvements for create and drop operations by @drmingdrmer in #16338
• refactor: Add BaseApi to abstract usually used meta-service access pattern by @drmingdrmer in #16351
• refactor: simplify SchemaApi::list_indexes(), get_index() list_background_jobs() by @drmingdrmer in #16357
• chore(query): optimize window sort by @sundy-li in #16355
• chore(query): reduce hash join memory usage by @Dousir9 in #16359
• refactor: simplify drop_index() with NameIdValueApi by @drmingdrmer in #16366
• refactor: simplify dictionary API by @drmingdrmer in #16371
• refactor: Add item count check to KVPbApi::get_pb_stream() by @drmingdrmer in #16377
• refactor(query): optimize show drop tables by @TCeason in #16370
• chore(query): Simplified sort simple rows implementation by @forsaken628 in #16382
• refactor(query): refactor some codes of parquet2 by @sundy-li in #16386
• refactor: remove all share related functions by @drmingdrmer in #16383
• refactor: replace get_pb_value() with get_pb() by @drmingdrmer in #16395
• refactor: Add NameValueApi, simplify virtual column APIs by @drmingdrmer in #16398
• refactor: simplify catalog API by @drmingdrmer in #16406
• refactor: Simplify table lock API. by @drmingdrmer in #16413
• chore(query): introduce EmptyResultScan plan by @sundy-li in #16411
• refactor: remove unused TableInfo.tenant by @drmingdrmer in #16416
• refactor: simplify least-visible-time schema API, add LeastVisibleTimeIdent by @drmingdrmer in #16418
• refactor: add mget_id_value_compat() by @drmingdrmer in #16425
• refactor(query): first check privilege in SystemEngine get_full_data by @TCeason in #16421
• chore(query): limit max concurrency of spill io requests by @zhang2014 in #16442
• chore(query): make string to int respect behavior like PG by @sundy-li in #16428
• refactor: simplify vacuum drop table handling by @drmingdrmer in #16446
• chore: use logforth to replace fern by @andylokandy in #16249
• refactor(executor): use event cause to refactor shuffle processor by @zhang2014 in #16445
• refactor: cleanup logic to get tables to vacuum by @drmingdrmer in #16450
• refactor: enhancements of HTTP client session. by @youngsofun in #16452
• refactor(query): sort kernel optimization by @sundy-li in #16458
• refactor(storage): improve inverted index read fst file first to reduce load index by @b41sh in #16385
• refactor: refactor vacuum by @drmingdrmer in #16454
• chore: simplify get_sequence() and get_sequence_next_value() by @drmingdrmer in #16463
• refactor: add DatabaseMeta.gc_in_progress by @drmingdrmer in #16464
• refactor(query): use pipeline exchange to refactor window scatter by @zhang2014 in #16471
• refactor(query): simplify expression kernel by @sundy-li in #16482
• refactor: add KVPbCrudApi as abstraction layer for commonly used CRUD operations by @drmingdrmer in #16479
• refactor: add KVPbCrudApi::crud_try_insert() and crud_try_upsert() by @drmingdrmer in #16484
• refactor: SchemaApi::list_databases() by @drmingdrmer in #16485
• refactor: SchemaApi::get_database_history() by @drmingdrmer in #16487
• refactor: test SchemaApi::get_tenant_history_database() by @drmingdrmer in #16488
• refactor(query): refactor license manager by @zhang2014 in #16489
• refactor(query): refactor license manager by @zhang2014 in #16492
• refactor: simplify DroppedId. Replace manual comparison with string comparison by @drmingdrmer in #16495
• refactor: update dependency datafusion-orc. by @youngsofun in #16491
• refactor: get_tableinfos_by_ids() by @drmingdrmer in #16503
• chore(query): refactor like function by @sundy-li in #16493
• chore: simplify SchemaApi::truncate_table() by @drmingdrmer in #16506
• refactor: remove get_pb_value() by @drmingdrmer in #16516
• refactor: Output whole error message with source while retry by @Xuanwo in #16519
• refactor: unify path of write segment by @SkyFan2002 in #16517
• refactor: Improve gc_dropped_table_by_id() method by @drmingdrmer in #16528
• refactor: dropped table listing and GC by @drmingdrmer in #16531
• refactor: gc_dropped_db_by_id() by @drmingdrmer in #16538
• refactor: SchemaAPI::do_get_table_history by @drmingdrmer in #16540
• refactor: DroppedId for listing db/tables for gc by @drmingdrmer in #16542
• refactor: get_history_tables_for_gc() should not return TableInfo, but just table name, id and values by @drmingdrmer in #16545
• refactor(storage): improve inverted index match phrase query by @b41sh in #16547
• refactor: Automatically chunk large key sets in KVPbApi methods by @drmingdrmer in #16561
• refactor: simplify get_tables_history() by @drmingdrmer in #16572
• refactor(iceberg): Use iceberg file IO instead of our own operator by @Xuanwo in #16577
• refactor: add SchemaApi::get_table_in_db by @drmingdrmer in #16582
• refactor: SchemaApi::get_table_meta_history() by @drmingdrmer in #16586
• refactor: fix error message. by @youngsofun in #16578
• refactor(storage): inverted index use merge io to read data by @b41sh in #16589
• refactor: isolate session_id and session states of diff users. by @youngsofun in #16592
• refactor: Unify the set and unset structures by @TCeason in #16622
• refactor(query): make delta table support partition prunning by @sundy-li in #16621
• refactor(storage): inverted index support calculate the score by @b41sh in #16609
• refactor: some http APIs do not need sticky. by @youngsofun in #16644
• refactor: delete user_procedure and refactor procedure interpreter by @TCeason in #16532
• refactor(query): use arrow-flight to exchange datas by @sundy-li in #16657
• refactor: refine cast variant to map by @andylokandy in #16691
• refactor: streamline system.temp_files table scan by @dantengsky in #16659
• refactor: Align versions for arrow, opendal, hyper, tonic, prost by @Xuanwo in #16704
• refactor: refine column statistic map serialization by @dantengsky in #16728
• refactor: MetaGrpcClient needs to specify timeout to auto-reconnect to restarted server by @drmingdrmer in #16733
• refactor: replace direct GitHub dependencies with published crates.io versions by @drmingdrmer in #16700
• refactor: Enforce all deps must be declared at root by @Xuanwo in #16741
• refactor: Remove not used async-std by @Xuanwo in #16745
• refactor: change blocking pruning process to non-blocking channel by @dqhl76 in #16718
• refactor: ensure last-purged log-id cursor in exported from databend-meta service by @drmingdrmer in #16759
• refactor: Consolidate MetaStorageError variants by @drmingdrmer in #16760
• refactor(storage): refactor read virtual columns by @b41sh in #16758
• refactor: remove location_prefix, which is no longer supported by @dantengsky in #16757
• refactor(base): add stacktrace to replace backtrace by @zhang2014 in #16643
• refactor: when upgrading, do not import already purged garbage log entries by @drmingdrmer in #16809
• refactor: align metrics field names with current behavior by @drmingdrmer in #16826
• refactor(base): support stacktrace in cluster mode by @zhang2014 in #16821
• refactor: refactor cte binder and fix materialized cte used in subquery by @xudong963 in #16785
• refactor: remove EmbeddedMeta by @drmingdrmer in #16847
• refactor: remove unused sled related codes by @drmingdrmer in #16857
• refactor: make ListKV non-blocking by returning stream directly by @drmingdrmer in #16868
• refactor: optimize query system.tables when query single table by @TCeason in #16869
• refactor(query): refactor geometry functions by @b41sh in #16870
• refactor: use jiff crate replace chrono in date/timestamp func by @TCeason in #16787
• refactor: add new raft-log metrics to "metactl status" response by @drmingdrmer in #16899
• refactor: remove obsolete gRPC read API by @drmingdrmer in #16909
• refactor: deprecate: Operation::AsIs and will be removed by @drmingdrmer in #16913
• refactor(sqlsmith): refactor sqlsmith using http client by @b41sh in #16890
• refactor: remove ValueRef and use Value instead by @andylokandy in #16891
• refactor: make test_watch_expired_events() less sensitive to inaccurate time by @drmingdrmer in #16966
• refactor: use temp table to refactor materialized cte by @xudong963 in #16900
• refactor: set path of cookie. by @youngsofun in #17008
• refactor: replace RangeMap with span-map for better range lookup by @drmingdrmer in #17006
• refactor: revert pr #16869 by @TCeason in #17012
• refactor: auto split large mget_database_names_by_ids into chunks with KVPbApi::get_pb_values_vec() by @drmingdrmer in #17011
• refactor: mget db names and table names remove duplicate ids to avoid too many parameters by @b41sh in #17013
• refactor: integrate fuse table block pruning into pipeline by @dqhl76 in #16841
• refactor: optimize query system.tables when query single table (#16869) by @TCeason in #17021
• refactor(cluster): refactor cluster for dyn cluster [part 1] by @zhang2014 in #17023
• refactor(query): refactor udf server batch rows by @b41sh in #17092
• refactor: set name for all runtime worker threads. by @youngsofun in #17084
• refactor(query): refactor udf server connection reuse endpoint by @b41sh in #17106
• refactor: simplify alter and drop cluster key logic by @zhyass in #17128
• refactor: integrate segment pruning into pipeline by @dqhl76 in #17126

Build/Testing/CI Infra Changes 🔌

• ci: Add iceberg tpch sqllogictests by @Xuanwo in #16598
• ci: Enable incremental back to see how it works by @Xuanwo in #16817
• ci: fix flaky test by @youngsofun in #16945
• ci: ignore RUSTSEC-2024-0402. by @youngsofun in #16998

Documentation 📔

• docs: add stream to readme by @BohuTANG in #16810

Others 📒

• chore(query): add more snapshot/segment serde tests by @sundy-li in #16273
• chore: Use tagged iceberg crates by @Xuanwo in #16278
• chore: add a setting to decide if table sample is deterministic by @xudong963 in #16275
• chore(ci): tmp disable sharing test by @everpcpc in #16285
• chore(ci): flaky test by @zhyass in #16207
• chore: replace condition and if_then with txn in txn methods by @drmingdrmer in #16289
• chore(query): bump ast 0.0.3 by @sundy-li in #16283
• chore(query): avoid create query context each auth by @zhang2014 in #16279
• chore(query): remove unused parquet2 codes by @sundy-li in #16282
• chore: add setting auto_compaction_segments_limit by @zhyass in #16298
• chore: remove the log about if histogram exists by @xudong963 in #16303
• chore: simplify create_data_mask by @drmingdrmer in #16309
• chore(ci): adjust release time by @everpcpc in #16334
• chore(query): explain ast work like explain syntax by @sundy-li in #16335
• chore: support temporary table with memory engine by @SkyFan2002 in #16332
• chore(query): improve project set by @Dousir9 in #16326
• chore: remove unused serde impl by @drmingdrmer in #16347
• chore: remove empty reply struct by @drmingdrmer in #16350
• chore(query): fix window index out of range by @Dousir9 in #16352
• chore: separate test run for temp_table suite by @SkyFan2002 in #16358
• chore(code): update the version of gix-path in Cargo.lock by @Dousir9 in #16365
• chore(ci): add internal test by @TCeason in #16369
• chore: improve Debug and Display for DataId and TIdent by @drmingdrmer in #16379
• chore: add logging for embedded meta test by @drmingdrmer in #16384
• chore(ci): fix cargo clippy by @TCeason in #16397
• chore: add error stack to cmd entrence by @andylokandy in #16391
• chore(query): add external server request profile by @zhang2014 in #16400
• chore(query): refactor udf retry by @sundy-li in #16405
• chore(query): add connect duration metric for udf by @zhang2014 in #16407
• chore: add join type for explain analyze partial by @xudong963 in #16408
• chore: bump quinn-proto to 0.11.8 for cargo audit pass by @dqhl76 in #16419
• chore(planner): improve infer filter by @Dousir9 in #16361
• chore: log remote addr when meta-service raft API recevies a message by @drmingdrmer in #16438
• chore(query): add column: system.tables.total_columns by @TCeason in #16447
• chore(ci): fix dataset domain by @everpcpc in #16455
• chore(ci): release with musl by @everpcpc in #16460
• chore(executor): support for data exchange between different processors within the pipeline by @zhang2014 in #16467
• chore(executor): more edge detail for executing graph by @zhang2014 in #16468
• chore(ci): remove hdfs release by @everpcpc in #16477
• chore(query): null first behavior by @sundy-li in #16475
• chore: update nom-rule by @andylokandy in #16483
• chore: bump proj4rs 0.1.4 by @b41sh in #16474
• chore: remove logs from block writer by @dantengsky in #16497
• chore: simplify get_gc_table_info() by @drmingdrmer in #16499
• chore(query): replace to datafuse-extras/dtparse by @TCeason in #16509
• chore(storage): hook recluster avoid scan all segments by @zhyass in #16498
• chore: update source repo for async-compression by @everpcpc in #16513
• chore(query): move recursion & backtrace crate to datafuse-extras by @zhang2014 in #16520
• chore(query): add more asserts by @sundy-li in #16536
• chore: update repo for workflows by @everpcpc in #16546
• chore(storage): refactor compact source by @zhyass in #16527
• chore(query): SHOW/DESC is not DDL by @TCeason in #16583
• chore: check lock revision before watch delete by @zhyass in #16590
• chore(ci): flaky test by @zhyass in #16596
• chore(query): improve variable parser by @sundy-li in #16599
• chore: refine table sample tests by @xudong963 in #16600
• chore(query): add more logs on aggregation by @sundy-li in #16552
• chore: reset recluster block size by @zhyass in #16623
• chore(query): add http address info to cluster node info by @zhang2014 in #16620
• chore: drop guard when acquire lock failed by @zhyass in #16616
• chore: revert “lock mgr may failed to release lock in time (#16611)" by @dantengsky in #16630
• chore(query): reduce unsafe codes in kernels by @sundy-li in #16633
• chore(query): add warn log if vacuum temporary file is error by @zhang2014 in #16646
• chore: resolve todo in dynamic sample by @xudong963 in #16629
• chore(query): enable loser tree merge sort default by @forsaken628 in #16649
• chore(query): replace arrow2 with arrow-rs in serialize_column and deserialize_column by @forsaken628 in #16655
• chore(query): remove hash join build detail status by @zhang2014 in #16674
• chore: upon startup, meta-service output human-readable error if trying to open future version on-disk data by @drmingdrmer in #16679
• chore(query): modify rbac test result display by @TCeason in #16686
• chore: adjust default value of setting max_storage_io_requests by @SkyFan2002 in #16680
• chore(query): returns nan for aggregate_kurtosis and aggregate_skewness by @sundy-li in #16688
• chore: display port number on error from bind to port failure by @lyang24 in #16695
• chore: remove SortColumnDescription.is_nullable by @forsaken628 in #16699
• chore: remove transaction keyword by @SkyFan2002 in #16707
• chore: add timeout for detect_region operation to prevent hanging request by @dantengsky in #16709
• chore(query): add current_query_id and database id for admin api by @zhang2014 in #16703
• chore(query): index scalar support variant by @sundy-li in #16719
• chore: display log.file.limit and prefix_filter by @TCeason in #16713
• chore: avoid returning error during computing binary comparison selectivity by @xudong963 in #16725
• chore: Simplifying BlockMetaInfo Definitions by @forsaken628 in #16716
• chore: tweak cargo feature "jemalloc" by @dantengsky in #16693
• chore: remove flaky test by @TCeason in #16739
• chore: refine vacuum by @dantengsky in #16743
• chore: Remove not used dep config by @Xuanwo in #16746
• chore(query): Introduce VecExt and SliceExt to check unsafe bounds in debug mode by @sundy-li in #16747
• chore: make VACUUM TEMPORARY FILES killable by @dantengsky in #16751
• chore: tweak batch deletion concurrency by @dantengsky in #16770
• chore: refine raft-types import path by @drmingdrmer in #16764
• chore: remove obsolete version by @everpcpc in #16789
• chore: add snapshot location cache for attached table by @dantengsky in #16795
• chore: update meta-service version info and compatibility doc by @drmingdrmer in #16800
• chore(query): embedded public key of license by @zhang2014 in #16813
• chore(query): refactor common/arrow removes arrow/io crates by @sundy-li in #16808
• chore: purge dropped view metadata during vacuum by @dantengsky in #16819
• chore(query): keep queries detail and profile log format by @zhang2014 in #16822
• chore(ci): test with different license by @everpcpc in #16815
• chore(base): rollback to kernel signal handler after dump backtrace by @zhang2014 in #16828
• chore: remove unused codes by @zhyass in #16834
• chore: fix typo in ColumnCacheKeyBuilder by @dantengsky in #16838
• chore(cluster): disable parallel commit of cluster tasks by @zhang2014 in #16851
• chore(query): remove SliceExt by @sundy-li in #16853
• chore(ci): separate build profile by @everpcpc in #16855
• chore(query): remove useless new line for query log by @zhang2014 in #16860
• chore: add some log by @SkyFan2002 in #16863
• chore(query): improve distributed runtime filter by @Dousir9 in #16862
• chore: enable stderr logging for table_meta_inspector by @dantengsky in #16865
• chore(query): remove common/arrow crate by @sundy-li in #16846
• chore(ci): adjust runner size by @everpcpc in #16879
• chore(settings): enable spill by @Dousir9 in #16888
• chore: remove unused config_id from RaftConfig by @drmingdrmer in #16902
• chore(ci): flaky test by @zhyass in #16917
• chore(ci): enable overflow-checks in ci profile by @sundy-li in #16895
• chore(ci): flaky test by @zhyass in #16919
• chore(query): add snapshot logs in read partitions by @sundy-li in #16918
• chore: bump rustls version 0.23.18 by @TCeason in #16936
• chore(query): spilt binary symbol by @zhang2014 in #16861
• chore: k way merge sort add boundary check to prevent process crash by @forsaken628 in #16934
• chore(ci): remove aarch64 musl in release by @zhang2014 in #16946
• chore(planner): improve cardinality estimation by @Dousir9 in #16938
• chore(planner): refine project set cardinality by @Dousir9 in #16944
• chore: support recursive cte with normal cte by @xudong963 in #16932
• chore(binder): add enable_materialized_cte settings by @Dousir9 in #16950
• chore(storage): hook compact avoid scan all segments by @zhyass in #16954
• chore(storage): do compact before recluster during compact hook by @zhyass in #16949
• chore(ci): upgrade bendsql by @everpcpc in #16965
• chore(storage): refine error message for stream read offset snapshot by @zhyass in #16964
• chore: bump ruzstd from 0.7.2 to 0.7.3 by @dqhl76 in #16969
• chore(ci): install bendsql from packages by @everpcpc in #16970
• chore(planner): improve physical join by @Dousir9 in #16962
• chore(ci): fix some flaky test by @TCeason in #16980
• chore(query): improve decimal aggregate state by @sundy-li in #16983
• chore(planner): refine physical join by @Dousir9 in #16988
• chore: adjust fuse_time_travel_size() by @SkyFan2002 in #16995
• chore(ci): use new dsn for create database by @zhang2014 in #16968
• chore(ci): rust toolchain 2024-11-28 build image by @sundy-li in #17001
• chore(query): fix topk in native by @sundy-li in #17003
• chore(query): bump up rust toolchain to 2024-11-28 by @sundy-li in #16981
• chore: tweak attach table refreshing by @dantengsky in #17019
• chore(query): remove old aggregate codes by @sundy-li in #17025
• chore: add more test to meta-service txn value-matching by @drmingdrmer in #17033
• chore(cluster): disable parallel commit for cluster task by @zhang2014 in #17034
• chore: improve error handling in partition sending loop by @dqhl76 in #17035
• chore(ci): remove musl release by @everpcpc in #17041
• chore(query): improve comparison functions by @sundy-li in #17032
• chore(binder): fix bind materialized cte by @Dousir9 in #17046
• chore: refmt crash log message by @TCeason in #17059
• chore: bump dtparse by @TCeason in #17073
• chore: remove unused dev by @zhyass in #17088
• chore(query): Revert "feat(query): support javascript/python script User Defined Aggregate Function" by @sundy-li in #17107
• chore: add more log by @SkyFan2002 in #17110
• chore(planner): fix except and intersect operation by @Dousir9 in #17111
• chore(query): add fuzz tests by @sundy-li in #17100
• chore(query): bump federated mysql version by @TCeason in #17118
• chore(query): reduce-functions-code-expand by @sundy-li in #17114
• chore(query): improve hook vacuum temp after query by @sundy-li in #17125
• chore(query): bump enum-as-inner 0.6 by @forsaken628 in #17120
• chore: rename default_cluster_key_id to cluster_key_seq by @zhyass in #17135
• chore(query): fix udf metrics by @sundy-li in #17134

New Contributors

• @arkzuse made their first contribution in #16502
• @chagelo made their first contribution in #16560
• @Dragonliu2018 made their first contribution in #16631
• @Maricaya made their first contribution in #16543
• @ding-young made their first contribution in #16685
• @lyang24 made their first contribution in #16695
• @florann made their first contribution in #16181
• @Rowlandev made their first contribution in #16824
• @kursataktas made their first contribution in #16773
• @flashmouse made their first contribution in #16926
• @notauserx made their first contribution in #16912

Full Changelog: v1.2.615...v1.2.680