[DocDB] Manual flush on the table fails due to Tablet not found

[shishir2001-yb]

Jira Link: DB-14705

Description

Version: 2024.1.4.0-b106

One of the test encountered the following failure while performing a manual flush on a table.

Error running flush_table: Internal error (yb/client/client-internal.cc:1233): Unable to flush table pc_db_001.pc_table_001: Flush operation for id: 0be246a140ab4d57ae247f99f798776d failed

Master logs

I1227 06:38:40.760607 85585 flush_manager.cc:172] Handling Flush Tablets Response from TS 31a9ae6bf1e44faa9fc2e4524f5772f0: flush request id: 0be246a140ab4d57ae247f99f798776d, status: Not found (yb/tserver/ts_tablet_manager.cc:2322): Tablet d15e10f36b31458185d1b45d0563264c not found (tablet server error 6)

Test Details:

""
    Test: Packed columns with Backup and restore, DDL, automatic tablet splitting,
          Materialized views, RPC Compression(1), Indexes, Collation, Flush and Compaction using
          tables containing columns of 26 different data types including domains, sequences,
          enums, user defined data types.
    Description:
        1. Creates universe with YSQL packed columns enabled, stream compression(1-GZIP),
           automatic tablet splitting enabled
        2. Create a db, table(26 data types with collated columns, sequences, enum, domains),
           a materialized view, regular, partial and unique indexes on the table and insert
           a workload into the table
        3. Refresh materialized view
        4. Create a backup- BACKUP_001
        5. Execute a manual flush and compaction
        6. Execute Alter table DDLs' to add columns and execute workloads based on DDL
        7. Create a backup- BACKUP_002
        8. Execute workload to insert data.
        9. Drop the database
        10. Restore to backup BACKUP_001
        11. Verify successful restore and execute select * from the table
        12. Execute DDLs' on the table to rename columns, set not null constraint,
            add more columns and execute a workload to insert data.
        13. Restore to backup BACKUP_002
        14. Verify successful restore and execute select *, insert
        15. Execute a manual flush and compaction
        16. Disable YSQL packed columns on the universe
        17. Insert more data
        18. Drop the database
        19. Restore to backup BACKUP_001 created with a different schema and
            when YSQL Packed columns was enabled to a point where it is disabled in the universe
        20. Create a backup - BACKUP_004
        21. Invalidate the INDEX
        22. Create a backup - BACKUP_005
        23. Insert more data
        24. Drop the database
        25. Restore to BACKUP_005
        26. Verify successful restore and execute select * from the table
        27. Drop the database
        28. Restore to BACKUP_004
        29. Verify successful restore and execute select * from the table
        30. Revalidate the INDEX
        31. Insert more data
        32. Execute manual flush and compaction ------------> Fails here
        33. Destroy Universe
    """

G-flags

master_flags = {
            "enable_automatic_tablet_splitting": "true",
            "tablet_split_low_phase_size_threshold_bytes": "13421",
            "tablet_split_low_phase_shard_count_per_node": "16",
            "ysql_enable_packed_row": "true",
            "enable_stream_compression": "true",
            "stream_compression_algo": "1",  # GZIP compression
            "ysql_enable_packed_row": "true",
            "tablet_replicas_per_gib_limit": "0", 
            "ysql_suppress_unsafe_alter_notice": "true"
        }
        tserver_flags = {
            "ysql_num_shards_per_tserver": "1",
            "ysql_enable_packed_row": "true",
            "enable_stream_compression": "true",
            "stream_compression_algo": "1",  # GZIP compression
            "ysql_enable_packed_row": "true",
           "tablet_replicas_per_gib_limit": "0",
        }

Issue Type

kind/bug

Warning: Please confirm that this issue does not contain any sensitive information

• I confirm this issue does not contain any sensitive information.

[rthallamko3]

@shishir2001-yb , Is it possible that the tablet got split and then the flush request comes in. cc @arybochkin to clarify if this can happen.

[shishir2001-yb]

Yes, @rthallamko3. Test uses aggressive tablet splitting threshold so it seems possible.