[Question]: 0.23.1 upgrade to 0.24.0 - infinity error occurred #13501
Description
Self Checks
- I have searched for existing issues search for existing issues, including closed ones.
- I confirm that I am using English to submit this report (Language Policy).
- Non-english title submitions will be closed directly ( 非英文标题的提交将会被直接关闭 ) (Language Policy).
- Please do not modify this template :) and fill in all the required fields.
Describe your problem
Precondition:
Helm deployment, version 0.23.1
Ragflow image: version 0.23.1
Infinity image: version 0.6.15
The operation to upgrade to version v0.24.0:
- Update the ragflow image to v0.24.0
- Update the infinity image to v0.7.0-dev2
- Open the ragflow web
- Infinity error occurred（When I set the ragflow replicas to 0 and updated the Infinity image, no error occurred. However, when I set ragflow replicas to 1, errors started to appear.）:
`
[06:49:16.344] [15744] [info] Trigger periodic checkpoint, after 30 seconds
[06:49:16.344] [290] [info] Optimize all indexes begin ts: 165149
[06:49:16.345] [288] [info] checkpoint ts: 165149, txn: 1503086
[06:49:16.396] [288] [info] CurrentObjFinalizeNoLock added empty object 019cd681-5448-7919-ab34-1a587df73bda
[06:49:16.425] [288] [info] PersistenceManager::CheckValid part 1: 29.14621 ms, part2: 0.33738 ms
[06:49:16.431] [238] [info] Checkpoint begin at 165149, cur txn commit_ts: 165150, txn_id: 1503086, swap to new wal file
[06:49:16.431] [238] [info] Wal /var/infinity/wal/wal.log swap to new path: /var/infinity/wal/wal.log.165148, error_if_duplicate: true
[06:49:16.432] [238] [info] Open new wal file /var/infinity/wal/wal.log
[06:49:26.345] [290] [info] Optimize all indexes begin ts: 165151
[06:49:36.346] [290] [info] Optimize all indexes begin ts: 165151
[06:49:46.347] [15744] [info] Trigger periodic checkpoint, after 30 seconds
[06:49:46.347] [288] [info] checkpoint ts: 165151, txn: 1503089
[06:49:46.347] [288] [info] Last checkpoint ts 165149, this checkpoint begin ts: 165151, SKIP CHECKPOINT
[06:49:46.347] [290] [info] Optimize all indexes begin ts: 165151
[06:49:46.347] [288] [info] Cleaning ts < 165149 dropped entities...
[06:49:56.348] [290] [info] Optimize all indexes begin ts: 165153
[06:50:06.349] [290] [info] Optimize all indexes begin ts: 165153
[06:50:16.349] [15744] [info] Trigger periodic checkpoint, after 30 seconds
[06:50:16.350] [290] [info] Optimize all indexes begin ts: 165153
[06:50:16.350] [288] [info] checkpoint ts: 165153, txn: 1503095
[06:50:16.352] [288] [info] CurrentObjFinalizeNoLock added empty object 019cd681-5448-7919-ab34-1a587df73bda
[06:50:16.379] [288] [info] PersistenceManager::CheckValid part 1: 27.25248 ms, part2: 0.129561 ms
[06:50:16.387] [238] [info] Checkpoint begin at 165153, cur txn commit_ts: 165154, txn_id: 1503095, swap to new wal file
[06:50:16.387] [238] [info] Wal /var/infinity/wal/wal.log swap to new path: /var/infinity/wal/wal.log.165152, error_if_duplicate: true
[06:50:16.388] [238] [info] Open new wal file /var/infinity/wal/wal.log
[06:50:26.351] [290] [info] Optimize all indexes begin ts: 165155
[06:50:36.351] [290] [info] Optimize all indexes begin ts: 165155
[06:50:43.103] [25] [warning] GetObjCache Failed to find object for local path db_1/tbl_240/idx_25/seg_0/chunk_0.idx: Key not found: pm|object|db_1/tbl_240/idx_25/seg_0/chunk_0.idx
[06:50:43.103] [25] [warning] GetObjCache Failed to find object for local path db_1/tbl_240/idx_25/seg_0/chunk_0.idx: Key not found: pm|object|db_1/tbl_240/idx_25/seg_0/chunk_0.idx
[06:50:43.111] [25] [warning] GetObjCache Failed to find object for local path db_1/tbl_240/seg_0/blk_0/38.col: Key not found: pm|object|db_1/tbl_240/seg_0/blk_0/38.col
[06:50:43.111] [25] [warning] GetObjCache Failed to find object for local path db_1/tbl_240/seg_0/blk_0/38.col: Key not found: pm|object|db_1/tbl_240/seg_0/blk_0/38.col
[06:50:43.127] [25] [warning] GetObjCache Failed to find object for local path db_1/tbl_240/idx_26/seg_0/chunk_0.idx: Key not found: pm|object|db_1/tbl_240/idx_26/seg_0/chunk_0.idx
[06:50:43.127] [25] [warning] GetObjCache Failed to find object for local path db_1/tbl_240/idx_26/seg_0/chunk_0.idx: Key not found: pm|object|db_1/tbl_240/idx_26/seg_0/chunk_0.idx
[06:50:43.127] [246] [critical] Txn ID: 1503082, Text: checkpoint, Begin TS: 165145, Commit TS: 165148, KV Commit TS: 165149, State: Committed
Txn ID: 1503086, Text: checkpoint, Begin TS: 165149, Commit TS: 165150, KV Commit TS: 165151, State: Committed
Txn ID: 1503091, Text: clean up, Begin TS: 165151, Commit TS: 165152, KV Commit TS: 165153, State: Committed
Txn ID: 1503095, Text: checkpoint, Begin TS: 165153, Commit TS: 165154, KV Commit TS: 165155, State: Committed
Txn ID: 1503355, Text: CREATE INDEX Ignore sec_kb_id ON ragflow_ab141fc1088711f195924b12f4642de6_00f800831b6311f1b6acf5082b37d648(kb_id USING SECONDARY WITH(cardinality low);, Begin TS: 165155, Commit TS: 165156, KV Commit TS: 165157, State: Committed
Create Index:
db name: default_db
db id: 1
table name: ragflow_ab141fc1088711f195924b12f4642de6_00f800831b6311f1b6acf5082b37d648
table id: 240
index def: IndexBase: SECONDARY, name: sec_kb_id, [kb_id]
table key: catalog|tbl|1|ragflow_ab141fc1088711f195924b12f4642de6_00f800831b6311f1b6acf5082b37d648|154482
Txn ID: 1503360, Text: AddColumnsStatement, Begin TS: 165157, Commit TS: 165158, KV Commit TS: 165159, State: Committed
Txn ID: 1503369, Text: CREATE INDEX Ignore sec_available_int ON ragflow_ab141fc1088711f195924b12f4642de6_00f800831b6311f1b6acf5082b37d648(available_int USING SECONDARY WITH(cardinality low);, Begin TS: 165159, Commit TS: 165160, KV Commit TS: 18446744073709551615, State: Committing
Create Index:
db name: default_db
db id: 1
table name: ragflow_ab141fc1088711f195924b12f4642de6_00f800831b6311f1b6acf5082b37d648
table id: 240
index def: IndexBase: SECONDARY, name: sec_available_int, [available_int]
table key: catalog|tbl|1|ragflow_ab141fc1088711f195924b12f4642de6_00f800831b6311f1b6acf5082b37d648|154482
[06:50:43.128] [246] [critical] Error: Table index cache with id: 1 already exists@src/storage/catalog/catalog_cache_impl.cpp:170
[06:50:43.481] [246] [critical] 0# infinity::PrintStacktrace@infinity_core(std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&) at /home/zhichyu/github.com/infiniflow/infinity/src/common/utility/exception_impl.cpp:47
1# infinity::UnrecoverableError@infinity_core(std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&, char const*, unsigned int) at /home/zhichyu/github.com/infiniflow/infinity/src/common/utility/exception_impl.cpp:81
2# infinity::TableCache@infinity_core::AddTableIndexCacheNolock(std::shared_ptrinfinity::TableIndexCache@infinity_core const&) at /home/zhichyu/github.com/infiniflow/infinity/src/storage/catalog/catalog_cache_impl.cpp:170
3# infinity::SystemCache@infinity_core::AddNewIndexCache(unsigned long, unsigned long, std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&) at /home/zhichyu/github.com/infiniflow/infinity/src/storage/catalog/catalog_cache_impl.cpp:252
4# infinity::NewTxnManager@infinity_core::CommitBottom(infinity::NewTxn@infinity_core*) at /home/zhichyu/github.com/infiniflow/infinity/src/storage/new_txn/new_txn_manager_impl.cpp:454
5# infinity::NewTxn@infinity_core::CommitBottom() at /home/zhichyu/github.com/infiniflow/infinity/src/storage/new_txn/new_txn_impl.cpp:4749
6# infinity::BottomExecutor@infinity_core::Start(unsigned long)::$_0::operator()() const at /home/zhichyu/github.com/infiniflow/infinity/src/storage/wal/bottom_executor_impl.cpp:51
7# execute_native_thread_routine at ../../../../../libstdc++-v3/src/c++11/thread.cc:104
8# at :0
9# __clone at :0
10#
[06:50:43.481] [246] [critical] Txn ID: 1503082, Text: checkpoint, Begin TS: 165145, Commit TS: 165148, KV Commit TS: 165149, State: Committed
Txn ID: 1503086, Text: checkpoint, Begin TS: 165149, Commit TS: 165150, KV Commit TS: 165151, State: Committed
Txn ID: 1503091, Text: clean up, Begin TS: 165151, Commit TS: 165152, KV Commit TS: 165153, State: Committed
Txn ID: 1503095, Text: checkpoint, Begin TS: 165153, Commit TS: 165154, KV Commit TS: 165155, State: Committed
Txn ID: 1503355, Text: CREATE INDEX Ignore sec_kb_id ON ragflow_ab141fc1088711f195924b12f4642de6_00f800831b6311f1b6acf5082b37d648(kb_id USING SECONDARY WITH(cardinality low);, Begin TS: 165155, Commit TS: 165156, KV Commit TS: 165157, State: Committed
Create Index:
db name: default_db
db id: 1
table name: ragflow_ab141fc1088711f195924b12f4642de6_00f800831b6311f1b6acf5082b37d648
table id: 240
index def: IndexBase: SECONDARY, name: sec_kb_id, [kb_id]
table key: catalog|tbl|1|ragflow_ab141fc1088711f195924b12f4642de6_00f800831b6311f1b6acf5082b37d648|154482
Txn ID: 1503360, Text: AddColumnsStatement, Begin TS: 165157, Commit TS: 165158, KV Commit TS: 165159, State: Committed
Txn ID: 1503369, Text: CREATE INDEX Ignore sec_available_int ON ragflow_ab141fc1088711f195924b12f4642de6_00f800831b6311f1b6acf5082b37d648(available_int USING SECONDARY WITH(cardinality low);, Begin TS: 165159, Commit TS: 165160, KV Commit TS: 18446744073709551615, State: Committing
Create Index:
db name: default_db
db id: 1
table name: ragflow_ab141fc1088711f195924b12f4642de6_00f800831b6311f1b6acf5082b37d648
table id: 240
index def: IndexBase: SECONDARY, name: sec_available_int, [available_int]
table key: catalog|tbl|1|ragflow_ab141fc1088711f195924b12f4642de6_00f800831b6311f1b6acf5082b37d648|154482
[06:50:43.481] [246] [critical] Error: TerminateHandler: Unhandled Exception: Table index cache with id: 1 already exists@src/storage/catalog/catalog_cache_impl.cpp:170
[06:50:43.489] [246] [critical] 0# infinity::PrintStacktrace@infinity_core(std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&) at /home/zhichyu/github.com/infiniflow/infinity/src/common/utility/exception_impl.cpp:47
1# (anonymous namespace)::TerminateHandler() at /home/zhichyu/github.com/infiniflow/infinity/src/bin/infinity_main.cpp:221
2# __cxxabiv1::__terminate(void ()()) at ../../../../libstdc++-v3/libsupc++/eh_terminate.cc:48
3# std::terminate() at ../../../../libstdc++-v3/libsupc++/eh_terminate.cc:58
4# __cxa_throw at ../../../../libstdc++-v3/libsupc++/eh_throw.cc:98
5# infinity::UnrecoverableError@infinity_core(std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&, char const, unsigned int) at /home/zhichyu/github.com/infiniflow/infinity/src/common/utility/exception_impl.cpp:84
6# infinity::TableCache@infinity_core::AddTableIndexCacheNolock(std::shared_ptrinfinity::TableIndexCache@infinity_core const&) at /home/zhichyu/github.com/infiniflow/infinity/src/storage/catalog/catalog_cache_impl.cpp:170
7# infinity::SystemCache@infinity_core::AddNewIndexCache(unsigned long, unsigned long, std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&) at /home/zhichyu/github.com/infiniflow/infinity/src/storage/catalog/catalog_cache_impl.cpp:252
8# infinity::NewTxnManager@infinity_core::CommitBottom(infinity::NewTxn@infinity_core*) at /home/zhichyu/github.com/infiniflow/infinity/src/storage/new_txn/new_txn_manager_impl.cpp:454
9# infinity::NewTxn@infinity_core::CommitBottom() at /home/zhichyu/github.com/infiniflow/infinity/src/storage/new_txn/new_txn_impl.cpp:4749
10# infinity::BottomExecutor@infinity_core::Start(unsigned long)::$_0::operator()() const at /home/zhichyu/github.com/infiniflow/infinity/src/storage/wal/bottom_executor_impl.cpp:51
11# execute_native_thread_routine at ../../../../../libstdc++-v3/src/c++11/thread.cc:104
12# at :0
13# __clone at :0
14#
[06:50:43.489] [246] [critical] Txn ID: 1503082, Text: checkpoint, Begin TS: 165145, Commit TS: 165148, KV Commit TS: 165149, State: Committed
Txn ID: 1503086, Text: checkpoint, Begin TS: 165149, Commit TS: 165150, KV Commit TS: 165151, State: Committed
Txn ID: 1503091, Text: clean up, Begin TS: 165151, Commit TS: 165152, KV Commit TS: 165153, State: Committed
Txn ID: 1503095, Text: checkpoint, Begin TS: 165153, Commit TS: 165154, KV Commit TS: 165155, State: Committed
Txn ID: 1503355, Text: CREATE INDEX Ignore sec_kb_id ON ragflow_ab141fc1088711f195924b12f4642de6_00f800831b6311f1b6acf5082b37d648(kb_id USING SECONDARY WITH(cardinality low);, Begin TS: 165155, Commit TS: 165156, KV Commit TS: 165157, State: Committed
Create Index:
db name: default_db
db id: 1
table name: ragflow_ab141fc1088711f195924b12f4642de6_00f800831b6311f1b6acf5082b37d648
table id: 240
index def: IndexBase: SECONDARY, name: sec_kb_id, [kb_id]
table key: catalog|tbl|1|ragflow_ab141fc1088711f195924b12f4642de6_00f800831b6311f1b6acf5082b37d648|154482
Txn ID: 1503360, Text: AddColumnsStatement, Begin TS: 165157, Commit TS: 165158, KV Commit TS: 165159, State: Committed
Txn ID: 1503369, Text: CREATE INDEX Ignore sec_available_int ON ragflow_ab141fc1088711f195924b12f4642de6_00f800831b6311f1b6acf5082b37d648(available_int USING SECONDARY WITH(cardinality low);, Begin TS: 165159, Commit TS: 165160, KV Commit TS: 18446744073709551615, State: Committing
Create Index:
db name: default_db
db id: 1
table name: ragflow_ab141fc1088711f195924b12f4642de6_00f800831b6311f1b6acf5082b37d648
table id: 240
index def: IndexBase: SECONDARY, name: sec_available_int, [available_int]
table key: catalog|tbl|1|ragflow_ab141fc1088711f195924b12f4642de6_00f800831b6311f1b6acf5082b37d648|154482
[06:50:43.489] [246] [critical] Error: Aborted
[06:50:43.489] [246] [critical] 0# infinity::PrintStacktrace@infinity_core(std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&) at /home/zhichyu/github.com/infiniflow/infinity/src/common/utility/exception_impl.cpp:47
1# (anonymous namespace)::SignalHandler(int, siginfo_t*, void*) at /home/zhichyu/github.com/infiniflow/infinity/src/bin/infinity_main.cpp:160
2# at :0
3# pthread_kill at :0
4# raise at :0
5# abort at :0
6# (anonymous namespace)::TerminateHandler() at /home/zhichyu/github.com/infiniflow/infinity/src/bin/infinity_main.cpp:222
7# __cxxabiv1::__terminate(void ()()) at ../../../../libstdc++-v3/libsupc++/eh_terminate.cc:48
8# std::terminate() at ../../../../libstdc++-v3/libsupc++/eh_terminate.cc:58
9# __cxa_throw at ../../../../libstdc++-v3/libsupc++/eh_throw.cc:98
10# infinity::UnrecoverableError@infinity_core(std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&, char const, unsigned int) at /home/zhichyu/github.com/infiniflow/infinity/src/common/utility/exception_impl.cpp:84
11# infinity::TableCache@infinity_core::AddTableIndexCacheNolock(std::shared_ptrinfinity::TableIndexCache@infinity_core const&) at /home/zhichyu/github.com/infiniflow/infinity/src/storage/catalog/catalog_cache_impl.cpp:170
12# infinity::SystemCache@infinity_core::AddNewIndexCache(unsigned long, unsigned long, std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&) at /home/zhichyu/github.com/infiniflow/infinity/src/storage/catalog/catalog_cache_impl.cpp:252
13# infinity::NewTxnManager@infinity_core::CommitBottom(infinity::NewTxn@infinity_core*) at /home/zhichyu/github.com/infiniflow/infinity/src/storage/new_txn/new_txn_manager_impl.cpp:454
14# infinity::NewTxn@infinity_core::CommitBottom() at /home/zhichyu/github.com/infiniflow/infinity/src/storage/new_txn/new_txn_impl.cpp:4749
15# infinity::BottomExecutor@infinity_core::Start(unsigned long)::$_0::operator()() const at /home/zhichyu/github.com/infiniflow/infinity/src/storage/wal/bottom_executor_impl.cpp:51
16# execute_native_thread_routine at ../../../../../libstdc++-v3/src/c++11/thread.cc:104
17# at :0
18# __clone at :0
19#
[06:50:43.489] [246] [critical] Txn ID: 1503082, Text: checkpoint, Begin TS: 165145, Commit TS: 165148, KV Commit TS: 165149, State: Committed
Txn ID: 1503086, Text: checkpoint, Begin TS: 165149, Commit TS: 165150, KV Commit TS: 165151, State: Committed
Txn ID: 1503091, Text: clean up, Begin TS: 165151, Commit TS: 165152, KV Commit TS: 165153, State: Committed
Txn ID: 1503095, Text: checkpoint, Begin TS: 165153, Commit TS: 165154, KV Commit TS: 165155, State: Committed
Txn ID: 1503355, Text: CREATE INDEX Ignore sec_kb_id ON ragflow_ab141fc1088711f195924b12f4642de6_00f800831b6311f1b6acf5082b37d648(kb_id USING SECONDARY WITH(cardinality low);, Begin TS: 165155, Commit TS: 165156, KV Commit TS: 165157, State: Committed
Create Index:
db name: default_db
db id: 1
table name: ragflow_ab141fc1088711f195924b12f4642de6_00f800831b6311f1b6acf5082b37d648
table id: 240
index def: IndexBase: SECONDARY, name: sec_kb_id, [kb_id]
table key: catalog|tbl|1|ragflow_ab141fc1088711f195924b12f4642de6_00f800831b6311f1b6acf5082b37d648|154482
Txn ID: 1503360, Text: AddColumnsStatement, Begin TS: 165157, Commit TS: 165158, KV Commit TS: 165159, State: Committed
Txn ID: 1503369, Text: CREATE INDEX Ignore sec_available_int ON ragflow_ab141fc1088711f195924b12f4642de6_00f800831b6311f1b6acf5082b37d648(available_int USING SECONDARY WITH(cardinality low);, Begin TS: 165159, Commit TS: 165160, KV Commit TS: 18446744073709551615, State: Committing
Create Index:
db name: default_db
db id: 1
table name: ragflow_ab141fc1088711f195924b12f4642de6_00f800831b6311f1b6acf5082b37d648
table id: 240
index def: IndexBase: SECONDARY, name: sec_available_int, [available_int]
table key: catalog|tbl|1|ragflow_ab141fc1088711f195924b12f4642de6_00f800831b6311f1b6acf5082b37d648|154482
[06:50:43.489] [246] [critical] Error: Segmentation fault
[06:50:43.489] [246] [critical] 0# infinity::PrintStacktrace@infinity_core(std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&) at /home/zhichyu/github.com/infiniflow/infinity/src/common/utility/exception_impl.cpp:47
1# (anonymous namespace)::SignalHandler(int, siginfo_t*, void*) at /home/zhichyu/github.com/infiniflow/infinity/src/bin/infinity_main.cpp:160
2# at :0
3# abort at :0
4# (anonymous namespace)::TerminateHandler() at /home/zhichyu/github.com/infiniflow/infinity/src/bin/infinity_main.cpp:222
5# __cxxabiv1::__terminate(void ()()) at ../../../../libstdc++-v3/libsupc++/eh_terminate.cc:48
6# std::terminate() at ../../../../libstdc++-v3/libsupc++/eh_terminate.cc:58
7# __cxa_throw at ../../../../libstdc++-v3/libsupc++/eh_throw.cc:98
8# infinity::UnrecoverableError@infinity_core(std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&, char const, unsigned int) at /home/zhichyu/github.com/infiniflow/infinity/src/common/utility/exception_impl.cpp:84
9# infinity::TableCache@infinity_core::AddTableIndexCacheNolock(std::shared_ptrinfinity::TableIndexCache@infinity_core const&) at /home/zhichyu/github.com/infiniflow/infinity/src/storage/catalog/catalog_cache_impl.cpp:170
10# infinity::SystemCache@infinity_core::AddNewIndexCache(unsigned long, unsigned long, std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&) at /home/zhichyu/github.com/infiniflow/infinity/src/storage/catalog/catalog_cache_impl.cpp:252
11# infinity::NewTxnManager@infinity_core::CommitBottom(infinity::NewTxn@infinity_core*) at /home/zhichyu/github.com/infiniflow/infinity/src/storage/new_txn/new_txn_manager_impl.cpp:454
12# infinity::NewTxn@infinity_core::CommitBottom() at /home/zhichyu/github.com/infiniflow/infinity/src/storage/new_txn/new_txn_impl.cpp:4749
13# infinity::BottomExecutor@infinity_core::Start(unsigned long)::$_0::operator()() const at /home/zhichyu/github.com/infiniflow/infinity/src/storage/wal/bottom_executor_impl.cpp:51
14# execute_native_thread_routine at ../../../../../libstdc++-v3/src/c++11/thread.cc:104
15# at :0
16# __clone at :0
17#
`