How to get WideColumns data in " Raw Mode" from rocksdb #122
Description
The db is created in C++ app and contains a lot of WideColumns data. Is it possible to access these data using RocksDict?
In the code example below value (v) always is 0 but keys (k) is shown as expected.
cf_lst=Rdict.list_cf(GLB_PATH)
opts=Options(raw_mode=True)
db = Rdict(path=GLB_PATH, options=opts, column_families={cf_lst[1]: opts})
db_cf1 = db.get_column_family(cf_lst[1])
#
for k, v in db_cf1.items():
print(f'k={k}, v size={len(v)}')
db.close()OS: Windows 2019 Server
compiler: msvc 19.39.33523
rocksdb: v9.0
RocksDict: v0.3.23
Activity
Congyuwang commentedon May 9, 2024
Looks like WideColumns is not yet supported here, and we will need to add APIs like
GetEntityandPutEntityto support it.Congyuwang commentedon May 9, 2024
Quoting rocksdb wide column doc:
The classic Get, MultiGet, GetMergeOperands, and Iterator::value APIs return the value of the default column when they encounter an entity, while the new APIs GetEntity, MultiGetEntity, and Iterator::columns return any plain key-value in the form of an entity with a single column, namely the anonymous default column.
Iterator returns the default value, which is empty. Needs
columns()api, which is not yet supported by rocksdict yet.Congyuwang commentedon May 10, 2024
Just curious, what do you use WideColumns for?
For the moment, if the object is not that large, I would suggest to use some custom deserialization for the entities. The APIs related to WideColumns have not yet been explosed to C interface yet by rocksdb. So, I would need some time and wait for rocksdb to design a proper C interface for WideColumns related APIs.
Congyuwang commentedon May 10, 2024
Related: meta/rocksdb#12635
fmvin commentedon May 10, 2024
Some kind of in-memory tables with random culumn's number in each row which are being frequently updated. I have found that using WideColumns fits well with my app architecture and allowed me to easily migrate from kx kdb.
For iterating with python I'm going to create a special copies of several tables using MessagePack serializer for the entities in the way you proposed. But it is some kind of overhead.
Congyuwang commentedon May 15, 2024
I've already drafted an up-stream PR: meta/rocksdb#12653
Congyuwang commentedon May 15, 2024
Check wide_columns_raw examples with
pip install rocksdict==0.3.24b1(pypi link).Tell me if it works 🙂.
fmvin commentedon May 16, 2024
No success. From real db I cannot access wide columns from column family (CF). Please provide a simple example how to use the
get_entitymethod with CF. The db itself works fine, checked it withldbtool.Congyuwang commentedon May 16, 2024
I'm about to release a beta.2, which will make opening DB created by other languages (c++, java, rust) much straightforward.
fmvin commentedon May 16, 2024
It seems I didn’t clearly explain the problem. In other words, I can’t figure out how to pass CF to the get_entity method.
Congyuwang commentedon May 16, 2024
Ok. Try
pip install rocksdict==0.3.24b2, andTell me if it works.
Congyuwang commentedon May 16, 2024
The logic of rocksdict is that, we do not pass cf argument to any of
get, put, iter, get_entity, and etc.. Instead, usesome_cf = db.get_column_family("some_cf_name")which returns an object with exact identical methods asRdictincludingget, put, delete, get_entity, and etc.All of these operations returns only data fromsome_cf