Skip to content

does RocksDict compatible with RocksDB-Python? #61

Closed

Description

@zjuzhfbloodz

Here is my code, I am trying to save some sst files and read with rocksdb in python:

from rocksdict import Rdict, Options, SstFileWriter
import random
import rocksdb

# generate some rand bytes
rand_bytes1 = [random.randbytes(200) for _ in range(100000)]
rand_bytes1.sort()

# write to file1.sst
writer = SstFileWriter(options=Options(raw_mode=True))
writer.open("file1.sst")
for k, v in zip(rand_bytes1, rand_bytes1):
    writer[k] = v

writer.finish()

# bulid Rdict
d = Rdict("tmp", options=Options(raw_mode=True))
d.ingest_external_file(["file1.sst"])
d.close()

# rocksdb read
db = rocksdb.DB("tmp", read_only=True, opts=rocksdb.Options())

And I am getting the error as below:

---------------------------------------------------------------------------
Corruption                                Traceback (most recent call last)
remote%2B10.97.150.93/opt/huawei/data1/z00811450/code/test.ipynb#Y211sdnNjb2RlLXJlbW90ZQ%3D%3D?line=22) db = rocksdb.DB("tmp", read_only=True, opts=rocksdb.Options())

File /opt/huawei/wisebrain/anaconda3/lib/python3.9/site-packages/rocksdb/_rocksdb.pyx:1636, in rocksdb._rocksdb.DB.__cinit__()

File /opt/huawei/wisebrain/anaconda3/lib/python3.9/site-packages/rocksdb/_rocksdb.pyx:76, in rocksdb._rocksdb.check_status()

Corruption: b'Corruption: unknown checksum type 4 from footer of tmp/000019.sst, while checking block at offset 22672231 size 35'

Does this mean that rocksdict is not compatible with rocksdb in python? Looking forward to anyone's help!

Activity

  1. Congyuwang commented on Apr 18, 2023

    @Congyuwang
    Collaborator

    rocksdict uses rocksdb 8.0 for the moment. So it maybe incompatible to rocksdb 6 which is used by python-rocksdb

  2. Congyuwang commented on Apr 18, 2023

    @Congyuwang
    Collaborator

    But it should be compatible with other rocksdb clients, e.g., java, c++, rust

  3. locked and limited conversation to collaborators on Apr 18, 2023
  4. converted this issue into a discussion #62 on Apr 18, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions