how can i export data

[hxlsky]

i want to export milvusdb data,how can i do?
is there a tools such as "mysqldump" in mysql?

[xiaofan-luan]

you can export data with iterator. or backup the collection to another

[hxlsky]

i want to export the data with "CSV" format,how can i do?

[yhmo]

Use the query_iterator to export milvus data row by row: https://milvus.io/docs/with-iterators.md#Query-with-an-iterator

This is an example of exporting data of a 10000 rows collection to a CSV file.

import csv
import random

from pymilvus import (
     connections,
     utility,
     FieldSchema, CollectionSchema, DataType,
     Collection,
 )

dim = 128
name = "AAA"


if __name__ == "__main__":
    connections.connect()

    schema = CollectionSchema(fields=[
        FieldSchema(name="id", dtype=DataType.INT64, is_primary=True, auto_id=False),
        FieldSchema(name="vector", dtype=DataType.FLOAT_VECTOR, dim=dim),
        FieldSchema(name="title", dtype=DataType.VARCHAR, max_length=100),
    ])

    utility.drop_collection(name)
    collection = Collection(name, schema)

    batch = 1000
    for i in range(10):
        data = [
            [i*batch+k for k in range(batch)],
            [[random.random() for _ in range(dim)] for _ in range(batch)],
            [f"title_{k}" for k in range(batch)]
        ]
        collection.insert(data)

    index_params = {
        "index_type": "IVF_FLAT",
        "metric_type": "L2",
        "params": {"nlist": 64},
    }
    collection.create_index("vector", index_params=index_params)

    collection.load()
    print("loaded")

    # print row count
    # set consistency_level="Strong" to make all the inserted data is ready
    print(collection.query(expr="", output_fields=["count(*)"], consistency_level="Strong"))


    # iterate all data
    query_iterator = collection.query_iterator(expr="id>0",
                                               output_fields=["*"],
                                               batch_size=100)

    with open('/tmp/data.csv', 'w', newline='') as file:
        writer = csv.writer(file)
        writer.writerow(["id", "title", "vector"])
        while True:
            res = query_iterator.next()
            if len(res) == 0:
                print("query iteration finished, close")
                query_iterator.close()
                break
            for i in range(len(res)):
                row = res[i]
                print(row)
                writer.writerow([row["id"], row["title"], row["vector"]])

[hxlsky]

if there have a tool maybe good!

[yhmo]

It is easy to use a short script to achieve this, so we don't have a specific tool.

[xiaofan-luan]

If anyone want to add a functionality to functionality to backup data to Json,parquet or csv format it is warmly welcomed

[nianliuu]

@hxlsky VTS: https://github.com/zilliztech/vts will add a new sink connector and support export data into Json, parquet, csv soon. Please keep an eye on this, I will update the doc in a few weeks.