Skip to content

DVC can not push large file to OSS anymore once the uploading has been interrupted #10644

Open
@CNLHC

Description

@CNLHC

Bug Report

Description

Currently dvc can not upload large file to oss, after applying fix described in #10643, dvc still suffer from a subtle bug: if the uploading process is interrupted, then it can not been upload again.

Reproduce

dvc init
mkfile -n 200m test.blob
dvc add test.blob
dvc remote add foo oss://<oss_bucket>
dvc push -r foo #use ctrl+c to interrupt this function call
dvc push  -r foo 

the error message of last push will contain

ERROR: failed to transfer 'xxxx' - 'coroutine' object has no attribute 'parts'

Expected

dvc should push file to remote, either by recovering from the local state or pruning local state and then launch another new uploading session.

Environment information

same env to #10643

with patch applied.

Additional Information (if any):

A quick fix to this issue is deleting the uploading store

rm -rf ~/.py-oss-upload/

This directory is created by the oss2 .

The root cause is this line https://github.com/karajan1001/aiooss2/blob/875a06b99881df6fe900b1fed29e3a91dec12a7f/src/aiooss2/resumable.py#L339

aiooss2 pass an AioBucket to the oss2 and the later one does not invoke await properly.

flowchart LR
   dvc --> dvc-oss --> ossfs --> aiooss2 --> oss2
Loading

Metadata

Metadata

Assignees

No one assigned

    Labels

    A: data-syncRelated to dvc get/fetch/import/pull/pushfs: ossRelated to the Alibaba Cloud OSS filesystem

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions