Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Backup fails for Cohesity S3 compatible storage #477

Closed
smatvienko-tb opened this issue May 17, 2022 · 3 comments · Fixed by #807
Closed

Backup fails for Cohesity S3 compatible storage #477

smatvienko-tb opened this issue May 17, 2022 · 3 comments · Fixed by #807
Labels
enhancement New feature or request help-wanted Issues in the state 'help-wanted' stale

Comments

@smatvienko-tb
Copy link

smatvienko-tb commented May 17, 2022 •
edited by sync-by-unito bot
Loading

Project board link

Hey!

I had an issue backing up the ThingsBoard's Cassandra data to the Cohesity S3 compatible storage.

After investigating the issue, I found that Cohesity S3 does not return the content-type header, confirmed by their specs later.

I made a patch to work without a content-type header, and Medusa for Apache Cassandra worked fine on production for about one week.

My PR on libcloud that fix the issue is here apache/libcloud#1695

Hopefully, this will help someone to deploy easier!

image

┆Issue is synchronized with this Jira Story by Unito
┆Reviewer: Radovan Zvoncek
┆Issue Number: MED-45
@adejanovski adejanovski added the new Issues requiring triage label May 17, 2022
@adejanovski adejanovski moved this to To Groom in K8ssandra Nov 8, 2022
@rzvoncek
Copy link
Contributor

rzvoncek commented Mar 5, 2024

Hi! We managed to drop libcloud usage meanwhile, so this probably became a non-issue.
However, we still need to check if medusa works with Cohesity.

@rzvoncek rzvoncek added assess Issues in the state 'assess' and removed new Issues requiring triage to-groom labels Mar 5, 2024
@adejanovski adejanovski moved this to Assess/Investigate in K8ssandra Mar 5, 2024
@rzvoncek
Copy link
Contributor

rzvoncek commented Apr 4, 2024

We can't easily get access to Cohesity. @smatvienko-tb , any chance you could give it a try with a recent medusa?

@rzvoncek rzvoncek added enhancement New feature or request help wanted Extra attention is needed stale and removed assess Issues in the state 'assess' labels Apr 4, 2024
@smatvienko-tb
Copy link
Author

We can't easily get access to Cohesity. @smatvienko-tb , any chance you could give it a try with a recent medusa?

OK. Here is some feedback. We have used 0.16.3 for a couple of months.
We did a libcloud patch as a legacy patch (has no effect)
But we faced another issue with upload to Cohesity S3 storage issue

"Invalid part number 8193 specified. Part number must be a number from 1 to 8192 inclusive"

The Cohesity S3 storage has a limited chunk count (others as well). So we were not able to send files of more than 64 GiB (8192*8MiB)

For a SizeTieredCompactionStrategy or after major compaction without splitting, that might be a problem.

FYI: AWS S3 limits to 10k chunks as well

The current workaround

We replaced the hardcoded value 8 * MB with 50 * MB as a workaround that suits our needs.

It would be nice to have a parameter to set the multipart_chunksize

BTW, 8M transfer time in 2024 is a fraction of a second, even at home. 😄

Unfortunately, we are still running with a patched library.

Here a @volodymyr-babak's comments on this

# Changes:
# transfer.py: multipart_chunksize=8 * MB -> multipart_chunksize=50 * MB,
# processpool.py: multipart_chunksize=8 * MB -> multipart_chunksize=50 * MB,
# manager.py: multipart_chunksize=8 * MB -> multipart_chunksize=50 * MB,
# __init__.py: multipart_chunksize=8 * MB -> multipart_chunksize=50 * MB,

Libraries and files patched :

COPY transfer.py /opt/bitnami/python/lib/python3.9/site-packages/boto3/s3/transfer.py
COPY processpool.py /opt/bitnami/python/lib/python3.9/site-packages/s3transfer/processpool.py
COPY manager.py /opt/bitnami/python/lib/python3.9/site-packages/s3transfer/manager.py
COPY __init__.py /opt/bitnami/python/lib/python3.9/site-packages/s3transfer/__init__.py

Hopefully, it will help somehow to improve the Medusa !

@rzvoncek rzvoncek added help-wanted Issues in the state 'help-wanted' and removed help wanted Extra attention is needed labels Apr 8, 2024
@adejanovski adejanovski moved this from Assess/Investigate to Help Wanted in K8ssandra Apr 8, 2024
@SoaigM SoaigM mentioned this issue Jul 25, 2024
Closed
@SoaigM SoaigM mentioned this issue Sep 27, 2024
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
enhancement New feature or request help-wanted Issues in the state 'help-wanted' stale
Projects
None yet
Milestone
No milestone
3 participants
@rzvoncek @adejanovski @smatvienko-tb