Skip to content

Conversation

@ehigham
Copy link
Member

@ehigham ehigham commented Nov 5, 2025

Change Description

This PR refactors the filesystem abstraction layer to improve the URL handling interface. The main changes include:

  • Convert FSURL from a trait to an abstract class with a generic type parameter
  • Add a standard path component addition operator / to replace addPathComponent
  • Remove redundant methods like readNoCompression, deleteOnExit, stripCodecExtension, and caching-related methods
  • Rename getPath to simply path for cleaner access
  • Update implementations in AzureStorageFS, GoogleStorageFS, HadoopFS, and RouterFS to use the new interface

These changes make the filesystem API more consistent and easier to use while removing unnecessary functionality.

Security Assessment

This change cannot impact the Hail Batch instance as deployed by Broad Institute in GCP

Copy link
Member Author

ehigham commented Nov 5, 2025

@ehigham ehigham force-pushed the ehigham/scala-fs-cleanup branch from 38bd775 to 7ea3e1a Compare November 5, 2025 19:13
@ehigham ehigham marked this pull request as ready for review November 8, 2025 02:17
Copy link
Collaborator

@chrisvittal chrisvittal left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this is fantastic

val account: String,
val container: String,
val path: String,
override val path: String,
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this override for clarity? It's not necessary.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you explain why it's not necessary?
For clarity and better tool support. I've gotten into the habit of adding override to any member that overrides the superclass as intellij's tooling is much better at refactoring/renaming these. In the past when I've renamed/added or removed params/moved it didn't apply the changes to members not marked override which lead to complile errors and time wasting.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've long wanted a way to enforce using the override keyword, as I agree with Ed about the benefits, but it's very hard to remember to add it every time without tooling support. This comment made me do another look around, and I think I found a way to do that. I'll give it a try shortly.

@ehigham ehigham force-pushed the ehigham/scala-fs-cleanup branch from 7ea3e1a to 7bdccf4 Compare November 13, 2025 20:57
@hail-ci-robot hail-ci-robot merged commit cea5bde into main Nov 14, 2025
3 checks passed
@hail-ci-robot hail-ci-robot deleted the ehigham/scala-fs-cleanup branch November 14, 2025 03:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants