Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

GoogleCloud SCIR connector #1471

Merged
merged 17 commits into from
Feb 14, 2025
Merged

GoogleCloud SCIR connector #1471

merged 17 commits into from
Feb 14, 2025

Conversation

prioux
Copy link
Member

@prioux prioux commented Feb 14, 2025

After many months, this PR includes the class that implements the GoogleCloud submission system.

It assumes a bunch of stuff has already been prepared in advance on the google cloud side (a Bourreau is configured, a compute image was prepared, storage is mounted/configured etc etc).

On the portal side, the configuration in the interface requires the administrator to enter three values as NAME=value ; they are

GCLOUD_PROJECT=projectname          # e.g. cbrain-449118
GCLOUD_LOCATION=locationname        # e.g. northamerica-northeast1
GCLOUD_IMAGE_BASENAME=name-of-image # e.g. cbrain-compute-4cores-20G or whatever name you preapre on the gcloud side

These values are entered space-separated in the fields called "Extra QSUB Args"; there are two places for these:

  1. at the Bourreau level, the attribute is called cms_extra_qsub_args
  2. at the ToolConfig level, the attribute is called extra_qsub_args

In the interface, for both models, the forms show them as Extra 'qsub' options

Entering a configuration at the ToolConfig level has higher priority than at the Bourreau level. So the 'default' parameters for the Gcloud configuration can be made on the Bourreau, and a ToolConfig can override them. In general though, only the GCLOUD_IMAGE_BASENAME is likely to change for a ToolConfig.

My recommendation is:

For the Bourreau, provide all three:

  GCLOUD_PROJECT=a GCLOUD_LOCATION=b GCLOUD_IMAGE_BASENAME=c

and these will be the default for all tools on the bourreau; then if needed, for a specific ToolConfig, change the image name only:

  GCLOUD_IMAGE_BASENAME=d

It is acknowledged that the field names 'exta qsub args', initially created to hold command-line options for the "sbatch" or "qsub" commands, is no longer really appropriate for GCLOUD, but I plan to add notes in the interface to that effect.

Special review note: commits made on or after Feb 13th have not been tested yet, I'll test them in production. They're just esthetic changes. Famous last words.

prioux added 17 commits December 8, 2024 16:50
Adjusted hardcoded image name; TODO make it configurable.
Three variables need to be configured by the CBRAIN admin:

GCLOUD_PROJECT=project_id
GCLOUD_LOCATION=location
GCLOUD_IMAGE_BASENAME=computeimagename

These are normally set either at the bourreau level in
the 'extra qsub arguments' field, or at the tool config level
(also in the extra qesub arguments). They should be
put together separated by spaces, e.g.

   GCLOUD_PROJECT=abcd GCLOUD_LOCATION=northamerica-northeast1-b
@prioux prioux added Enhancement User Interface Priority: Normal Admin Features or bugs related to administrative features labels Feb 14, 2025
@prioux
Copy link
Member Author

prioux commented Feb 14, 2025

@MontrealSergiy @natacha-beck I am going to merge this and deploy it, but feel free to have a look at the code and make comments anyway. I'm not expecting it to deploy it yourself, it's kind of impossible without a Gcloud org to connect to.

@prioux prioux merged commit 1b87603 into aces:master Feb 14, 2025
1 check passed
@MontrealSergiy
Copy link
Contributor

seems ok to me, though it is a bit off gcloud_location is both defined as constant and extracted from qsub_args, is not qsub enough?

@prioux
Copy link
Member Author

prioux commented Feb 17, 2025

@MontrealSergiy good catch but you are looking at old code already, I fixed that last week.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Admin Features or bugs related to administrative features Enhancement Priority: Normal User Interface
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants