-
Notifications
You must be signed in to change notification settings - Fork 42
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
GoogleCloud SCIR connector #1471
Conversation
Adding gcloud support
Adjusted hardcoded image name; TODO make it configurable.
Three variables need to be configured by the CBRAIN admin: GCLOUD_PROJECT=project_id GCLOUD_LOCATION=location GCLOUD_IMAGE_BASENAME=computeimagename These are normally set either at the bourreau level in the 'extra qsub arguments' field, or at the tool config level (also in the extra qesub arguments). They should be put together separated by spaces, e.g. GCLOUD_PROJECT=abcd GCLOUD_LOCATION=northamerica-northeast1-b
@MontrealSergiy @natacha-beck I am going to merge this and deploy it, but feel free to have a look at the code and make comments anyway. I'm not expecting it to deploy it yourself, it's kind of impossible without a Gcloud org to connect to. |
seems ok to me, though it is a bit off |
@MontrealSergiy good catch but you are looking at old code already, I fixed that last week. |
After many months, this PR includes the class that implements the GoogleCloud submission system.
It assumes a bunch of stuff has already been prepared in advance on the google cloud side (a Bourreau is configured, a compute image was prepared, storage is mounted/configured etc etc).
On the portal side, the configuration in the interface requires the administrator to enter three values as NAME=value ; they are
These values are entered space-separated in the fields called "Extra QSUB Args"; there are two places for these:
cms_extra_qsub_args
extra_qsub_args
In the interface, for both models, the forms show them as
Extra 'qsub' options
Entering a configuration at the ToolConfig level has higher priority than at the Bourreau level. So the 'default' parameters for the Gcloud configuration can be made on the Bourreau, and a ToolConfig can override them. In general though, only the GCLOUD_IMAGE_BASENAME is likely to change for a ToolConfig.
My recommendation is:
For the Bourreau, provide all three:
and these will be the default for all tools on the bourreau; then if needed, for a specific ToolConfig, change the image name only:
It is acknowledged that the field names 'exta qsub args', initially created to hold command-line options for the "sbatch" or "qsub" commands, is no longer really appropriate for GCLOUD, but I plan to add notes in the interface to that effect.
Special review note: commits made on or after Feb 13th have not been tested yet, I'll test them in production. They're just esthetic changes. Famous last words.