Cleanup staging buckets in boskos

[ameukam]

Each GCP project in the boskos pool have a staging bucket used to upload binaries built during test execution.

Those buckets were never cleaned up and now impact our budget with the GCS pricing change.

We should clean up those buckets, make them regional and introduce some retention policy (7 days ?).

/sig testing
/milestone v1.27

[ameukam]

cc @BenTheElder @thockin

[ameukam]

For a boskos project:

gcloud alpha storage buckets list --project k8s-infra-e2e-boskos-001 --format='table(name,locationType,location)'
NAME                                        LOCATION_TYPE  LOCATION
kubernetes-staging-485128143e               multi-region   US
kubernetes-staging-485128143e-asia          multi-region   ASIA
kubernetes-staging-485128143e-eu            multi-region   EU
kubernetes-staging-485128143e-europe-west6  region         EUROPE-WEST6

gsutil du -sh gs://kubernetes-staging-485128143e
288.45 GiB   gs://kubernetes-staging-485128143e

[BenTheElder]

I think we should set a small TTL of no more than 1 day and rotate these to regional.

We could delete-and-recreate on every run safely in boskos leased projects, but in other fixed projects where CI jobs are still sharing them (5k node scale testing maybe?) we can't do that as easily.

There might be an argument to just update the scripts that ensure they exist to use the new settings when creating and then do a mass deletion of existing buckets one evening ...

[ameukam]

#115634 is breaking CI by preventing object creation during job execution.

kubernetes/kubernetes#116222 should help fix this.

At the same time I ran a quick script to clear retention policy around all the buckets. This should be enough fix the issue.

projects=$(gcloud projects list --filter='projectId~^k8s-infra-e2e-boskos-' --format="value(projectId)" --sort-by=projectId)

for prj in ${projects}; do
    bucket=$(gcloud storage buckets list --project "${prj}" --filter='location=us-central1' --format="value(name)")
    if [[ -n "${bucket}" ]]; then
        if ! gsutil ls "${bucket}" >/dev/null 2>&1; then
            echo "clearing retention policy for ${bucket} in ${prj}"
            gsutil retention clear "gs://${bucket}"
        fi
    fi
done

[dims]

thanks @ameukam