Wait for the VMSS capacity update for 1 minute before async

[hakman]

What type of PR is this?

/kind bug

What this PR does / why we need it:

In rare situations when the VMSS fails to scale (OverconstrainedAllocationRequest, OverconstrainedZonalAllocationRequest, ...), the error is not handled until the context timeout expires (max-node-provision-time).

Which issue(s) this PR fixes:

Fixes: N/A

Special notes for your reviewer:

This is just a suggestion from someone that is not that familiar with the CAS code, there may be better options available.

Does this PR introduce a user-facing change?

NONE

Additional documentation e.g., KEPs (Kubernetes Enhancement Proposals), usage docs, etc.:

N/A

[hakman]

/cc @tallaxes
/assign @tallaxes

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: hakman
Once this PR has been reviewed and has the lgtm label, please ask for approval from tallaxes. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

• cluster-autoscaler/cloudprovider/azure/OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[linux-foundation-easycla]

The committers listed above are authorized under a signed CLA.

• ✅ login: hakman / name: Ciprian Hacman (5292339, a292788)

[jackfrancis]

rather than throw away this error here I'd get it and do something like this:

klog.Errorf("waitForCreateOrUpdateInstances(%s) failed, err: %v", scaleSet.Name, err)

And then get rid of the above log statement in waitForCreateOrUpdateInstances

[hakman]

Done, I also kept the success message, same as I did above.

[hakman]

The issue should also be addressed by setting AZURE_ENABLE_FAST_DELETE_ON_FAILED_PROVISIONING=true, see #8323.