instance-manager always creates normal EC2 instances regardless of the spotPrice value

[amandre]

Is this a BUG REPORT or FEATURE REQUEST?: BUG REPORT

What happened:
spotPrice does not seem to behave properly. When we set it either to the value higher than the current price or lower than the current price, the instance always gets created and is always marked as lifecycle=normal in AWS EC2 view, while instance-manager claims it is lifecycle=spot.

What you expected to happen:

1. InstanceGroup is Ready
2. k8s node becomes NotReady because the spot Price does not match the current price

How to reproduce it (as minimally and precisely as possible):

1. create InstanceGroup with spotPrice="0.0001" and instanceType=m5.xlarge
2. check the EC2 type and Spot Requests in AWS console (EC2 type is normal and Spot Requests are empty)

Anything else we need to know?:
Documentation is a bit misleading saying that spotPrice is the minimal spot price.
As per https://github.com/keikoproj/instance-manager/blob/master/docs/EKS.md#eksconfiguration

spotPrice: <string> : must be a decimal number represnting a minimal spot price

Environment:

• Kubernetes version:

kubectl version -o yaml
clientVersion:
  buildDate: "2021-09-15T21:31:32Z"
  compiler: gc
  gitCommit: 8b5a19147530eaac9476b0ab82980b4088bbc1b2
  gitTreeState: clean
  gitVersion: v1.22.2
  goVersion: go1.16.8
  major: "1"
  minor: "22"
  platform: linux/amd64
serverVersion:
  buildDate: "2021-07-31T01:36:57Z"
  compiler: gc
  gitCommit: 8df2700a72a2598fa3a67c05126fa158fd839620
  gitTreeState: clean
  gitVersion: v1.19.13-eks-8df270
  goVersion: go1.15.14
  major: "1"
  minor: 19+
  platform: linux/amd64

Other debugging information (if applicable):

• InstanceGroup status:

Status:
  Active Launch Template Name: foobar-low-price-20211026052943
  Active Scaling Group Name:    foobar-low-price
  Conditions:
    Status:                     True
    Type:                       NodesReady
  Current Max:                  8
  Current Min:                  1
  Current State:                Ready
  Latest Template Version:      2
  Lifecycle:                    spot
  Nodes Instance Role Arn:      arn:aws:iam::foobar123:role/foo-role
  Provisioner:                  eks
  Strategy:                     crd
  Strategy Resource Name:       rollup
  Strategy Resource Namespace:  instance-manager

• controller logs:

k logs -n instance-manager instance-manager-d69bf9db4-54qw6 | grep low-price
2021-10-27T07:35:33.752Z        INFO    controllers.instancegroup.eks   bootstrapping arn to aws-auth   {"instancegroup": "instance-manager/foobar-low-price", "arn": "arn:aws:iam::foobar123:role/foo-role"}
2021-10-27T07:35:33.761Z        INFO    controllers.instancegroup.eks   waiting for node readiness conditions   {"instancegroup": "instance-manager/foobar-low-price"}
2021-10-27T07:35:33.761Z        INFO    v1alpha1        state transition occured        {"instancegroup": "instance-manager/foobar-low-price", "state": "InitUpgrade", "previousState": "ReconcileModifying"}
2021-10-27T07:35:33.807Z        INFO    kubernetes-provider     watching custom resource status {"instancegroup": "instance-manager/foobar-low-price", "resource": "rollup", "status": "completed"}
2021-10-27T07:35:33.807Z        INFO    kubernetes-provider     custom resource succeeded       {"instancegroup": "instance-manager/foobar-low-price", "resource": "rollup", "status": "completed"}
2021-10-27T07:35:33.807Z        INFO    controllers.instancegroup.eks   strategy processing completed   {"instancegroup": "instance-manager/foobar-low-price", "strategy": "crd"}
2021-10-27T07:35:33.807Z        INFO    controllers.instancegroup.eks   waiting for node readiness conditions   {"instancegroup": "instance-manager/foobar-low-price"}
2021-10-27T07:35:33.807Z        INFO    controllers.instancegroup       reconcile event ended with requeue      {"instancegroup": "instance-manager/foobar-low-price", "provisioner": "eks"}
2021-10-27T07:35:33.807Z        INFO    controllers.instancegroup       patching resource status        {"instancegroup": "instance-manager/foobar-low-price", "patch": "{}", "resourceVersion": "13224500"}
2021-10-27T07:35:43.818Z        INFO    controllers.instancegroup       reconcile event started {"instancegroup": "instance-manager/foobar-low-price", "provisioner": "eks"}
2021-10-27T07:35:43.818Z        INFO    v1alpha1        state transition occured        {"instancegroup": "instance-manager/foobar-low-price", "state": "Init", "previousState": "InitUpgrade"}
2021-10-27T07:35:44.483Z        INFO    controllers.instancegroup.eks   using manually configured spot price    {"instancegroup": "instance-manager/foobar-low-price", "spotPrice": "0.0001"}
2021-10-27T07:35:46.111Z        INFO    v1alpha1        state transition occured        {"instancegroup": "instance-manager/foobar-low-price", "state": "InitUpdate", "previousState": "Init"}
2021-10-27T07:35:46.111Z        INFO    v1alpha1        state transition occured        {"instancegroup": "instance-manager/foobar-low-price", "state": "ReconcileModifying", "previousState": "InitUpdate"}
2021-10-27T07:35:46.112Z        INFO    scaling drift not detected      {"instancegroup": "instance-manager/foobar-low-price"}
2021-10-27T07:35:46.115Z        INFO    controllers.instancegroup.eks   bootstrapping arn to aws-auth   {"instancegroup": "instance-manager/foobar-low-price", "arn": "arn:aws:iam::foobar123:role/foo-role"}
2021-10-27T07:35:46.122Z        INFO    controllers.instancegroup.eks   waiting for node readiness conditions   {"instancegroup": "instance-manager/foobar-low-price"}
2021-10-27T07:35:46.127Z        INFO    controllers.instancegroup.eks   desired nodes are ready {"instancegroup": "instance-manager/foobar-low-price", "instances": "i-0a338c6f48cce3ab9"}
2021-10-27T07:35:46.127Z        INFO    v1alpha1        state transition occured        {"instancegroup": "instance-manager/foobar-low-price", "state": "ReconcileModified", "previousState": "ReconcileModifying"}
2021-10-27T07:35:46.127Z        INFO    controllers.instancegroup.eks   bootstrapping arn to aws-auth   {"instancegroup": "instance-manager/foobar-low-price", "arn": "arn:aws:iam::foobar123:role/foo-role"}
2021-10-27T07:35:46.132Z        INFO    v1alpha1        state transition occured        {"instancegroup": "instance-manager/foobar-low-price", "state": "Ready", "previousState": "ReconcileModified"}
2021-10-27T07:35:46.132Z        INFO    controllers.instancegroup       reconcile event ended   {"instancegroup": "instance-manager/foobar-low-price", "provisioner": "eks"}
2021-10-27T07:35:46.132Z        INFO    controllers.instancegroup       patching resource status        {"instancegroup": "instance-manager/foobar-low-price", "patch": "{\"status\":{\"conditions\":[{\"status\":\"True\",\"type\":\"NodesReady\"}],\"currentState\":\"Ready\"}}", "resourceVersion": "13224500"}
2021-10-27T07:35:46.146Z        INFO    controllers.instancegroup       reconcile event started {"instancegroup": "instance-manager/foobar-low-price", "provisioner": "eks"}
2021-10-27T07:35:46.146Z        INFO    v1alpha1        state transition occured        {"instancegroup": "instance-manager/foobar-low-price", "state": "Init", "previousState": "Ready"}
2021-10-27T07:35:46.375Z        INFO    controllers.instancegroup.eks   using manually configured spot price    {"instancegroup": "instance-manager/foobar-low-price", "spotPrice": "0.0001"}
2021-10-27T07:35:47.033Z        INFO    v1alpha1        state transition occured        {"instancegroup": "instance-manager/foobar-low-price", "state": "InitUpdate", "previousState": "Init"}
2021-10-27T07:35:47.034Z        INFO    v1alpha1        state transition occured        {"instancegroup": "instance-manager/foobar-low-price", "state": "ReconcileModifying", "previousState": "InitUpdate"}
2021-10-27T07:35:47.034Z        INFO    scaling drift not detected      {"instancegroup": "instance-manager/foobar-low-price"}
2021-10-27T07:35:47.039Z        INFO    controllers.instancegroup.eks   bootstrapping arn to aws-auth   {"instancegroup": "instance-manager/foobar-low-price", "arn": "arn:aws:iam::foobar123:role/foo-role"}
2021-10-27T07:35:47.047Z        INFO    controllers.instancegroup.eks   waiting for node readiness conditions   {"instancegroup": "instance-manager/foobar-low-price"}
2021-10-27T07:35:47.047Z        INFO    controllers.instancegroup.eks   desired nodes are ready {"instancegroup": "instance-manager/foobar-low-price", "instances": "i-0a338c6f48cce3ab9"}
2021-10-27T07:35:47.047Z        INFO    v1alpha1        state transition occured        {"instancegroup": "instance-manager/foobar-low-price", "state": "ReconcileModified", "previousState": "ReconcileModifying"}
2021-10-27T07:35:47.047Z        INFO    controllers.instancegroup.eks   bootstrapping arn to aws-auth   {"instancegroup": "instance-manager/foobar-low-price", "arn": "arn:aws:iam::foobar123:role/foo-role"}
2021-10-27T07:35:47.053Z        INFO    v1alpha1        state transition occured        {"instancegroup": "instance-manager/foobar-low-price", "state": "Ready", "previousState": "ReconcileModified"}
2021-10-27T07:35:47.053Z        INFO    controllers.instancegroup       reconcile event ended   {"instancegroup": "instance-manager/foobar-low-price", "provisioner": "eks"}
2021-10-27T07:35:47.053Z        INFO    controllers.instancegroup       patching resource status        {"instancegroup": "instance-manager/foobar-low-price", "patch": "{}", "resourceVersion": "13227720"}

[shrinandj]

Can you check if the LaunchConfig/LaunchTemplate created by the instance-manager? Does it show the spot price being set?

[eytan-avisror]

There might be a problem the with setting of lifecycle=spot in this case. I think the current logic depends on whether a spot price is set or not, and not whether instances are actually switched to spot.
In the case of using spot recommendations with minion manager that works fine since the price is available, but if someone just sets a spotPrice which is not available the result would be lifecycle=spot and it cannot switch because the price is not available.

My recommendation for now is to either use spot recommendations via minion-manager or switch to spot via LaunchTemplate/Mixed Instance Policies which is a more automated mechanism via AWS.

https://github.com/keikoproj/instance-manager/blob/master/docs/EKS.md#spot-instances

Leaving this ticket open so we can look into better logic around manually setting spotPrice

[amandre]

Thank you guys for your thoughts!
Yes, it worked with MixedInstancePolicies or with minion-manager but was curious if it is expected to work for spotPrice.
I'll gladly collaborate and help in introducing a better logic here (or at least fixing the documentation explaining the desired logic).