[FEATURE_REQUEST] Do not warn about dangling service linked to deployment or statefulset without a pod

[tspearconquest]

Description of the problem/feature request
Service definitions with a label selector matching a deployment or statefulset definition without any pod definition are generating a dangling service warning in kube-linter because the dangling service check appears to only concern itself with pods.

Description of the existing behavior vs. expected behavior
Consider the following service definition:

apiVersion: v1
kind: Service
metadata:
  name: appnameA
  labels:
    app: appnameALabel
spec:
  ports:
  - name: http
    port: 8080
    targetPort: 8080
  selector:
    app: appnameALabel
  type: ClusterIP

And the following (partial) deployment definition:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: appnameA
  labels:
    app: appnameALabel
spec:
  replicas: 1
  selector:
    matchLabels:
      app: appnameALabel

When running kube-linter, we get the following:

kubernetes/60-services/appnameA-svc.yaml: (object: <no namespace>/appnameA /v1, Kind=Service) no pods found matching service labels (map[app:appnameALabel]) (check: dangling-service, remediation: Make sure your service's selector correctly matches the labels on one of your deployments.)

Additional context
The warning message is technically correct, there is no pod which matches the service label. However this is not a dangling service, and should not generate a warning. We currently annotate for this but we are looking to improve the kubernetes definitions and this is one that sticks out because it's doesn't seem to be a real problem.

[viswajithiii]

Hi @tspearconquest, thanks for filing this issue. I will admit I've never seen a deployment definition without a pod spec. How does that work? If you kubectl create that, will it create pods with some default spec?

[tspearconquest]

Hey Viswajith, I checked in the namespace just now. It looks like it does create a pod with the same spec (or nearly the same, as I could not see any difference) as the deployment.

[viswajithiii]

But how does it know what image to create, for example?

[tspearconquest]

The definition I provided was only a partial example for demonstration purposes.

In our full setup, we have a CI pipeline which builds the image from our Dockerfile and the app source code, then runs kube-linter, and then deploys the app to a kubernetes cluster. The kubernetes definition is relatively simple but I cannot share the full spec as it is internal to my organization.

The deployment spec defines an image and the auto-generated pod uses that image.

[viswajithiii]

Ohh, got it. I was really confused. 😅

Here's a very minimal example I put together. In this example, kube-linter does not flag the service as a dangling service. You can see it does not find an error.

kube-linter lint - <<EOF --do-not-auto-add-defaults --include dangling-service
apiVersion: apps/v1
kind: Deployment
metadata:
  name: test
spec:
  selector:
    matchLabels:
      app: test
  template:
    metadata:
      labels:
        app: test
    spec:
      containers:
      - image: nginx:latest
        name: nginx 
---
apiVersion: v1
kind: Service
metadata:
  name: test
  labels:
    app: test
spec:
  ports:
  - name: http
    port: 8080
    targetPort: 8080
  selector:
    app: test
  type: ClusterIP
EOF

[viswajithiii]

Can you help me modify this example so as to reproduce the problem you're seeing?

[tspearconquest]

Hi Viswajith,

When I run kube-linter in the way you have done with a heredoc, using our internal definitions, it is not reporting the issue.

However, when I place them in separate files (deployment.yaml and service.yaml, for example) and run kube-linter lint . then I can see the warning.

Is kube-linter understanding that the deployment.yaml is part of the same context as service.yaml? It should do that. There should be no difference between your heredoc and my 2 files.

[viswajithiii]

Hmm, it should understand that it is part of the same context, and it does. I just tested it that way:

viswa:~/kube-linter-demo/yamls/temp (docker-desktop)$ ls
dep.yaml  svc.yaml
viswa:~/kube-linter-demo/yamls/temp (docker-desktop)$ cat dep.yaml 
apiVersion: apps/v1
kind: Deployment
metadata:
  name: test
spec:
  selector:
    matchLabels:
      app: test
  template:
    metadata:
      labels:
        app: test
    spec:
      containers:
      - image: nginx:latest
        name: nginx 

viswa:~/kube-linter-demo/yamls/temp (docker-desktop)$ cat svc.yaml 
apiVersion: apps/v1
kind: Deployment
metadata:
  name: test
spec:
  selector:
    matchLabels:
      app: test
  template:
    metadata:
      labels:
        app: test
    spec:
      containers:
      - image: nginx:latest
        name: nginx 
---
apiVersion: v1
kind: Service
metadata:
  name: test
  labels:
    app: test
spec:
  ports:
  - name: http
    port: 8080
    targetPort: 8080
  selector:
    app: test
  type: ClusterIP

viswa:~/kube-linter-demo/yamls/temp (docker-desktop)$ kube-linter lint . --do-not-auto-add-defaults --include dangling-service
KubeLinter 0.2.1-0-gc53952b8e3

No lint errors found!

[tspearconquest]

Your svc.yaml contains the deployment a second time...

[viswajithiii]

Your svc.yaml contains the deployment a second time...

Ah, my bad. I just fixed and retried. Same result though...

viswa:~/kube-linter-demo/yamls/temp (docker-desktop)$ ls
dep.yaml  svc.yaml
viswa:~/kube-linter-demo/yamls/temp (docker-desktop)$ cat dep.yaml 
apiVersion: apps/v1
kind: Deployment
metadata:
  name: test
spec:
  selector:
    matchLabels:
      app: test
  template:
    metadata:
      labels:
        app: test
    spec:
      containers:
      - image: nginx:latest
        name: nginx 

viswa:~/kube-linter-demo/yamls/temp (docker-desktop)$ cat svc.yaml 
apiVersion: v1
kind: Service
metadata:
  name: test
  labels:
    app: test
spec:
  ports:
  - name: http
    port: 8080
    targetPort: 8080
  selector:
    app: test
  type: ClusterIP

viswa:~/kube-linter-demo/yamls/temp (docker-desktop)$ kube-linter lint . --do-not-auto-add-defaults --include dangling-service
KubeLinter 0.2.1-0-gc53952b8e3

No lint errors found!

[tspearconquest]

Great, I have reproduced the issue on my end with these 2 manifests.
In my setup, we segregate the deployment definition and the service definition into subfolders and run kube-linter against the parent folder.

~/gitlab/workspaces/test ❯ ls
deploy/ svc/
~/gitlab/workspaces/test ❯ ls -R
deploy/ svc/

./deploy:
dep.yaml

./svc:
svc.yaml
~/gitlab/workspaces/test ❯ cat deploy/dep.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: test
spec:
  selector:
    matchLabels:
      app: test
  template:
    metadata:
      labels:
        app: test
    spec:
      containers:
      - image: nginx:latest
        name: nginx
~/gitlab/workspaces/test ❯ cat svc/svc.yaml
apiVersion: v1
kind: Service
metadata:
  name: test
  labels:
    app: test
spec:
  ports:
  - name: http
    port: 8080
    targetPort: 8080
  selector:
    app: test
  type: ClusterIP
~/gitlab/workspaces/test ❯ kube-linter lint . --do-not-auto-add-defaults --include dangling-service
KubeLinter 0.2.1-0-gc53952b8e3

svc/svc.yaml: (object: <no namespace>/test /v1, Kind=Service) no pods found matching service labels (map[app:test]) (check: dangling-service, remediation: Make sure your service's selector correctly matches the labels on one of your deployments.)

Error: found 1 lint errors

[tspearconquest]

Coincidentally, I found through further testing that an issue we have with service accounts being inaccurately reported as non-existent also is resolved by having our service account definitions in the same directory as the deployment. It seems there is some bug with parsing the subdirectories.

We would like to keep our directory structure. Please advise how the presence of subdirectories causes interdependent checks (sa <-> deployment <-> service) to fail?

[viswajithiii]

Aha! Yes, that is indeed working as expected. Your use-case is valid, though, so let's figure out how to solve for it.

Some background: kube-linter currently assumes that all YAMLs in a directory are part of the same app, and that YAMLs not in the same directory are not. In kube-linter terminology, each app is called a context. This explains the behaviour you are seeing.

The intention behind this design is to support running kube-linter lint . at the root of a git repository, and have it do all the work that's needed automatically, much like most code linters work. This means that we are supporting directory structures like the following, where multiple independent applications coexist in a repository:

Example 1:

my-git-root/
   my-first-app/
      my-first-dep.yaml
      my-first-svc.yaml
   my-second-app/
      my-second-dep.yaml
      my-second-svc.yaml

(For context, this is the approach we have used for storing configurations for our own internal services.)

Obviously, the weakness of this is that it doesn't handle use-cases like yours. Fundamentally, the problem is that we can't differentiate between a directory structure like yours, and one like the example above.

Whatever solution we design will have to handle the above case, as well as the following two cases:

Example 2:

my-git-root/
   my-app/
      deps/ 
        dep.yaml
       svc/
         svc.yaml

Example 3:

my-git-root/
   my-first-app/
      deps/ 
        dep.yaml
       svc/
         svc.yaml
   my-second-app/
      my-second-dep.yaml
      my-second-svc.yaml

I can think of a couple of different solutions, and I would be very interested in your opinion here:

1. Allow users to create a file (.kube-linterrc, say) in a directory, which allows them to specify (in a to-be-determined format) that all YAML files in this directory, recursively, should be treated as belonging to the same context. (Downside: requires users to create another file, which is some friction.)
2. Add a flag to kube-linter lint that instructs kube-linter to treat files discovered as belonging to the same context. (Downside: makes it impossible to handle cases like example 3 correctly, without multiple kube-linter lint invocations).

Curious to hear any other ideas you have as well, as well as whether you have a preference between the two.

[tspearconquest]

Hey Viswajith, yes I agree this is an interesting situation to have to account for.

I don't think option 1 will work in our case, as its an extra step to add that file to numerous pipelines.

Option 2, as you mentioned doesn't handle example 3, and we would want to accommodate other use cases, as we would hope others would do for us, so I have thought of another possibility and need your insight.

What can be done with annotations by kube-linter?
Could a new annotation be added that does as described in option 2, without the downsides of option 2?
The ignore-check.kube-linter.io/check-name syntax is great for ignoring checks, but if we want to hard code the context as one of the 3 options, could a new annotation be created to handle that?

[viswajithiii]

Hmm, what would that look like in this case? It feels like that would require a lot more effort than option 1? To clarify, with option 1, I'm envisioning that the config file would be checked with to your code, and live and be maintained along with your Kube YAMLs.