Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Release.key URL redirect causes problem with Cloud Init and gpg #6188

Closed
johnmanko opened this issue Dec 14, 2023 · 8 comments
Closed

Release.key URL redirect causes problem with Cloud Init and gpg #6188

johnmanko opened this issue Dec 14, 2023 · 8 comments
Labels
sig/k8s-infra Categorizes an issue or PR as relevant to SIG K8s Infra. sig/release Categorizes an issue or PR as relevant to SIG Release.

Comments

@johnmanko
Copy link

johnmanko commented Dec 14, 2023 •
edited
Loading

I've got a Clout Init setup that creates/adds apt sources upon VM creation (my original query into this can be found at Stackoverflow).

The official installation documentation states the following for configuring deb sources on Debian-based systems:

curl -fsSL https://pkgs.k8s.io/core:/stable:/v1.29/deb/Release.key | sudo gpg --dearmor -o /etc/apt/keyrings/kubernetes-apt-keyring.gpg

The URL https://pkgs.k8s.io/core:/stable:/v1.29/deb/Release.key is a 302 redirect to https://prod-cdn.packages.k8s.io/repositories/isv:/kubernetes:/core:/stable:/v1.28/deb/Release.key

When configuring a system via Cloud Init, gpg fails:

    apt:
      sources:
        docker:
          keyid: '9DC858229FC7DD38854AE2D88D81803C0EBFCD88'
          keyserver: 'https://download.docker.com/linux/ubuntu/gpg'
          source: 'deb [signed-by=$KEY_FILE] https://download.docker.com/linux/ubuntu mantic stable'
        kubernetes:
          keyid: 'DE15B14486CD377B9E876E1A234654DA9A296436'
          keyserver: 'https://pkgs.k8s.io/core:/stable:/v1.28/deb/Release.key'
          source: 'deb [signed-by=$KEY_FILE] https://pkgs.k8s.io/core:/stable:/v1.28/deb/ /'
      

2023-12-10 18:29:02,444 - gpg.py[ERROR]: Failed to obtain gpg key DE15B14486CD377B9E876E1A234654DA9A296436
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/cloudinit/gpg.py", line 101, in recv_key
    naplen = next(sleeps)
             ^^^^^^^^^^^^
StopIteration

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/cloudinit/gpg.py", line 130, in getkeybyid
    recv_key(keyid, keyserver=keyserver)
  File "/usr/lib/python3/dist-packages/cloudinit/gpg.py", line 109, in recv_key
    raise ValueError(
ValueError: Failed to import key 'DE15B14486CD377B9E876E1A234654DA9A296436' from keyserver 'https://pkgs.k8s.io/core:/stable:/v1.28/deb/Release.key' after 3 tries: Unexpected error while running command.
Command: ['gpg', '--no-tty', '--keyserver=https://pkgs.k8s.io/core:/stable:/v1.28/deb/Release.key', '--recv-keys', 'DE15B14486CD377B9E876E1A234654DA9A296436']
Exit code: 2
Reason: -
Stdout: 
Stderr: gpg: WARNING: unacceptable HTTP redirect from server was cleaned up
        gpg: keyserver receive failed: No data
2023-12-10 18:29:02,465 - gpg.py[WARNING]: Failed delete key "DE15B14486CD377B9E876E1A234654DA9A296436": Unexpected error while running command.
Command: ['gpg', '--batch', '--yes', '--delete-keys', 'DE15B14486CD377B9E876E1A234654DA9A296436']
Exit code: 2
Reason: -
Stdout: 
Stderr: gpg: key "DE15B14486CD377B9E876E1A234654DA9A296436" not found: Not found
        gpg: DE15B14486CD377B9E876E1A234654DA9A296436: delete key failed: Not found
2023-12-10 18:29:02,466 - util.py[WARNING]: Running module apt_configure (<module 'cloudinit.config.cc_apt_configure' from '/usr/lib/python3/dist-packages/cloudinit/config/cc_apt_configure.py'>) failed
Cloud-init v. 23.3.1-0ubuntu2 running 'modules:final' at Sun, 10 Dec 2023 18:29:02 +0000. Up 23.60 seconds.
Cloud-init v. 23.3.1-0ubuntu2 finished at Sun, 10 Dec 2023 18:29:02 +0000. Datasource DataSourceNoCloud [seed=/dev/sr0][dsmode=net].  Up 23.74 seconds

$ curl https://pkgs.k8s.io/core:/stable:/v1.28/deb/Release.key
<html>
<head><title>302 Found</title></head>
<body>
<center><h1>302 Found</h1></center>
<hr><center>nginx</center>
</body>
</html>

The solution is to use the redirect URL in the Cloud Init config.

I guess this begs the questions:

  1. Should the documentation be update to reflect this issue?
  2. How stable is the redirect URL?
@johnmanko johnmanko added the sig/k8s-infra Categorizes an issue or PR as relevant to SIG K8s Infra. label Dec 14, 2023
@ScottS-byte
Copy link

Tickle this as it still happens.

@BenTheElder
Copy link
Member

/sig release
@xmudrii

@k8s-ci-robot k8s-ci-robot added the sig/release Categorizes an issue or PR as relevant to SIG Release. label Mar 6, 2024
@BenTheElder
Copy link
Member

Should the documentation be update to reflect this issue?

This seems like a cloud-init bug, if anything?

How stable is the redirect URL?

Officially you should not depend on implementation details. We could have to switch backing hosts at any time. It is community run, funded on donations from vendors, and subject to change at any time.

For registry.k8s.io (container images) that is documented at https://registry.k8s.io#stability

I don't think we have a landing page or separate docs for pkgs.k8s.io currently.

Sorry for the delayed response, most of us are doing a LOT these days.

@BenTheElder
Copy link
Member

Lots of services serve 302 to current hosts, cloud-init really ought to have a curl -L equivalent for this IMHO ...

@johnmanko
Copy link
Author

@BenTheElder

If you look at the error, it's gpg that fails before the keyserver specified in the docs is a 302 redirect. gpg doesn't like that.

See my community question to get around this issue, notably "gpg doesn't have a built-in option to automatically follow redirects": https://serverfault.com/questions/1149572/cloud-init-cant-create-apt-source-from-k8s-io-due-to-302-redirect

I agree that this is Cloud Init, or rather gpg, issue at it's core, but I'm surprised that the docs don't at least mention the gpg restriction - that seems like it would be a common issue that users would face following the docs.

Anyway, if nothing else, it's documented here.

@xmudrii
Copy link
Member

xmudrii commented Mar 7, 2024

Should the documentation be update to reflect this issue?

The documentation covers a way that's known to work; it's to use curl and gpg combined together to the import the key. If you don't want to follow that, it's up to you to figure out the details. I don't think that we should have every possible combination documented.

This is in my opinion a topic for SIG Docs rather than SIG Release, I recommend reaching out to them if you think that the documentation should be improved.

How stable is the redirect URL?

@BenTheElder clearly described this (thank you!), we don't provide any guarantees on what URL is actually used by pkgs.k8s.io, that can be changed at any time and without any prior notification. If cloud-init can't handle this, I recommend finding another way around this.

This is documented in the official announcement, although it would be nice if pkgs.k8s.io had a landing page with the relevant information.

@xmudrii xmudrii mentioned this issue Mar 7, 2024
Open
@xmudrii
Copy link
Member

xmudrii commented Mar 7, 2024

I created a new issue to track adding a landing page for pkgs.k8s.io: kubernetes/release#3496
I think we can close this issue because there's nothing else that we can do on our side. If you have any further questions or issues, please reopen this issue or create a new one.
/close

@k8s-ci-robot
Copy link
Contributor

@xmudrii: Closing this issue.

In response to this:

I created a new issue to track adding a landing page for pkgs.k8s.io: kubernetes/release#3496
I think we can close this issue because there's nothing else that we can do on our side. If you have any further questions or issues, please reopen this issue or create a new one.
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
sig/k8s-infra Categorizes an issue or PR as relevant to SIG K8s Infra. sig/release Categorizes an issue or PR as relevant to SIG Release.
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
5 participants
@BenTheElder @johnmanko @xmudrii @k8s-ci-robot @ScottS-byte