nvmeof service fails to start

[sanderdescamps]

The nvmeof service continues to fail. I tweaked multiple properties in the service file but without success. Even with the standard commands the service doesn't start. Could someone have a look at the logs and tell me in which direction I need to look for the issue?

Service file:

---
service_type: nvmeof
service_name: nvmeof.gateway-group
placement:
  count: 2
  hosts:
    - ctrl01.mydomain.com
    - ctrl02.mydomain.com
    # - ctrl03.mydomain.com
networks:
- 10.0.0.0/24
- 10.1.0.0/22
- 10.2.0.0/22
abort_on_errors: false
spec:
  pool: nvme-pool
  group: gateway-group
  addr_map:
    ctrl01.mydomain.com: 10.0.0.85
    ctrl02.mydomain.com: 10.0.0.131
    ctrl03.mydomain.com: 10.0.0.47
  discovery_addr_map:
    ctrl01.mydomain.com: 10.1.1.101
    ctrl02.mydomain.com: 10.1.1.102
    ctrl03.mydomain.com: 10.1.1.103
  log_level: DEBUG
  abort_on_errors: false    #<--(doesn't seem to work)
  spdk_protocol_log_level: DEBUG
  # enable_monitor_client: false

Systemd logs

nvmeof.gateway-group.ctrl01-anon.log

Setup in short:

• Ceph version: 20.2.0
• NVMe-OF version: 1.5.16
• Nodes:
  • 7 nodes total (4 data OSD + 3 ctrl/nvmeof-gw)
• Network:
  • 10.0.0.0/24 mgmt network
  • 10.1.0.0/22 public network
  • 10.2.0.0/22 cluster network

Context:
The Ceph cluster is installed as a POC to test if Ceph is a viable storage alternative for our environment. We need the NVMe-oF gateway to be able to connect our VMware environment. Any thoughts or ideas? Feel free to share as well.

[caroav]

From the log, it seems that the nvmeof gw mon client is not able to connect to the ceph monitor. The ceph monitor is running on ceph.

[caroav]

enable_monitor_client cannot be false !!!!!

[sanderdescamps]

I've had the feeling indeed that enable_monitor_client is important. That is also why it is in comment. But I've used enable_monitor_client=false for debugging because this is the only way I've found to prevent automatic removal of the container once it fails. Because the abort_on_errors propery doesn't do anything. If you know a better way, let me know.

I also like to know what the addresses (addr and discovery_addr) in the config exactly are. From what I see the nvmeof service exists out of 5 processes on 5 different tcp-ports. The gateway mgmt API on 5500, nvme monitor group on 5499, NVMe-of on 4420, NVMe-of discovery on 8009 and Prometheus metrics on 10008. Which setting maps to which process?

[caroav]

I would start without defining the "addr_map" and the "discovery_addr_map" in the spec. Not defining it is the default and the gws should come up without issues, using the host ip for these addresses.

All of the ports you mention belong to the nvmeof gw - but internally if you want to know:
5500 is the grpc server (for example for CLI/remote commands)
8009 is the nvmeof discovery port - internally the gw is running a discovery service listening on this port
4420 is used by spdk that also started by the gw

Let me know if removing the "addr_map" and the "discovery_addr_map" helped in bringing up the gws.

[sanderdescamps]

I've tried it without "addr_map" and the "discovery_addr_map" but result is the same. The both gateways continue to fail.

[caroav]

How do you deploy the gws? Are you using cephadm? If not, what exactly are you doing?

[sanderdescamps]

Yes, I've used cephadm to bootstrap the cluster on top of Podman.

[caroav]

can you send the command you used to deploy the gws?

[sanderdescamps]

I've reinstalled all hosts and started fresh. I redeployed the nvmeof gateways and still get the same issue. Bellow you have the summary of the installation I did.

• Started with standard RHEL9.7 image

• Disable selinux (just to be sure)

• Configure network interfaces

  • bond of 4x25Gbit
  • untagged mgmt 10.0.0.0/24
  • VLAN tagged public network 10.1.0.0/22
  • VLAN tagged cluster network 10.2.0.0/22

• Create ceph-conn user and configure the ssh

• Configure the ceph repos

• Configure chronny

• Install cephadm, ceph-common, ceph-base and podman

• Add nvme_fabrics kernel module

• Configure http_proxy in podman.service and /etc/containers/containers.conf

• Bootstrap Ceph

    cephadm bootstrap  --config /opt/ceph/ceph.conf --mon-ip 10.1.1.101 --cluster-network 10.2.0.0/22 --ssh-user ceph-conn --ssh-private-key /opt/ceph/ssh/ceph_id_ed25519 --ssh-public-key /opt/ceph/ssh/ceph_id_ed25519.pub --initial-dashboard-password '<admin-password>' --initial-dashboard-user 'admin' --dashboard-password-noupdate --allow-fqdn-hostname --with-centralized-logging | tee ceph-bootstrap.log
  

  ceph.conf

    [global]
    fsid = de63b81e-2d37-4344-8bfd-e528b27fc46d
    mon_host = 10.1.1.101, 10.1.1.111, 10.1.1.112
    public_network = 10.1.0.0/22
    cluster_network = 10.2.0.0/22
  
    ## MON
    [mon]
    mon_allow_pool_delete = true
  
    [mon.ctrl01]
    host = ctrl01.mydomain.com
    # mon_addr = 10.1.1.101
  
    [mon.data01]
    host = data01.mydomain.com
    # mon_addr = 10.1.1.111
  
    [mon.data02]
    host = data02.mydomain.com
    # mon_addr = 10.1.1.112
  

• Add other hosts using service files like this

    ---
    addr: data02.mydomain.com
    hostname: data02.mydomain.com
    labels:
    - osd
    - mon
    service_name: data02
    service_type: host
  

• Add osd's using service file

    service_type: osd
    service_id: encrypted
    placement:
      label: osd
    spec:
      crush_device_class: nvme
      objectstore: bluestore
      data_devices:
    	all: true
      encrypted: true
  

• Create pool with the name nvme-pool

    [root@ctrl01 user]# ceph osd pool ls detail
    pool 1 '.mgr' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode on last_change 44 flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr read_balance_score 30.00
    pool 2 'nvme-pool' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change 99 lfor 0/0/97 flags hashpspool stripe_width 0 application rbd read_balance_score 2.81
  

• Deploy nvmeof gateway using service file

    ---
    service_type: nvmeof
    service_id: gateway-group
    service_name: nvmeof.gateway-group
    placement:
      count: 2
      hosts:
    	- ctrl01.mydomain.com
    	- ctrl02.mydomain.com
    networks:
    - 10.1.0.0/22
    - 10.2.0.0/22
    - 10.0.0.0/24
    spec:
      pool: nvme-pool
      group: gateway-group
      addr_map:
    	ctrl01.mydomain.com: 10.0.0.85
    	ctrl02.mydomain.com: 10.0.0.131
    	ctrl03.mydomain.com: 10.0.0.47
      discovery_addr_map:
    	ctrl01.mydomain.com: 10.1.1.101
    	ctrl02.mydomain.com: 10.1.1.102
    	ctrl03.mydomain.com: 10.1.1.103
      log_level: DEBUG
      abort_on_errors: false
      abort_discovery_on_errors: false
      spdk_protocol_log_level: DEBUG
  

Here are the logs again.
nvmeof-gateway-2-ano.log

[webalexeu]

I've tested on a fresh simple deployment and got the same error

-1> 2026-01-22T13:36:55.635+0000 7f381096b640  4 nvmeofgw void NVMeofGwMonitorClient::connect_panic() Triggering a panic: did not receive initial map from monitor, elapsed 32, configured connect panic duration 30 seconds.

0> 2026-01-22T13:36:55.638+0000 7f381096b640 -1 *** Caught signal (Aborted) **

I use this command to deploy it
ceph orch apply nvmeof nvme group1 --placement="3 ceph1 ceph2 ceph3"

Let me know if other commands or logs can help for the investigation

[sanderdescamps]

With the help of some colleagues, we found the problem. In our environment we use a http proxy. The http proxy was configured on the Podman engine and also injected as environment variable into the containers. It turned out that not all Ceph containers properly handle the no_proxy env var. no_proxy was set to exclude all internal domains and private cidr block but the cidr blocks were not properly applied. Causing the issues described above.

We changed the proxy config so it is only applied to the Podman engine and now the nvmeof service is working properly.

The inconsistent handling of the no_proxy env var is a known issue. You can find plenty of articles about it. I'm not sure if we can fix it here. But can it be added into the Ceph documentation?

Below you have the working proxy config

File: /etc/containers/containers.conf

[containers]
http_proxy=false

[engine]
env = [
"http_proxy=http://proxy.mydomain.com:3128", 
"https_proxy=http://proxy.mydomain.com:3128",
"no_proxy=.ginfra.net,.giservices.io,127.0.0.0/8,172.16.0.0/12,192.168.0.0/16,10.0.0.0/8",
"HTTP_PROXY=http://proxy.mydomain.com:3128",
"HTTPs_PROXY=http://proxy.mydomain.com:3128",
"NO_PROXY=.ginfra.net,.giservices.io,127.0.0.0/8,172.16.0.0/12,192.168.0.0/16,10.0.0.0/8"
]