x509 auth always fails unless connection_options: - tlsAllowInvalidHostnames=true

[davidwartell]

SUMMARY

x509 auth always fails unless:
connection_options: - tlsAllowInvalidHostnames=true

ISSUE TYPE

• Bug Report

COMPONENT NAME

x509 auth handling with pymongo

ANSIBLE VERSION

From mysql replica set primary node:

ansible --version
ansible [core 2.13.6]
  config file = /etc/ansible/ansible.cfg
  configured module search path = ['/root/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/lib/python3/dist-packages/ansible
  ansible collection location = /root/.ansible/collections:/usr/share/ansible/collections
  executable location = /usr/bin/ansible
  python version = 3.10.6 (main, Nov  2 2022, 18:53:38) [GCC 11.3.0]
  jinja version = 3.0.3
  libyaml = True

From controller host:

ansible --version
ansible [core 2.13.6]
  config file = /Users/dwartell/freebird/rwsecure-pod-dev/ansible/ansible.cfg
  configured module search path = ['/Users/dwartell/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /opt/homebrew/Cellar/ansible/6.6.0/libexec/lib/python3.10/site-packages/ansible
  ansible collection location = /Users/dwartell/.ansible/collections:/usr/share/ansible/collections
  executable location = /opt/homebrew/bin/ansible
  python version = 3.10.8 (main, Oct 13 2022, 09:48:40) [Clang 14.0.0 (clang-1400.0.29.102)]
  jinja version = 3.1.2
  libyaml = True

Pymongo version on replica set master node

python3 -m pip list | grep mongo
pymongo                4.3.3

COLLECTION VERSION

ansible-galaxy collection list | grep mong
community.mongodb             1.4.2

CONFIGURATION

DEFAULT_HOST_LIST(/Users/dwartell/freebird/rwsecure-pod-dev/ansible/ansible.cfg) = ['/Users/dwartell/freebird/rwsecure-pod-dev/ansible/inventory']
DEFAULT_PRIVATE_KEY_FILE(/Users/dwartell/freebird/rwsecure-pod-dev/ansible/ansible.cfg) = /Users/dwartell/freebird/rwsecure-pod-dev/terraform/certs/bastionhost/id_rsa
DEFAULT_REMOTE_USER(/Users/dwartell/freebird/rwsecure-pod-dev/ansible/ansible.cfg) = ubuntu

OS / ENVIRONMENT

mysql replica set primary node:

cat /proc/version
Linux version 5.15.0-1023-aws (buildd@bos02-arm64-076) (gcc (Ubuntu 11.2.0-19ubuntu1) 11.2.0, GNU ld (GNU Binutils for Ubuntu) 2.38) #27-Ubuntu SMP Thu Oct 20 16:46:19 UTC 2022
lsb_release -r
Release:	22.04

mongosh --tls --host localhost --tlsCertificateKeyFile /etc/mongo-certs/mongo-rwadmin-user-cert-and-key.pem --tlsCAFile /etc/mongo-certs/mongo-ca-cert.pem --authenticationDatabase '$external' --authenticationMechanism MONGODB-X509 --eval "db.version()"
Current Mongosh Log ID:	6387f46f68b9c93944b9c978
Connecting to:		mongodb://localhost:27017/?directConnection=true&serverSelectionTimeoutMS=2000&tls=true&tlsCertificateKeyFile=%2Fetc%2Fmongo-certs%2Fmongo-rwadmin-user-cert-and-key.pem&tlsCAFile=%2Fetc%2Fmongo-certs%2Fmongo-ca-cert.pem&authSource=%24external&authMechanism=MONGODB-X509&appName=mongosh+1.6.0
Using MongoDB:		6.0.3
Using Mongosh:		1.6.0

For mongosh info see: https://docs.mongodb.com/mongodb-shell/

------
   The server generated these startup warnings when booting
   2022-11-30T17:58:03.060+00:00: Using the XFS filesystem is strongly recommended with the WiredTiger storage engine. See http://dochub.mongodb.org/core/prodnotes-filesystem
------

6.0.3

Controller host:
Macos 13.0.1 (22A400) arm64

STEPS TO REPRODUCE

#!/usr/bin/env ansible-playbook

- name: Check replica set status for all replica set nodes are in a healthy state
  hosts: mongo
  gather_facts: no
  tasks:
    - name: check mongo replica set state
      become: true
      community.mongodb.mongodb_status:
        login_host: localhost
        replica_set: rs0
        auth_mechanism: "MONGODB-X509"
        tls: true
        tlsCAFile: /etc/mongo-certs/mongo-ca-cert.pem
        tlsCertificateKeyFile: /etc/mongo-certs/mongo-rwadmin-user-cert-and-key.pem
        login_database: "$external"
        # this tlsAllowInvalidHostnames option is needed or the ansible community mongo ansible collection wont work with x509
        # must be a defect, works correctly with mongosh
        #connection_options:
        #  - "tlsAllowInvalidHostnames=true"

# mongosh --tls --host localhost --tlsCertificateKeyFile /etc/mongo-certs/mongo-rwadmin-user-cert-and-key.pem --tlsCAFile /etc/mongo-certs/mongo-ca-cert.pem --authenticationDatabase '$external' --authenticationMechanism MONGODB-X509 admin
Current Mongosh Log ID:	6387f52f3e68e0f7ec7a7caa
Connecting to:		mongodb://localhost:27017/admin?directConnection=true&serverSelectionTimeoutMS=2000&tls=true&tlsCertificateKeyFile=%2Fetc%2Fmongo-certs%2Fmongo-rwadmin-user-cert-and-key.pem&tlsCAFile=%2Fetc%2Fmongo-certs%2Fmongo-ca-cert.pem&authSource=%24external&authMechanism=MONGODB-X509&appName=mongosh+1.6.0
Using MongoDB:		6.0.3
Using Mongosh:		1.6.0

For mongosh info see: https://docs.mongodb.com/mongodb-shell/

------
   The server generated these startup warnings when booting
   2022-11-30T17:58:03.060+00:00: Using the XFS filesystem is strongly recommended with the WiredTiger storage engine. See http://dochub.mongodb.org/core/prodnotes-filesystem
------

rs0 [direct: primary] admin> db.getSiblingDB("$external").auth({mechanism: "MONGODB-X509", user: "emailAddress=cert@stealthsoft.io,CN=rwadmin,OU=mongo-user.dw.dev.stealthsoft.io,O=StealthMode Inc,L=Austin,ST=Texas,C=US"})
{ ok: 1 }
rs0 [direct: primary] admin> db.getSiblingDB("$external").getUsers();
{
  users: [
    {
      _id: '$external.emailAddress=cert@stealthsoft.io,CN=rwadmin,OU=mongo-user.dw.dev.stealthsoft.io,O=StealthMode Inc,L=Austin,ST=Texas,C=US',
      userId: new UUID("8599dc7b-8eb6-42c2-82ec-d6babd85c590"),
      user: 'emailAddress=cert@stealthsoft.io,CN=rwadmin,OU=mongo-user.dw.dev.stealthsoft.io,O=StealthMode Inc,L=Austin,ST=Texas,C=US',
      db: '$external',
      roles: [ { role: 'root', db: 'admin' } ],
      mechanisms: [ 'external' ]
    },
    {
      _id: '$external.emailAddress=cert@stealthsoft.io,CN=rwapp,OU=mongo-user.dw.dev.stealthsoft.io,O=StealthMode Inc,L=Austin,ST=Texas,C=US',
      userId: new UUID("f64f84c2-a6d4-441b-b1d8-7d236851c471"),
      user: 'emailAddress=cert@stealthsoft.io,CN=rwapp,OU=mongo-user.dw.dev.stealthsoft.io,O=StealthMode Inc,L=Austin,ST=Texas,C=US',
      db: '$external',
      roles: [ { role: 'readWrite', db: 'rwsecure' } ],
      mechanisms: [ 'external' ]
    }
  ],
  ok: 1,
  '$clusterTime': {
    clusterTime: Timestamp({ t: 1669854524, i: 1 }),
    signature: {
      hash: Binary(Buffer.from("0398d62ed09262dce376cd4aac40801db40a7445", "hex"), 0),
      keyId: Long("7171583554449571845")
    }
  },
  operationTime: Timestamp({ t: 1669854524, i: 1 })
}
rs0 [direct: primary] admin>

EXPECTED RESULTS

It should work like mongosh

# mongosh --tls --host localhost --tlsCertificateKeyFile /etc/mongo-certs/mongo-rwadmin-user-cert-and-key.pem --tlsCAFile /etc/mongo-certs/mongo-ca-cert.pem --authenticationDatabase '$external' --authenticationMechanism MONGODB-X509 admin
Current Mongosh Log ID:	6387f6426814dbd6590f5161
Connecting to:		mongodb://localhost:27017/admin?directConnection=true&serverSelectionTimeoutMS=2000&tls=true&tlsCertificateKeyFile=%2Fetc%2Fmongo-certs%2Fmongo-rwadmin-user-cert-and-key.pem&tlsCAFile=%2Fetc%2Fmongo-certs%2Fmongo-ca-cert.pem&authSource=%24external&authMechanism=MONGODB-X509&appName=mongosh+1.6.0
Using MongoDB:		6.0.3
Using Mongosh:		1.6.0

For mongosh info see: https://docs.mongodb.com/mongodb-shell/

------
   The server generated these startup warnings when booting
   2022-11-30T17:58:03.060+00:00: Using the XFS filesystem is strongly recommended with the WiredTiger storage engine. See http://dochub.mongodb.org/core/prodnotes-filesystem
------

rs0 [direct: primary] admin> db.getSiblingDB("$external").auth({mechanism: "MONGODB-X509", user: "emailAddress=cert@stealthsoft.io,CN=rwadmin,OU=mongo-user.dw.dev.stealthsoft.io,O=StealthMode Inc,L=Austin,ST=Texas,C=US"})
{ ok: 1 }
rs0 [direct: primary] admin>

rs0 [direct: primary] admin> rs.status();
{
  set: 'rs0',
  date: ISODate("2022-12-01T00:33:17.870Z"),
  myState: 1,
  term: Long("30"),
  syncSourceHost: '',
  syncSourceId: -1,
  heartbeatIntervalMillis: Long("2000"),
  majorityVoteCount: 2,
  writeMajorityCount: 2,
  votingMembersCount: 3,
  writableVotingMembersCount: 3,
  optimes: {
    lastCommittedOpTime: { ts: Timestamp({ t: 1669854794, i: 1 }), t: Long("30") },
    lastCommittedWallTime: ISODate("2022-12-01T00:33:14.626Z"),
    readConcernMajorityOpTime: { ts: Timestamp({ t: 1669854794, i: 1 }), t: Long("30") },
    appliedOpTime: { ts: Timestamp({ t: 1669854794, i: 1 }), t: Long("30") },
    durableOpTime: { ts: Timestamp({ t: 1669854794, i: 1 }), t: Long("30") },
    lastAppliedWallTime: ISODate("2022-12-01T00:33:14.626Z"),
    lastDurableWallTime: ISODate("2022-12-01T00:33:14.626Z")
  },
  lastStableRecoveryTimestamp: Timestamp({ t: 1669854784, i: 3 }),
  electionCandidateMetrics: {
    lastElectionReason: 'priorityTakeover',
    lastElectionDate: ISODate("2022-11-30T17:58:24.118Z"),
    electionTerm: Long("30"),
    lastCommittedOpTimeAtElection: { ts: Timestamp({ t: 1669831103, i: 1 }), t: Long("29") },
    lastSeenOpTimeAtElection: { ts: Timestamp({ t: 1669831103, i: 1 }), t: Long("29") },
    numVotesNeeded: 2,
    priorityAtElection: 1,
    electionTimeoutMillis: Long("10000"),
    priorPrimaryMemberId: 1,
    numCatchUpOps: Long("0"),
    newTermStartDate: ISODate("2022-11-30T17:58:24.224Z"),
    wMajorityWriteAvailabilityDate: ISODate("2022-11-30T17:58:25.243Z")
  },
  members: [
    {
      _id: 0,
      name: 'mgo-wj-a.dw.dev.stealthsoft.io:27017',
      health: 1,
      state: 1,
      stateStr: 'PRIMARY',
      uptime: 23714,
      optime: { ts: Timestamp({ t: 1669854794, i: 1 }), t: Long("30") },
      optimeDate: ISODate("2022-12-01T00:33:14.000Z"),
      lastAppliedWallTime: ISODate("2022-12-01T00:33:14.626Z"),
      lastDurableWallTime: ISODate("2022-12-01T00:33:14.626Z"),
      syncSourceHost: '',
      syncSourceId: -1,
      infoMessage: '',
      electionTime: Timestamp({ t: 1669831104, i: 1 }),
      electionDate: ISODate("2022-11-30T17:58:24.000Z"),
      configVersion: 1,
      configTerm: 30,
      self: true,
      lastHeartbeatMessage: ''
    },
    {
      _id: 1,
      name: 'mgo-wj-b.dw.dev.stealthsoft.io:27017',
      health: 1,
      state: 2,
      stateStr: 'SECONDARY',
      uptime: 23713,
      optime: { ts: Timestamp({ t: 1669854794, i: 1 }), t: Long("30") },
      optimeDurable: { ts: Timestamp({ t: 1669854794, i: 1 }), t: Long("30") },
      optimeDate: ISODate("2022-12-01T00:33:14.000Z"),
      optimeDurableDate: ISODate("2022-12-01T00:33:14.000Z"),
      lastAppliedWallTime: ISODate("2022-12-01T00:33:14.626Z"),
      lastDurableWallTime: ISODate("2022-12-01T00:33:14.626Z"),
      lastHeartbeat: ISODate("2022-12-01T00:33:16.058Z"),
      lastHeartbeatRecv: ISODate("2022-12-01T00:33:17.097Z"),
      pingMs: Long("0"),
      lastHeartbeatMessage: '',
      syncSourceHost: 'mgo-wj-a.dw.dev.stealthsoft.io:27017',
      syncSourceId: 0,
      infoMessage: '',
      configVersion: 1,
      configTerm: 30
    },
    {
      _id: 2,
      name: 'mgo-wj-c.dw.dev.stealthsoft.io:27017',
      health: 1,
      state: 2,
      stateStr: 'SECONDARY',
      uptime: 23713,
      optime: { ts: Timestamp({ t: 1669854794, i: 1 }), t: Long("30") },
      optimeDurable: { ts: Timestamp({ t: 1669854794, i: 1 }), t: Long("30") },
      optimeDate: ISODate("2022-12-01T00:33:14.000Z"),
      optimeDurableDate: ISODate("2022-12-01T00:33:14.000Z"),
      lastAppliedWallTime: ISODate("2022-12-01T00:33:14.626Z"),
      lastDurableWallTime: ISODate("2022-12-01T00:33:14.626Z"),
      lastHeartbeat: ISODate("2022-12-01T00:33:16.122Z"),
      lastHeartbeatRecv: ISODate("2022-12-01T00:33:16.116Z"),
      pingMs: Long("1"),
      lastHeartbeatMessage: '',
      syncSourceHost: 'mgo-wj-b.dw.dev.stealthsoft.io:27017',
      syncSourceId: 1,
      infoMessage: '',
      configVersion: 1,
      configTerm: 30
    }
  ],
  ok: 1,
  '$clusterTime': {
    clusterTime: Timestamp({ t: 1669854794, i: 1 }),
    signature: {
      hash: Binary(Buffer.from("a61dcdb51a7939c6f7e972afb4797805c66f5089", "hex"), 0),
      keyId: Long("7171583554449571845")
    }
  },
  operationTime: Timestamp({ t: 1669854794, i: 1 })
}
rs0 [direct: primary] admin>

ACTUAL RESULTS

It fails unless I include these lines in the playbook yaml

        connection_options:
          - "tlsAllowInvalidHostnames=true"

./mongo-status.yml

PLAY [Check replica set status for all replica set nodes are in a healthy state] 

TASK [check mongo replica set state] 
fatal: [mgo-wj-a]: FAILED! => {
  "ansible_facts": {
    "discovered_interpreter_python": "/usr/bin/python3"
  },
  "changed": false,
  "msg": "Unable to determine if auth is enabled: Traceback (most recent call last):\n  File \"/tmp/ansible_community.mongodb.mongodb_status_payload_qsf9hv51/ansible_community.mongodb.mongodb_status_payload.zip/ansible_collections/community/mongodb/plugins/module_utils/mongodb_common.py\", line 279, in is_auth_enabled\n    myclient['admin'].command('listDatabases', 1.0)\n  File \"/usr/local/lib/python3.10/dist-packages/pymongo/_csot.py\", line 105, in csot_wrapper\n    return func(self, *args, **kwargs)\n  File \"/usr/local/lib/python3.10/dist-packages/pymongo/database.py\", line 805, in command\n    with self.__client._socket_for_reads(read_preference, session) as (\n  File \"/usr/local/lib/python3.10/dist-packages/pymongo/mongo_client.py\", line 1296, in _socket_for_reads\n    server = self._select_server(read_preference, session)\n  File \"/usr/local/lib/python3.10/dist-packages/pymongo/mongo_client.py\", line 1257, in _select_server\n    server = topology.select_server(server_selector)\n  File \"/usr/local/lib/python3.10/dist-packages/pymongo/topology.py\", line 272, in select_server\n    server = self._select_server(selector, server_selection_timeout, address)\n  File \"/usr/local/lib/python3.10/dist-packages/pymongo/topology.py\", line 261, in _select_server\n    servers = self.select_servers(selector, server_selection_timeout, address)\n  File \"/usr/local/lib/python3.10/dist-packages/pymongo/topology.py\", line 223, in select_servers\n    server_descriptions = self._select_servers_loop(selector, server_timeout, address)\n  File \"/usr/local/lib/python3.10/dist-packages/pymongo/topology.py\", line 238, in _select_servers_loop\n    raise ServerSelectionTimeoutError(\npymongo.errors.ServerSelectionTimeoutError: not enough values to unpack (expected 2, got 1), Timeout: 30s, Topology Description: <TopologyDescription id: 6387f1fed957c5ccc26eada3, topology_type: Single, servers: [<ServerDescription ('localhost', 27017) server_type: Unknown, rtt: None, error=ValueError('not enough values to unpack (expected 2, got 1)')>]>\n"
}

[rhysmeister]

Hello @davidwartell,

Thank you for the very detailed report. Can you also tell us how you generate your certs? Does it work if you use the hostname, i.e. mgo-wj-b.dw.dev.stealthsoft.io?

@fischerscode Have you experienced any issues around this?

Cheers,

Rhys

[rhysmeister]

Local test environment is broken impeding progress on this. Need to fix the versions of kubernetes (and related tools) to the versions in the CI.

[LeoSpyke]

Even if it is not the best solution, I overcame this by adding localhost into the list of SAN(s) in the csr.

To say it in openssl words the outcome is the following:
openssl req -new -newkey rsa:2048 -nodes -keyout replica.key -out replica.csr -addext "extendedKeyUsage = serverAuth, clientAuth" -addext "subjectAltName = DNS:<node1-fqdn>, DNS:<node2-fqdn>, DNS:<node3-fqdn>, DNS:localhost.

Hope it helps.

[rhysmeister]

This might be an option for a local test env... https://github.com/nektos/act