Skip to content

fix: persist auto-assigned network subnet across system restarts #836

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
wants to merge 1 commit into
base: main
Choose a base branch
from
Open

fix: persist auto-assigned network subnet across system restarts #836

wants to merge 1 commit into from

Conversation

@Ronitsabhaya75
Copy link
Contributor

@Ronitsabhaya75 Ronitsabhaya75 commented Nov 3, 2025

Type of Change

  • Bug fix
  • New feature
  • Breaking change
  • Documentation update

Motivation and Context

Closes #835

In ReservedVmnetNetwork.swift, after retrieving the auto-assigned subnet from macOS, save it to DefaultsStore so it's reused on subsequent restarts.

Code Changes:

// After getting subnet from macOS, persist it
if configuration.subnet == nil {
    DefaultsStore.setOptional(key: .defaultSubnet, value: runningSubnet.description)
}

Testing

  • Tested locally
  • Added/updated tests
  • Added/updated docs

@jglogan
Copy link
Contributor

jglogan commented Nov 14, 2025

@Ronitsabhaya75 Thank your for the contribution, but I don't think we want to merge this.

The point of auto assignment is to let vmnet pick a free network. Consider the following:

  • The user runs container system start and vmnet automatically assigns the subnet with CIDR 192.168.64.1/24 for the default subset.
  • The new code persists this value.
  • The user later runs container system stop.
  • The user starts another app that uses vmnet to create a network, and vmnet assigns 192.168.64.1/24.
  • The user runs container system start and the app fails because it tries got get a subnet that is already assigned.

If a user wants to set a fixed CIDR for the default subnet, they can do so explicitly with container system property set default.subnet cidr-address.

@jglogan
Copy link
Contributor

jglogan commented Nov 14, 2025

The other problem with this as it stands is:

container network create foo

will overwrite the default subnet setting even though it's not the default subnet.

@vsarunas
Copy link

vsarunas commented Nov 18, 2025

@jglogan , unfortunately:

$ container system property set default.subnet 192.168.100.1/24
Error: invalidArgument: "invalid property ID: default.subnet"

There is and a section on docs/technical-overview.md for macOS 15:

network.subnet     String  *undefined*                               Default subnet for IP allocation (used on macOS 15 only).

Default changes with this command works:

container system property set network.subnet 192.168.100.1/24

But if I just run:

container run --rm -ti alpine sh

And then kill the VM; and do:

container system stop
container system start
Verifying apiserver is running...
[ hangs and requires macOS reboot ]

Log show:

container-network-vmnet failed [error=unsupported: "failed to create vmnet network with status vmnet_return_t(rawValue: 1001)"] [id=default]

@jglogan
Copy link
Contributor

jglogan commented Nov 18, 2025

Could you describe what you mean by "kill the VM"? What are you doing to achieve that?

Can you capture the logs by running the following in one terminal and then starting the container service with that setting in another?

log stream --debug --info --predicate 'subsystem = "com.apple.container" or subsystem = "com.apple.NetworkSharing"'

@vsarunas
Copy link

vsarunas commented Nov 18, 2025

Killing in Activity Monitor Virtual Machine Service for container-runtime-linux.
Kill usually needs to be done as the container hangs.

Without corruption:

2025-11-17T13:40:37+0000 debug vminitd : exits=[88: 0] processes=1 [vminitd] checking for exit of managed process
2025-11-17T13:40:37+0000 debug vminitd : count=0 pid=88 status=0 [vminitd] managed process exited
2025-11-17T13:40:37+0000 info vminitd : id=linux status=0 [vminitd] managed process exit
2025-11-17T13:40:37+0000 info vminitd : id=linux [vminitd] closing relay for StandardIO stdout
2025-11-17T13:40:37+0000 info vminitd : id=linux [vminitd] closing relay for StandardIO stderr
2025-11-17T13:40:37+0000 debug vminitd : id=linux [vminitd] 2 managed process waiters signaled
2025-11-17T13:40:37+0000 debug vminitd : pid=-1 signal=9 [vminitd] kill
2025-11-17T13:40:37+0000 debug vminitd : containerID=linux id=linux [vminitd] waitProcess
2025-11-17T13:40:37+0000 debug vminitd : flags=0 path=/run/container/linux/rootfs [vminitd] umount
[ 9649.722052] EXT4-fs (vdb): unmounting filesystem ac94aefe-fe74-44b2-9230-f7016ff18400.
2025-11-17T13:40:37+0000 debug vminitd : containerID=linux id=linux [vminitd] deleteProcess

% container stop linux
< hang >

Filtering the log data using "subsystem ==[cd] "com.apple.container" OR subsystem ==[cd] "com.apple.NetworkSharing""
Timestamp                       Thread     Type        Activity             PID    TTL
2025-11-18 14:59:42.611834+0100 0x45883fa  Info        0x0                  50248  0    container-network-vmnet: [com.apple.container:NetworkVmnetHelper] starting container-network-vmnet [id=nomad]
2025-11-18 14:59:42.612578+0100 0x45883fa  Info        0x0                  50248  0    container-network-vmnet: [com.apple.container:NetworkVmnetHelper] configuring XPC server [id=nomad]
2025-11-18 14:59:42.613392+0100 0x45883fa  Info        0x0                  50248  0    container-network-vmnet: [com.apple.container:NetworkVmnetHelper] creating vmnet network [id=nomad]
2025-11-18 14:59:42.613412+0100 0x45883fa  Info        0x0                  50248  0    container-network-vmnet: [com.apple.container:NetworkVmnetHelper] created vmnet network [id=nomad]
2025-11-18 14:59:42.613577+0100 0x45883fa  Info        0x0                  50248  0    container-network-vmnet: [com.apple.container:NetworkVmnetHelper] starting vmnet network [id=nomad] [mode=nat]
2025-11-18 14:59:42.613663+0100 0x45883fa  Info        0x0                  50248  0    container-network-vmnet: [com.apple.container:NetworkVmnetHelper] configuring vmnet subnet [id=nomad] [cidr=192.168.100.0/24]
2025-11-18 14:59:42.613724+0100 0x45883fa  Info        0x0                  50248  0    container-network-vmnet: (Netrb) [com.apple.NetworkSharing:framework.netrb] connection 0x104dbe3f0 to daemon created
2025-11-18 14:59:42.613832+0100 0x45883fa  Info        0x0                  50248  0    container-network-vmnet: (Netrb) [com.apple.NetworkSharing:framework.netrb] listener connection 0x104dc07f0 created
2025-11-18 14:59:42.613858+0100 0x45883fa  Info        0x0                  50248  0    container-network-vmnet: (Netrb) [com.apple.NetworkSharing:framework.netrb] client 0x104dbd660 xpc send -> client create
2025-11-18 14:59:42.614961+0100 0x45883fa  Info        0x0                  50248  0    container-network-vmnet: (Netrb) [com.apple.NetworkSharing:framework.netrb] client 0x104dbd660 create xpc response received
2025-11-18 14:59:42.614651+0100 0x45875e6  Debug       0x0                  1043   0    InternetSharing: [com.apple.NetworkSharing:daemon] connection 0x854e50000 event from client
2025-11-18 14:59:42.614890+0100 0x45875e6  Debug       0x0                  1043   0    InternetSharing: [com.apple.NetworkSharing:daemon] client created <private> (0x85510c300) (<private>), client connection 0x854e50000 remote connection 0x855050b40
2025-11-18 14:59:42.614922+0100 0x45875e6  Debug       0x0                  1043   0    InternetSharing: [com.apple.NetworkSharing:daemon] sending reply 0x854e50000:
<private>
2025-11-18 14:59:42.615026+0100 0x45875e6  Debug       0x0                  1043   0    InternetSharing: [com.apple.NetworkSharing:daemon] netrbCreateNetwork: set ext if to any external
2025-11-18 14:59:42.615041+0100 0x45875e6  Info        0x0                  1043   0    InternetSharing: [com.apple.NetworkSharing:daemon] mis_network_set_external_interface: ext_if <private> already exists, reusing
2025-11-18 14:59:42.615051+0100 0x45875e6  Info        0x0                  1043   0    InternetSharing: [com.apple.NetworkSharing:daemon] mis_network_set_external_interface: added ext if <private> to network <private>
2025-11-18 14:59:42.615057+0100 0x45875e6  Info        0x0                  1043   0    InternetSharing: [com.apple.NetworkSharing:daemon] netrbCreateNetwork: ext if already created
2025-11-18 14:59:42.615636+0100 0x45875e6  Info        0x0                  1043   0    InternetSharing: [com.apple.NetworkSharing:daemon] mis_network_fill_default_options: generated ULA prefix <private>/64, network <private>
2025-11-18 14:59:42.615679+0100 0x45875e6  Debug       0x0                  1043   0    InternetSharing: [com.apple.NetworkSharing:daemon] mis_network_fill_default_options: set ext if to any external
2025-11-18 14:59:42.615746+0100 0x45883fa  Default     0x0                  50248  0    container-network-vmnet: (Netrb) [com.apple.NetworkSharing:framework.netrb] _NETRBCreateNetwork_block_invoke_2: interface creation failed
2025-11-18 14:59:42.615690+0100 0x45875e6  Info        0x0                  1043   0    InternetSharing: [com.apple.NetworkSharing:daemon] mis_network_validate_resource_availability: overlapping DHCP range between network <private> and network <private>
2025-11-18 14:59:42.615782+0100 0x45883fa  Default     0x0                  50248  0    container-network-vmnet: (vmnet) [com.apple.NetworkSharing:framework.vmnet] vmnet_network_create: _NETRBCreateNetwork
2025-11-18 14:59:42.615698+0100 0x45875e6  Default     0x0                  1043   0    InternetSharing: [com.apple.NetworkSharing:daemon] netrbCreateNetwork: unable to fulfill network
2025-11-18 14:59:42.615708+0100 0x45875e6  Debug       0x0                  1043   0    InternetSharing: [com.apple.NetworkSharing:daemon] sending reply 0x854e50000:
<private>
2025-11-18 14:59:42.616247+0100 0x45883fa  Error       0x0                  50248  0    container-network-vmnet: [com.apple.container:NetworkVmnetHelper] container-network-vmnet failed [error=unsupported: "failed to create vmnet network with status vmnet_return_t(rawValue: 1001)"] [id=nomad]
2025-11-18 14:59:42.617789+0100 0x45875e6  Info        0x0                  1043   0    InternetSharing: [com.apple.NetworkSharing:daemon] received connection error 0x854e50000:Connection invalid
2025-11-18 14:59:42.618907+0100 0x45875e6  Debug       0x0                  1043   0    InternetSharing: [com.apple.NetworkSharing:daemon] found client 0x85510c300 for connection 0x854e50000
2025-11-18 14:59:42.619148+0100 0x45875e6  Debug       0x0                  1043   0    InternetSharing: [com.apple.NetworkSharing:daemon] releasing remote connection 0x855050b40 to client
2025-11-18 14:59:42.619699+0100 0x45875e6  Debug       0x0                  1043   0    InternetSharing: [com.apple.NetworkSharing:daemon] mis_client_release: stopping all networks of <private>
2025-11-18 14:59:42.620053+0100 0x45875e6  Debug       0x0                  1043   0    InternetSharing: [com.apple.NetworkSharing:daemon] detached from <private> (0x85510c300)
2025-11-18 14:59:42.620485+0100 0x45875e6  Info        0x0                  1043   0    InternetSharing: [com.apple.NetworkSharing:daemon] event on remote connection 0x855050b40 from client connection 0x854e50000
2025-11-18 14:59:52.682289+0100 0x4588449  Info        0x0                  50249  0    container-network-vmnet: [com.apple.container:NetworkVmnetHelper] starting container-network-vmnet [id=nomad]
2025-11-18 14:59:52.683004+0100 0x4588449  Info        0x0                  50249  0    container-network-vmnet: [com.apple.container:NetworkVmnetHelper] configuring XPC server [id=nomad]
2025-11-18 14:59:52.683810+0100 0x4588449  Info        0x0                  50249  0    container-network-vmnet: [com.apple.container:NetworkVmnetHelper] creating vmnet network [id=nomad]
2025-11-18 14:59:52.683826+0100 0x4588449  Info        0x0                  50249  0    container-network-vmnet: [com.apple.container:NetworkVmnetHelper] created vmnet network [id=nomad]
2025-11-18 14:59:52.683992+0100 0x4588449  Info        0x0                  50249  0    container-network-vmnet: [com.apple.container:NetworkVmnetHelper] starting vmnet network [id=nomad] [mode=nat]
2025-11-18 14:59:52.684076+0100 0x4588449  Info        0x0                  50249  0    container-network-vmnet: [com.apple.container:NetworkVmnetHelper] configuring vmnet subnet [id=nomad] [cidr=192.168.100.0/24]
2025-11-18 14:59:52.684140+0100 0x4588449  Info        0x0                  50249  0    container-network-vmnet: (Netrb) [com.apple.NetworkSharing:framework.netrb] connection 0x105b4d600 to daemon created
2025-11-18 14:59:52.684257+0100 0x4588449  Info        0x0                  50249  0    container-network-vmnet: (Netrb) [com.apple.NetworkSharing:framework.netrb] listener connection 0x105b4e1b0 created
2025-11-18 14:59:52.684285+0100 0x4588449  Info        0x0                  50249  0    container-network-vmnet: (Netrb) [com.apple.NetworkSharing:framework.netrb] client 0x105b509b0 xpc send -> client create
2025-11-18 14:59:52.684423+0100 0x458821e  Debug       0x0                  1043   0    InternetSharing: [com.apple.NetworkSharing:daemon] connection 0x854e50000 event from client
2025-11-18 14:59:52.684646+0100 0x4588449  Info        0x0                  50249  0    container-network-vmnet: (Netrb) [com.apple.NetworkSharing:framework.netrb] client 0x105b509b0 create xpc response received
2025-11-18 14:59:52.684568+0100 0x458821e  Debug       0x0                  1043   0    InternetSharing: [com.apple.NetworkSharing:daemon] client created <private> (0x85510c300) (<private>), client connection 0x854e50000 remote connection 0x855050c80
2025-11-18 14:59:52.684598+0100 0x458821e  Debug       0x0                  1043   0    InternetSharing: [com.apple.NetworkSharing:daemon] sending reply 0x854e50000:

This machine has:

$ container network list
NETWORK  STATE    SUBNET
nomad    running  192.168.100.0/24
default  running  192.168.64.0/24

$ defaults read com.apple.container.defaults
{
    "dns.domain" = test;
}

On my machine which I tried to set the previous commands to change the default network:

defaults read com.apple.container.defaults
{
    "dns.domain" = test;
    "network.subnet" = "192.168.100.1/24";
}

After I deleted the new default subnet that was just added with defaults delete com.apple.container.defaults network.subnet, was able to start container system again.

container network list
NETWORK  STATE    SUBNET
default  running  192.168.64.0/24

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

[Bug]: container system stop / container system start causes a new default network

3 participants

@Ronitsabhaya75 @jglogan @vsarunas