tips: Port breakout

[wdoekes]

Description

How do we configure port breakout on the community version?

Requirements are listed below.

• User needs to provide port_config.ini for different port breakout mode.
• Specify default speed in port_config.ini file for split ports.
• Use the speed attribute in the minigraph to specify the speed of a port.
• Assume that the user needs to restart the SWSS when he needs to change the port breakout mode.
• Use create_port() and remove_port() SAI APIs, so SAI >= v1.0 is required.
• Assume that there is no port breakout in SAI profile by default.

https://github.com/sonic-net/SONiC/wiki/Port-Breakout-High-Level-Design
(and)
https://github.com/sonic-net/SONiC/wiki/How-to-breakout-a-port

^- this means manual reconfiguration of /etc/sonic/config_db.json

Instead of:
    "Ethernet240": {
        "admin_status": "up",
        "alias": "fourHundredGigE31",
        "index": "31",
        "lanes": "249,250,251,252,253,254,255,256",
        "mtu": "9100",
        "speed": "400000"
    },

We have:
    "Ethernet248": {
        "admin_status": "up",
        "alias": "hundredGigE32",
        "index": "32",
        "lanes": "241,242",
        "mtu": "9100",
        "speed": "100000"
    },
    "Ethernet250": {
        "admin_status": "up",
        "alias": "hundredGigE33",
        "index": "32",
        "lanes": "243,244",
        "mtu": "9100",
        "speed": "100000"
    },
    "Ethernet252": {
        "admin_status": "up",
        "alias": "hundredGigE34",
        "index": "32",
        "lanes": "245,246",
        "mtu": "9100",
        "speed": "100000"
    },
    "Ethernet254": {
        "admin_status": "up",
        "alias": "hundredGigE35",
        "index": "32",
        "lanes": "247,248",
        "mtu": "9100",
        "speed": "100000"
    }

And then restart (everything?).

Ports can be broken out to different speeds with various lanes in most HW today. However, on SONiC, the port breakout modes are hard-coded in the profiles and only loaded at initial time. In case we need to have a new port breakout mode, we would potentially need a new image or at least need to restart services which would impact the traffic of the box on irrelevant ports.

The feature in this document is to address the above issues.

https://github.com/sonic-net/SONiC/blob/master/doc/dynamic-port-breakout/sonic-dynamic-port-breakout-HLD.md

^- this would use the BREAKOUT_CFG config

root@spine1:~# cat change.json 
{
    "BREAKOUT_CFG": {
        "Ethernet32": {
            "brkout_mode": "1x400G[100G](4)"
        }
    }
}
root@spine1:~# config load change.json

Load config from the file(s) change.json ? [y/N]: y

Running command: /usr/local/bin/sonic-cfggen -j change.json --write-to-db
Then there's no change though. (Version SONiC.ossomain.0-41ea968fc.)
And show breakout is broken (both on SONiC.ossomain.0-41ea968fc and SONiC.osso202405.0-49b1c0f39).
root@spine2:~# show interfaces breakout 
Invalid JSON file format('/usr/share/sonic/platform/Accton-AS9716-32D/port_config.ini')
Expecting value: line 1 column 1 (char 0)
Aborted!

Did not get this to work yet on community version ossomain.0-41ea968fc.

Still needs test is if we get interfaces up. Right now all still appear down.

Which build are we running (if any)

Upstream issues/PRs

[wdoekes]

Commands that enable the dynamic port breakout:

root@spine1:~# config interface breakout Ethernet32 '4x100G[50G]'
Do you want to Breakout the port, continue? [y/N]: y

Running Breakout Mode : 1x400G[200G] 
Target Breakout Mode : 4x100G[50G]

Ports to be deleted : 
 {
    "Ethernet32": "400000"
}
Ports to be added : 
 {
    "Ethernet32": "100000",
    "Ethernet34": "100000",
    "Ethernet36": "100000",
    "Ethernet38": "100000"
}
Breakout process got successfully completed.
Please note loaded setting will be lost after system reboot. To preserve setting, run `config save`.

Works once we have the proper:
hwsku.json platform.json (and maybe fixed port_config.ini)
/etc/sonic/config_db.json

    "BREAKOUT_CFG": {
        "Ethernet0": {
            "brkout_mode": "1x400G[200G]"
        },

Check for N/A or down status of the interfaces.

root@spine1:~# show interfaces status
  Interface                            Lanes    Speed    MTU    FEC              Alias    Vlan    Oper    Admin                                             Type    Asym PFC
-----------  -------------------------------  -------  -----  -----  -----------------  ------  ------  -------  -----------------------------------------------  ----------
  Ethernet0          73,74,75,76,77,78,79,80     400G   9100    N/A   fourHundredGigE1  routed      up       up  QSFP-DD Double Density 8X Pluggable Transceiver         N/A
  Ethernet8          65,66,67,68,69,70,71,72     400G   9100    N/A   fourHundredGigE2  routed      up       up  QSFP-DD Double Density 8X Pluggable Transceiver         N/A
 Ethernet16          81,82,83,84,85,86,87,88     400G   9100    N/A   fourHundredGigE3  routed    down       up                                              N/A         N/A
 Ethernet24          89,90,91,92,93,94,95,96     400G   9100    N/A   fourHundredGigE4  routed    down       up                                              N/A         N/A
 Ethernet32                            97,98     100G   9100    N/A         Ethernet32  routed    down     down                                  QSFP28 or later         N/A
 Ethernet34                           99,100     100G   9100    N/A         Ethernet34  routed    down     down                                  QSFP28 or later         N/A
 Ethernet36                          101,102     100G   9100    N/A         Ethernet36  routed    down     down                                  QSFP28 or later         N/A
 Ethernet38                          103,104     100G   9100    N/A         Ethernet38  routed    down     down                                  QSFP28 or later         N/A
 Ethernet40  105,106,107,108,109,110,111,112     400G   9100    N/A   fourHundredGigE6  routed    down       up                                              N/A         N/A
...

root@spine1:~# config interface startup Ethernet32
root@spine1:~# config interface startup Ethernet34
root@spine1:~# config interface startup Ethernet36
root@spine1:~# config interface startup Ethernet38

As long as syncd etc. doesn't crash (because of mismatches in configs) then we can switch these on the fly.

2024 Nov 15 14:50:22.246686 spine1 NOTICE syncd#syncd: :- collectPortRelatedObjects: obtained 33 port oid:0x10000003c related RIDs
2024 Nov 15 14:50:22.246717 spine1 INFO syncd#syncd: [none] SAI_API_PORT:brcm_sai_remove_port:8756 REMOVE_PORT port_id:0x10000003c, bcm_port:60, phy:97
2024 Nov 15 14:50:22.246725 spine1 INFO syncd#syncd: [none] SAI_API_PORT:_brcm_sai_port_autoneg_set:1100 Port 60: autoneg set 0 -> 0
2024 Nov 15 14:50:22.246737 spine1 INFO syncd#syncd: [none] SAI_API_PORT:_brcm_sai_port_autoneg_set:1120 Port 60: autoneg skip off
2024 Nov 15 14:50:22.246888 spine1 INFO syncd#syncd: [none] SAI_API_PORT:_brcm_sai_flex_free_logical_port:16241 FREE bcm_port: 60
2024 Nov 15 14:50:22.284595 spine1 NOTICE syncd#syncd: :- postPortRemove: removed 8 lanes from redis lane map for port RID oid:0x10000003c
2024 Nov 15 14:50:22.284595 spine1 NOTICE syncd#syncd: :- postPortRemove: post port remove actions succeeded
2024 Nov 15 14:50:23.200761 spine1 INFO syncd#syncd: [none] SAI_API_PORT:brcm_sai_create_port:8251 request for phy_lane:97 at speed:100000, lane_count:2
2024 Nov 15 14:50:23.200761 spine1 INFO syncd#syncd: [none] SAI_API_PORT:_brcm_sai_flex_reserve_logical_port:16217 RESERVE bcm_port: 60
2024 Nov 15 14:50:23.280848 spine1 INFO syncd#syncd: [none] SAI_API_PORT:_brcm_sai_port_post_create_config:17278 Current hash offsets: 16, 0
2024 Nov 15 14:50:23.282046 spine1 INFO syncd#syncd: [none] SAI_API_PORT:brcm_sai_create_port:8597 Port OID:0x10000003c created for device port: 60
2024 Nov 15 14:50:23.283928 spine1 NOTICE syncd#syncd: :- discover: discover took 0.001528 sec
2024 Nov 15 14:50:23.283928 spine1 NOTICE syncd#syncd: :- discover: discovered objects count: 36
2024 Nov 15 14:50:23.283928 spine1 NOTICE syncd#syncd: :- discover: SAI_OBJECT_TYPE_PORT: 1
2024 Nov 15 14:50:23.284024 spine1 NOTICE syncd#syncd: :- discover: SAI_OBJECT_TYPE_QUEUE: 12
2024 Nov 15 14:50:23.284050 spine1 NOTICE syncd#syncd: :- discover: SAI_OBJECT_TYPE_SCHEDULER: 1
2024 Nov 15 14:50:23.284070 spine1 NOTICE syncd#syncd: :- discover: SAI_OBJECT_TYPE_SCHEDULER_GROUP: 13
2024 Nov 15 14:50:23.284092 spine1 NOTICE syncd#syncd: :- discover: SAI_OBJECT_TYPE_INGRESS_PRIORITY_GROUP: 8
2024 Nov 15 14:50:23.284117 spine1 NOTICE syncd#syncd: :- discover: SAI_OBJECT_TYPE_PORT_SERDES: 1
2024 Nov 15 14:50:23.284169 spine1 NOTICE syncd#syncd: :- onPostPortCreate: discovered 36 new objects (including port) after creating port VID: oid:0x10000000005b6
2024 Nov 15 14:50:23.284169 spine1 NOTICE syncd#syncd: :- onPostPortCreate: putting ALL new discovered objects to redis for port oid:0x10000000005b6
2024 Nov 15 14:50:23.304581 spine1 NOTICE syncd#syncd: :- redisUpdatePortLaneMap: added 2 lanes to redis lane map for port RID oid:0x10000003c
...

(Tested with flood ping on neighbouring device with 0.137756% packet loss before, during and after the breakout.)

[wdoekes]

Don't forget about ranges:

# config interface startup Ethernet248-254