OSFP port designation
Eight OSFP ports sit on the GPU tray; two slot ports sit on the motherboard. The default names below are what DGX OS shows; the optional column is the InfiniBand naming when a port is switched.
| Port | PCI bus | Default | RDMA | Optional | RDMA (opt.) |
|---|---|---|---|---|---|
| OSFP1 P1 left | ed:00.0 | eno13np0 / enp237s0f0np0 | mlx5_22 | ibo13 / ibp237s0 | mlx5_15 |
| OSFP1 P2 right | ed:00.1 | enp237s0f1np1 | mlx5_23 | ||
| OSFP2 P1 left | 97:00.0 | eno10np0 / enp151s0f0np0 | mlx5_14 | ibo10 / ibp151s0 | mlx5_10 |
| OSFP2 P2 right | 97:00.1 | enp151s0f1np1 | mlx5_15 | ||
| OSFP3 P1 left | dc:00.0 | eno12np0 / enp220s0f0np0 | mlx5_20 | ibo12 / ibp220s0 | mlx5_14 |
| OSFP3 P2 right | dc:00.1 | enp220s0f1np1 | mlx5_21 | ||
| OSFP4 P1 left | b9:00.0 | eno11np0 / enp185s0f0np0 | mlx5_16 | ibo11 / ibp185s0 | mlx5_11 |
| OSFP4 P2 right | b9:00.1 | enp185s0f1np1 | mlx5_17 | ||
| OSFP5 P1 left | 17:00.0 | eno6np0 / enp23s0f0np0 | mlx5_0 | ibo6 / ibp23s0 | mlx5_0 |
| OSFP5 P2 right | 17:00.1 | enp23s0f1np1 | mlx5_1 | ||
| OSFP6 P1 left | 70:00.0 | eno9np0 / enp112s0f0np0 | mlx5_12 | ibo9 / ibp112s0 | mlx5_9 |
| OSFP6 P2 right | 70:00.1 | enp112s0f1np1 | mlx5_13 | ||
| OSFP7 P1 left | 39:00.0 | eno7np0 / enp57s0f0np0 | mlx5_6 | ibo7 / ibp57s0 | mlx5_5 |
| OSFP7 P2 right | 39:00.1 | enp57s0f1np1 | mlx5_7 | ||
| OSFP8 P1 left | 5f:00.0 | eno8np0 / enp95s0f0np0 | mlx5_10 | ibo8 / ibp95s0 | mlx5_8 |
| OSFP8 P2 right | 5f:00.1 | enp95s0f1np1 | mlx5_11 | ||
| Slot 1 port 1 | cc:00.0 | ibp204s0f0 | mlx5_18 | enp204s0f0np0 | mlx5_12 |
| Slot 1 port 2 | cc:00.1 | enp204s0f1np1 | mlx5_19 | ibp204s0f1 | mlx5_13 |
| Slot 2 port 1 | 53:00.0 | ibp83s0f0 | mlx5_8 | enp83s0f0np0 | mlx5_6 |
| Slot 2 port 2 | 53:00.1 | enp83s0f1np1 | mlx5_9 | ibp83s0f1 | mlx5_7 |
| On-board | 2a:00.0 | eno1 | n/a | n/a |
Cables & adapters are not shipped with the node. Supported cables are set by the ConnectX-8 firmware; NVIDIA's adapter firmware release pages list validated cables & switches per firmware. BlueField-3 DPUs can be placed in NIC mode per NVIDIA's BlueField documentation.
Power button & LEDs
| Power button | Green flashing 1 Hz: standby, BMC booted · green flashing 4 Hz: POST in progress · green solid: powered on |
|---|---|
| ID / UID button | Blue LED at the front & a flashing LED at the rear for servicing; configurable from the BMC |
| Fault LED | Amber: system or component fault |
| BMC reset | Motherboard button; press to reset the BMC by hand |
| BMC port LEDs | Left: solid green 100M, solid amber 1G · right: green flashing on activity |
PSU rules
- Twelve PSUs, N+N. Minimum six for redundancy.
- Boots with three PSUs operational; fewer & only the BMC is available.
- Replace a failed PSU immediately; never run depopulated.
- AC: 3.3 kW per supply at 200 to 240 V, 16 A, 50 to 60 Hz, 15 kW system max. DC: 54 V, 300 A max, 15 kW system max.
- PDU side: push to insert, squeeze the clips to remove. PSU side twist-lock: unlock before push or pull.