Repurposing now somewhat older system, a Supermicro X11SSM-F motherboard in a 2U chassis with Xeon E3-1285 v6 and 64GB ECC.
- Old role: VM box for local infrastructure analytics
- New role: BSD router w/ CNAs for RoCE-v2 RDMA fabric.
Installed for the purpose: QLogic QL41234HLCU, and...
- NIC #1, QLogic QL41164HLRJ, 4x 10GbE BASE-T,
- NIC #2, Supermicro AOC-S40G-I2Q, 2x 40GbE QSFP+, Intel XL710
- NIC for Nodes: Qlogic QL41234HLCU, 4x 10/25GbE SFP28 w/ RDMA
Great, nice upgrades. The router will have its 4x 10GbE QL41164 ports connected in 802.3ad to the switch-fabric, with two VM host nodes direct-connected using SFP28 ports on the QL41234 NIC. These systems run lightweight validation suites for Project Coherent Flash (the full cluster lives in the colo) and therefore require RDMA and RoCE networking.
In other connectivity areas, the XL710 based NIC is for non-RDMA traffic to the nodes via direct non-switched connection. It does not participate in the memory fabric. In this mode, each QSFP+ 40GbE port has a quad-lane breakout cable going to 4x 10GbE SFP+ connectors, and each of those (8 total) offers 2x 10GbE to each of the four nodes in an adjacent cluster.
Just remember, on this board there's a jumper to inspect and possibly flip, for reasons better described via the following nerdery.
the X11SSM-F bridges the slot SMBus (SMCLK/SMDAT on PCIe pins B5/B6) to the board SMBus via jumpers JI2C1/2. Some PCIe cards expose an I²C device on those lines at the same addresses used by DIMM SPD EEPROMs (0x50–0x57). That creates address conflicts, which can block SPD reads, hide a DIMM, scramble IPMI sensors, or hang POST. Disabling JI2C1/2 isolates the slot SMBus and removes the conflict.
