Server networking decoded — switches, NICs and transceivers

The three layers, in order

• The switch sets the fabric speed and the feature set (VLANs, routing, redundancy). Its ports are almost always SFP or QSFP cages, not fixed connectors, so the port speed and the transceiver you plug in must agree.
• The NIC sets the server's ceiling. A 25GbE card cannot exceed 25GbE no matter how fast the switch is, and its PCIe slot must have the bandwidth to feed it.
• The transceiver or cable sets the reach and the running cost. The same 100GbE port can run a R-priced copper cable across a rack or a long-reach optic across a campus — the electronics are identical; only the module changes.

Switch speeds: 1GbE to 400GbE

Every Ethernet speed is defined by an amendment to the IEEE 802.3 standard. The speed determines the lane structure — how many electrical or optical lanes carry the traffic — which in turn dictates the connector and the cabling.

Speed	IEEE standard	Lane structure	Typical medium & reach	Where it fits
1GbE	802.3ab (1000BASE-T)	1 lane	Copper Cat5e, up to 100 m	Management, out-of-band, legacy access
10GbE	802.3ae (fibre); 802.3an (10GBASE-T copper)	1 × 10G	SFP+ fibre/DAC; Cat6a to 100 m	General server access, SMB, storage (iSCSI/NAS)
25GbE	802.3by (2016)	1 × 25G	SFP28 fibre/DAC	Modern top-of-rack server access
40GbE	802.3ba (2010)	4 × 10G	QSFP+	Legacy uplinks — superseded by 100G
100GbE	802.3ba (2010) and later single-lane amendments	4 × 25G	QSFP28	Spine/aggregation, storage back-ends, AI/GPU
200GbE	802.3bs (2017)	4 × 50G	QSFP56	High-density spine, AI
400GbE	802.3bs (2017); 802.3cu (FR4/LR4-6)	8 × 50G	QSFP-DD / OSFP	AI clusters, hyperscale spine

When each speed fits, in practice. Modern data-centre switches are multi-rate, so a single box usually spans several of these speeds. Three representative platforms show the pattern:

• The MikroTik CRS504-4XQ-IN carries four 100G QSFP28 ports, each of which can either run at 100G or break out with a breakout cable into four 25G links — up to sixteen 25GbE devices from one small switch. It is the affordable way into 100G/25G for a single rack.
• The Dell PowerSwitch S5248F-ON is a classic leaf/top-of-rack design: 48 × 25GbE SFP28 ports for servers plus 100GbE uplink ports (multi-rate 10/25/40/50/100GbE), on a 2.0 Tbps (4.0 Tbps full-duplex) non-blocking fabric. Servers land at 25G; the switch talks to the spine at 100G.
• The Aruba CX 8325 spans 1/10/25GbE (SFP/SFP+/SFP28) and 40/100GbE (QSFP+/QSFP28) in a 1U chassis, serving both as a campus core/aggregation switch and a data-centre top-of-rack.

The recurring shape is 25G to the server, 100G between switches — with 10G still the sensible floor for budget access and 40G best treated as legacy (see the companion 10G/25G/100G comparison).

Managed, smart or unmanaged?

The management tier is independent of speed. In short:

Tier	Configuration	Typical features	Best for
Unmanaged	None — plug and play	Basic switching only	Home office, very small networks
Smart / web-managed	Browser UI	Basic VLANs, port statistics, limited QoS	Small businesses needing some control on a budget
Managed	Full CLI + web UI, remote management	VLANs, port security and authentication, configurable QoS, L2/L3 routing, monitoring	Larger networks and anything business-critical

For any environment running servers, virtualisation or storage you want a managed switch — VLAN segmentation, link aggregation and routing are not optional at that point. This is a decode-level summary; the management tier deserves its own guide.

Network interface cards (NICs)

The NIC terminates the network inside the server. Beyond raw speed, modern server NICs add hardware offloads (checksum, segmentation, RDMA, RoCE) that take load off the CPU — which is why the controller family matters as much as the port count. Three families dominate enterprise servers: Intel (Ethernet Controller E810 series), Broadcom (NetXtreme E-Series) and NVIDIA/Mellanox (ConnectX).

NIC (controller)	Max Ethernet	Port speeds	Host interface	Connectors
Intel E810 (e.g. E810-CAM2)	Up to 100GbE	2×100 / 2×50 / 4×25 / 8×10GbE	PCIe 4.0	QSFP28 / SFP28
Broadcom NetXtreme P2100G	2 × 100GbE	2×100 / 50 / 25 / 10GbE	PCIe 4.0 ×16	QSFP56 / QSFP28
NVIDIA ConnectX-6 Dx	2×100 or 1×200GbE	25 / 50 / 100GbE (200 single)	PCIe 4.0	SFP28 / SFP56 / QSFP56
NVIDIA ConnectX-7	Up to 400GbE	10 / 25 / 40 / 50 / 100 / 200 / 400GbE	PCIe 4.0 / 5.0 ×16	OSFP / QSFP112 / SFP56

PCIe headroom is a real constraint

A NIC cannot deliver more than its PCIe slot allows. The vendor datasheets pair their 100GbE cards with PCIe 4.0 ×16 for a reason: a dual-port 100GbE adapter needs far more host bandwidth than an older PCIe 3.0 ×8 slot can supply, and NVIDIA's ConnectX-7 moves to PCIe 5.0 to feed its 400GbE ports. Before fitting a fast NIC, confirm the target slot's PCIe generation and lane width — a 100GbE card in a starved slot silently runs slow.

NRZ versus PAM4

How bits are encoded on each lane changes with speed. NRZ (two levels per symbol) carries 10G and 25G lanes; PAM4 (four levels per symbol) doubles that to 50G and 100G per lane. This is why a 100GbE QSFP28 module uses four 25G NRZ lanes, while a 400GbE QSFP-DD uses eight 50G PAM4 lanes, and a 200GbE QSFP56 uses four 50G PAM4 lanes. PAM4 buys density but is more sensitive to noise, which is why forward error correction (FEC) becomes mandatory at the higher rates.

OCP NIC 3.0

Alongside the traditional PCIe add-in card, modern servers increasingly use the OCP NIC 3.0 mezzanine form factor from the Open Compute Project. The common Small Form Factor (SFF) card is 76 × 115 mm and supports up to a ×16 PCIe edge; the Large Form Factor (LFF) is 139 × 115 mm and supports up to ×32 lanes. In practice the LFF is now uncommon: the OCP NIC 3.0 working group has deprecated it, steering new designs to the SFF and the newer Dual Small Form Factor (DSFF), so you are unlikely to find — or need — an LFF card. OCP 3.0 cards are slim and hot-swappable, serviced from the front or rear of the chassis without opening the lid — which is why NVIDIA's ConnectX-6 Dx and ConnectX-7, and Intel and Broadcom equivalents, all ship in OCP 3.0 variants alongside standard PCIe cards. When specifying a server, check whether the board expects an OCP 3.0 slot, a PCIe slot, or both.

Transceivers and cables

The module or cable is where reach and running cost are decided. First, the form factor — the physical cage — which is tied to the maximum speed:

Module	Max speed	Electrical lanes	Signalling	Pairs with
SFP+	10G	1 × 10G	NRZ	10GbE
SFP28	25G	1 × 25G	NRZ	25GbE
QSFP+	40G	4 × 10G	NRZ	40GbE (or 4×10G breakout)
QSFP28	100G	4 × 25G	NRZ	100GbE (or 4×25G breakout)
QSFP56	200G	4 × 50G	PAM4	200GbE
QSFP-DD	400G	8 × 50G	PAM4	400GbE (or 4×100 / 8×50 breakout)

The Q in QSFP stands for quad — four (or, for QSFP-DD, eight) lanes in one cage — which is what makes breakout possible: a 100G QSFP28 port can fan out to four 25G SFP28 links.

DAC, AOC or optical transceiver?

For a given speed you then choose how to carry it. There are three options, in ascending order of cost, power and reach:

Type	What it is	Reach	Cost & power	Use it for
DAC (direct-attach copper)	A fixed twinax cable with the module moulded on each end	~0.5–5 m (passive); ~7–10 m (active)	Lowest cost, near-zero power	In-rack: server to top-of-rack switch
AOC (active optical cable)	A fixed fibre cable with integrated optics on each end	~1–30 m	Middle	Adjacent racks, top-of-rack to end-of-row
Optical transceiver + fibre	A removable module (SR/LR/ER…) over separate structured fibre	100 m to 80 km	Highest per port	Between rows, across a building, campus links

DACs are the workhorse inside a rack — cheapest, coolest-running and reliable over short runs. Optics cost more and draw more power, but are the only option once you leave the rack.

Reach by speed and medium

Once you commit to optics, the module suffix (SR, LR, ER…) sets the distance. These are the reaches from the manufacturer datasheets:

Speed / cage	Multimode (OM3 / OM4)	Single-mode	Copper / DAC / AOC
10G SFP+	SR: 300 m / 400 m; LRM: 220 m	LR: 10 km; ER: 40 km; ZR: ~80 km	DAC 1–5 m; AOC 1–10 m; 10GBASE-T RJ45: 100 m Cat6a
25G SFP28	SR: 70 m / 100 m; CSR: 300 m / 400 m	LR: 10 km; ER: 40 km	DAC 1–5 m (FEC rules apply); AOC 1–10 m
40G QSFP+	SR4: 100 m / 150 m; CSR4: 300 m / 400 m	LR4: 10 km; ER4: 40 km	DAC 1–5 m; AOC available
100G QSFP28	SR4: 70 m / 100 m; BiDi: 100 m (OM4)	DR/PSM4: 500 m; FR/CWDM4: 2 km; LR: 10 km; ER4L: up to 40 km	DAC 1–5 m; AOC 1–30 m
400G QSFP-DD	SR8: 100 m (OM4)	DR4: 500 m; FR4: 2 km; LR4-6: 6 km (IEEE 802.3cu); LR4 10 km (MSA/vendor)	DAC 0.5–3 m; AOC 1–30 m
Copper RJ45	—	—	1000BASE-T: 100 m Cat5e; 10GBASE-T: 100 m Cat6a

Multimode (SR-type) optics are cheaper and fine inside a building; single-mode (LR/ER) optics reach kilometres and are needed for campus or metro links. Note that SR reaches depend on the fibre grade — OM4 always beats OM3.

A note on FEC

At 25G and above, forward error correction is often not optional. On 25G copper DACs, for example, Cisco's datasheet notes that 1–2 m cables need no FEC, 2.5–3 m need BASE-R FEC, and 4–5 m need RS-FEC. The switch and NIC must agree on the FEC mode or the link will not come up cleanly — a very common cause of a 25G link that flaps or refuses to establish.

Choosing for a South African deployment (refurb + new)

• 10G and 40G gear is abundant and inexpensive on the refurbished market, which makes 10G an excellent budget floor for server access and 40G tempting for uplinks. But 40GbE is a technological dead-end (four 10G lanes with no single-25G-lane path), so prefer 100G QSFP28 for new uplinks even when refurbished 40G looks cheaper.
• 25G is the modern server-access default and 100G the modern uplink — the two are natural partners because a 100G QSFP28 port breaks out to four 25G SFP28 links.
• Match the transceiver coding to the switch. Some vendors' switches accept only coded/compatible optics; a module that works in one brand may be rejected in another. Confirm compatibility before buying, especially with mixed new-and-refurbished kit.
• Keep DACs in the rack and optics between racks. A passive DAC is the cheapest, coolest 10/25/100G link you can buy for a server-to-switch run; reserve optics for distances a DAC cannot reach.
• Check PCIe and slot type on the server (PCIe generation, lane width, and whether it wants an OCP 3.0 card) before ordering the NIC, so the card is not throttled or physically incompatible.

Spec the switch, the NIC and the transceiver as one system and the network will run at the speed you paid for.

Switches, NICs and transceivers, priced in rand: