Blog

The cost was not where the request assumed it was

The ask: speed up TCP receive in NXP's gigabit Ethernet example by copying less payload. Before any code was written, the evidence showed there was no payload copy to remove — and LOCI showed where the time and energy per packet actually went.

Aurora Labs TeamAurora Labs4 min readUse Case, Embedded, Networking

NXP i.MX RT1170 · lwIP TCP receive path · pre-run execution prediction

The ask: speed up TCP receive in NXP's gigabit Ethernet example (MCUXpresso SDK 2.16, bare-metal lwIP) by handling pbufs better and copying less payload. Before any code was written, the evidence showed there was no payload copy to remove, and LOCI showed where the time and energy per packet actually went.

1 · No copy to remove

The port is already zero-copy on receive. ethernetif_rx_frame_to_pbufs wraps each DMA buffer as a custom pbuf (pbuf_alloced_custom(…, PBUF_REF, …)), and ethernet_input, ip4_input, tcp_input and tcp_receive move headers, never payload. The iperf receiver copies only its small settings header, once per connection, then counts bytes and frees. A copy-reduction feature would have had nothing to work on.

2 · Where the time went (LOCI preflight)

Receive function, per packetHot pathEnergy
tcp_input≥1,440 ns≥223 nWs
tcp_receive≥1,272 ns≥197 nWs
ENET_QOS_GetRxFrame≥255 ns≥39 nWs
pbuf_free≥252 ns≥39 nWs
ethernetif_linkinput173 ns27 nWs
pbuf_alloced_custom128 ns20 nWs

The two TCP functions dominate, and LOCI's callee list and disassembly showed why part of it is avoidable: they contain 16 call sites to lwip_htons (13) and lwip_htonl (3), separate functions around one or two instructions.

3 · The change: four lines in the port header

lwIP lets a port supply its own byte-order swaps. This port did not, so def.c's functions were used. Added to port/arch/cc.h (GCC section):

#if !defined(lwip_htons)
#define lwip_htons(x) ((u16_t)__builtin_bswap16((u16_t)(x)))
#define lwip_htonl(x) ((u32_t)__builtin_bswap32((u32_t)(x)))
#endif

No pbuf, memory or protocol change. IAR and Keil builds are untouched.

4 · Same results on every input

The original def.c functions and the new macros, extracted unchanged from each commit, were compared on all 65,536 16-bit and all 4,294,967,296 32-bit inputs, built so that no comparison could be optimised away: 0 differences, repeated across 10 runs.

5 · LOCI verdicts, as recorded

No runAdjust planOK
Explore: copy premise checked against the source.Preflight: the swap calls in tcp_input / tcp_receive.Post-edit: both functions lower, calls gone.

6 · Result

ChangeWhat moved
−11.6%tcp_input ≥1.44 → ≥1.27 µs, energy ≥223 → ≥197 nWs (LOCI post-edit)
−4.5%tcp_receive ≥1.27 → ≥1.22 µs, energy ≥197 → ≥188 nWs (LOCI post-edit)
−2.3%whole 10-function receive chain on the linked firmware, ≥3,588 → ≥3,505 ns (LOCI)
36 → 0byte-swap calls on the RX/TX path; code −860 B (LOCI, linker)

The gain per packet is modest and stated as such. The larger value is the redirect: the work went to the cost that exists, and the copy-reduction effort that would have found nothing was never started.

The agent read the request and the source. LOCI showed where the time went, before any code ran.

How these numbers were produced. No board was run and no throughput was measured. Times and energies are LOCI predictions from the compiled code, at the board's 392.73 MHz clock; ≥ marks a minimum where a loop count could not be derived, on both sides. The post-edit percentages compare each build's typical path, which LOCI selected separately because the edit restructured both functions. On the full firmware, with paths also picked per build, tcp_receive read +7 ns while the receive chain as a whole fell 83 ns; both readings are kept. Builds: lwIP 2d63c88 (before) and 2a60f14 (after).