|
| 1 | +--- |
| 2 | +title: High-Performance Networking |
| 3 | +description: | |
| 4 | + Developing highly specialized and performance-optimized unikernels with Unikraft by bypassing traditional OS layers and interacting directly with network drivers. |
| 5 | +--- |
| 6 | + |
| 7 | +## Overview |
| 8 | + |
| 9 | +In this guide, we explore the development of highly specialized and performance-optimized unikernels. |
| 10 | +Traditionally, network-based applications in Unikraft are built on top of the socket API, which requires a multi-layered library stack: |
| 11 | + |
| 12 | +```text |
| 13 | +.---------------------------. |
| 14 | +( Socket application ) |
| 15 | +'---------------------------' |
| 16 | + | |
| 17 | + V |
| 18 | ++---------------------------+ |
| 19 | +| libvfscore | |
| 20 | ++---------------------------+ |
| 21 | ++---------------------------+ |
| 22 | +| liblwip | |
| 23 | ++---------------------------+ |
| 24 | ++---------------------------+ |
| 25 | +| libuknetdev | |
| 26 | ++---------------------------+ |
| 27 | ++---------------------------+ |
| 28 | +| libkvmplat | |
| 29 | ++---------------------------+ |
| 30 | + | |
| 31 | + V |
| 32 | +.---------------------------. |
| 33 | +( Virtual Network Interface ) |
| 34 | +'---------------------------' |
| 35 | +``` |
| 36 | + |
| 37 | +The Virtual File System (VFS) layer (provided by `libvfscore`) and the TCP/IP network stack (provided by `liblwip`) are complex subsystems that can introduce additional overhead. |
| 38 | +For high-performance Network Functions (NFs), it is often more efficient to bypass these OS components and interact directly with the driver or hardware, similar to frameworks like Intel DPDK. |
| 39 | + |
| 40 | +In this scenario, the library stack is simplified: |
| 41 | + |
| 42 | +```text |
| 43 | +.---------------------------. |
| 44 | +( High performance NF ) |
| 45 | +'---------------------------' |
| 46 | + | |
| 47 | + V |
| 48 | ++---------------------------+ |
| 49 | +| libuknetdev | |
| 50 | ++---------------------------+ |
| 51 | ++---------------------------+ |
| 52 | +| libkvmplat | |
| 53 | ++---------------------------+ |
| 54 | + | |
| 55 | + V |
| 56 | +.---------------------------. |
| 57 | +( Virtual Network Interface ) |
| 58 | +'---------------------------' |
| 59 | +``` |
| 60 | + |
| 61 | +This guide demonstrates the process of developing a high-performance network packet generator using this direct-access model. |
| 62 | + |
| 63 | +## Requirements |
| 64 | + |
| 65 | +To develop and test high-performance networking unikernels, you need the Unikraft `kraft` tool and the following utilities: |
| 66 | + |
| 67 | +- `qemu-kvm` |
| 68 | +- `qemu-system-x86_64` |
| 69 | +- `bridge-utils` |
| 70 | +- `ifupdown` |
| 71 | +- `tshark` |
| 72 | +- `tcpdump` |
| 73 | + |
| 74 | +Installation on Debian/Ubuntu: |
| 75 | + |
| 76 | +```console |
| 77 | +$ sudo apt-get -y install qemu-kvm qemu-system-x86 sgabios socat bridge-utils ifupdown tshark tcpdump |
| 78 | +``` |
| 79 | + |
| 80 | +## Getting Started |
| 81 | + |
| 82 | +The development process typically begins with a template containing basic building blocks, such as utilities for crafting IPv4/UDP packets. |
| 83 | + |
| 84 | +1. Create a copy of the packet generator template: |
| 85 | + |
| 86 | + ```console |
| 87 | + $ cp -a sol/pktgen path/to/your/copy |
| 88 | + $ cd path/to/your/copy |
| 89 | + ``` |
| 90 | + |
| 91 | +1. Initialize and build the application with `kraft`: |
| 92 | + |
| 93 | + ```console |
| 94 | + $ kraft list update |
| 95 | + $ kraft list pull |
| 96 | + $ kraft configure |
| 97 | + $ kraft build |
| 98 | + ``` |
| 99 | + |
| 100 | +1. Verify the resulting image by running it to see the Unikraft banner: |
| 101 | + |
| 102 | + ```console |
| 103 | + $ kraft run |
| 104 | + ``` |
| 105 | + |
| 106 | +## Interacting with the Network Device |
| 107 | + |
| 108 | +Direct interaction with network device drivers is achieved through Unikraft's internal `uknetdev` API. |
| 109 | + |
| 110 | +### Library Dependency |
| 111 | + |
| 112 | +First, ensure the application states a dependency on `libuknetdev` in its `Config.uk` file: |
| 113 | + |
| 114 | +```kconfig |
| 115 | +depends on LIBUKNETDEV |
| 116 | +``` |
| 117 | + |
| 118 | +This provides access to the `<uk/netdev.h>` and `<uk/netbuf.h>` headers. |
| 119 | + |
| 120 | +### Bringing Up the Interface |
| 121 | + |
| 122 | +Bringing up a network interface involves transitioning the device through several configuration states before it can process traffic: |
| 123 | + |
| 124 | +1. **Discovery**: Determine the number of available interfaces using `uk_netdev_count()`. |
| 125 | + |
| 126 | +1. **Retrieval**: Retrieve the `struct uk_netdev *` handle for the desired device (e.g., device `0`). |
| 127 | + |
| 128 | +1. **Configuration**: Specify how many receive (RX) and transmit (TX) queues the device should provide. |
| 129 | + Note that drivers typically require at least one queue in each direction even if only one is used. |
| 130 | + |
| 131 | + ```c |
| 132 | + struct uk_netdev_conf ifconf = { |
| 133 | + .nb_rx_queues = 1, |
| 134 | + .nb_tx_queues = 1 |
| 135 | + }; |
| 136 | + ``` |
| 137 | +
|
| 138 | +1. **Queue Setup**: Configure the RX and TX queues, specifying the size and the allocators for internal descriptors and buffers. |
| 139 | +
|
| 140 | +1. **Probing**: Use `uk_netdev_probe()` to transition the device from an unprobed to an unconfigured state. |
| 141 | +
|
| 142 | +1. **Starting**: Use `uk_netdev_start()` to enable the device for operational traffic. |
| 143 | +
|
| 144 | +### Testing Environment |
| 145 | +
|
| 146 | +To test the unikernel's networking capabilities, a network bridge must be created on the Linux host: |
| 147 | +
|
| 148 | +```console |
| 149 | +# Ensure permissions for STP and create bridge 'usocbr0' |
| 150 | +sudo sysctl -w net.bridge.bridge-nf-call-arptables=0 |
| 151 | +sudo brctl addbr usocbr0 |
| 152 | +sudo brctl setfd usocbr0 0 |
| 153 | +sudo brctl stp usocbr0 off |
| 154 | +sudo ifconfig usocbr0 0.0.0.0 up |
| 155 | +
|
| 156 | +# Disable packet filtering on bridge interfaces |
| 157 | +echo 0 | sudo tee /proc/sys/net/bridge/bridge-nf-call-arptables |
| 158 | +echo 0 | sudo tee /proc/sys/net/bridge/bridge-nf-call-iptables |
| 159 | +echo 0 | sudo tee /proc/sys/net/bridge/bridge-nf-call-ip6tables |
| 160 | +``` |
| 161 | + |
| 162 | +Launch the unikernel attached to the bridge: |
| 163 | + |
| 164 | +```console |
| 165 | +$ kraft run -b usocbr0 |
| 166 | +``` |
| 167 | + |
| 168 | +## Packet Generation and Transmission |
| 169 | + |
| 170 | +### Generating Frames |
| 171 | + |
| 172 | +The provided `genpkt_udp4()` function (or the shorthand `genpkt_usoc21()`) generates Ethernet-IPv4-UDP frames. |
| 173 | +The generation requires specific parameters retrieved from the device via `uk_netdev_info_get()`, such as `ioalign` for buffer alignment and `nb_encap_tx` for headroom (reserved space for driver encapsulation). |
| 174 | + |
| 175 | +For performance testing, minimum-sized packets (60 bytes) are recommended, as they place the most stress on software and hardware parsing components. |
| 176 | + |
| 177 | +### Sending Packets |
| 178 | + |
| 179 | +Transmission is performed using `uk_netdev_tx_one()`. |
| 180 | +It is critical to manage the packet buffers correctly: |
| 181 | + |
| 182 | +- If a packet is successfully enqueued, the driver takes ownership and will free it automatically. |
| 183 | +- If transmission fails, the application must manually call `uk_netbuf_free()` to avoid memory leaks. |
| 184 | + |
| 185 | +```c |
| 186 | +status = uk_netdev_tx_one(netif, 0, pkt); |
| 187 | +if (!uk_netdev_status_successful(status)) { |
| 188 | + uk_netbuf_free(pkt); |
| 189 | +} |
| 190 | +``` |
| 191 | + |
| 192 | +Traffic can be verified on the host using tools like `tshark` or `tcpdump` attached to the bridge: |
| 193 | + |
| 194 | +```console |
| 195 | +$ tshark -i usocbr0 |
| 196 | +``` |
| 197 | + |
| 198 | +## Performance Optimization |
| 199 | + |
| 200 | +### Throttled Instrumentation |
| 201 | + |
| 202 | +Measuring performance requires tracking the total number of packets and bytes sent. |
| 203 | +However, synchronous printing in Unikraft is expensive and can significantly slow down the application. |
| 204 | +It is recommended to throttle statistics printing using a bitmask or counter: |
| 205 | + |
| 206 | +```c |
| 207 | +if ((total_nb_pkts & 0x3fffff) == 0x0) { |
| 208 | + print_netspeed(total_nb_pkts, total_nb_bytes); |
| 209 | +} |
| 210 | +``` |
| 211 | + |
| 212 | +### Optimization Techniques |
| 213 | + |
| 214 | +Several techniques can be applied to increase the packet rate: |
| 215 | + |
| 216 | +1. **Compiler Optimizations**: Enable *Link Time Optimizations* (LTO) and *Dead Code Elimination* (DCE) in the build options via `menuconfig`. |
| 217 | + |
| 218 | +1. **Persistent Packets**: Instead of freeing and regenerating packets on transmission failure, retry sending until the TX queue has space. |
| 219 | + |
| 220 | +1. **Packet Duplication**: Use `uk_netbuf_dup_single()` to `memcpy` from a primordial packet buffer, which can be cheaper than full header generation. |
| 221 | + |
| 222 | +1. **Memory Pools**: Use `libukallocpool` for $O(1)$ allocation and deallocation. |
| 223 | + This replaces the general-purpose allocator with a pre-allocated list of objects. |
| 224 | + |
| 225 | + ```c |
| 226 | + /* Add 'depends on LIBUKALLOCPOOL' to Config.uk */ |
| 227 | + pool = uk_allocpool_alloc(uk_alloc_get_default(), 1024, 2048, info.ioalign); |
| 228 | + struct uk_alloc *p = uk_allocpool2ukalloc(pool); |
| 229 | + ``` |
| 230 | + |
| 231 | +1. **Zero-copy**: While virtio-net has current limitations, some drivers support increasing the `netbuf` reference counter to avoid freeing and reallocating the same packet for repeated transmission. |
| 232 | + |
| 233 | +1. **Batching**: Sending multiple packets in a single batch reduces the overhead of notifying the device backend (functionality introduced in PR#243). |
| 234 | + |
| 235 | +## Receiving Traffic |
| 236 | + |
| 237 | +Implementing a receiver follows a similar pattern to the transmitter but requires a proper receive buffer allocation function (`alloc_rxpkts`). |
| 238 | +This callback is used by the driver to replenish the RX queue with empty buffers. |
| 239 | + |
| 240 | +```c |
| 241 | +uint16_t alloc_rxpkts(void *argp, struct uk_netbuf *pkts[], uint16_t count) { |
| 242 | + uint16_t i; |
| 243 | + for (i=0; i<count; ++i) { |
| 244 | + pkts[i] = uk_netbuf_alloc_buf(p, 2048, info.ioalign, info.nb_encap_rx, 0, NULL); |
| 245 | + if (!pkts[i]) break; |
| 246 | + } |
| 247 | + return i; |
| 248 | +} |
| 249 | +``` |
| 250 | + |
| 251 | +The receive loop then polls the device: |
| 252 | + |
| 253 | +```c |
| 254 | +status = uk_netdev_rx_one(netdev, 0, &pkt); |
| 255 | +if (uk_netdev_status_successful(status)) { |
| 256 | + // Process counters and free the received packet |
| 257 | + uk_netbuf_free(pkt); |
| 258 | +} |
| 259 | +``` |
0 commit comments