TRex – A Stateful/Stateless Traffic Generator
When the time comes to stress test new core switches, experiment with NGFW policy, drag race different access points, or simply light up a lab network to look more realistic, you’ll need something flexible and scalable to generate those flows. Unsurprisingly, when it comes to something network related, Cisco has an answer. On this front, the answer is an open-source, stateful and stateless traffic generator powered by DPDK called TRex.
As you’ll find on the official TRex website this powerful tool generates L4-7 traffic based on pre-processing and smart replay of real traffic templates. TRex amplifies both client and server side traffic and can scale up to 200Gb/sec with one UCS. TRex stateless functionality includes support for multiple streams, the ability to change any packet field and provides per stream statistics, latency and jitter. The advanced stateful functionality includes support for emulating L7 traffic with fully-featured scalable TCP layer. The stateless functionality of TRex even has a nice GUI.
In this post I’ll cover the initial configuration of TRex server, setup using a DUT (device under test), and provide an overview of the GUI.
TRex Server Setup
Part of the elegance of TRex is the deployment flexibility. It was designed for use with Cisco UCS servers, although it can run on a massive array of platforms from well provisioned VMs to low-end laptops. I’m a massive fan of UCS and just so happen to have a few suitable servers in my lab to experiment with, so I decided to piece together a purpose built TRex system. My system is a UCS C420 M3 decked out with (2) Cisco VIC 1225, (2) Intel X520-DA2, and (4) onboard 1G Intel interfaces. That’s a total of (8) 10G and (4) 1G interfaces at my disposal…should prove sufficient for a small lab! NICs aside, the server also has (4) Intel Xeon E5-4640 CPUs (8C/16T @ 2.4GHz) along with 768GB of RAM because…why not?
All that said, TRex is a linux application that can run on any x86 server using most popular nix variants. Main thing to watch out for here is NIC & transceiver compatibility, make sure to check the install guide. A 64bit variant of CentOS or RHEL is the current (July 2019) recommendation for compatibility with ConnectX-4 cards, so I’ve opted to go with the Minimal flavor of CentOS 7.4 because I feel as though I unjustly favor Ubuntu with nearly all my other projects.
To get started, let’s install a few necessary libs:
yum install kernel-devel-`uname -r` yum group install "Development tools" yum install pciutils yum install wget
This will provide our minimal install with the necessary tools to obtain and operate TRex. From here, use pciutils to ensure your NICs are detected:
[admin@trex ~]$ lspci | grep Ethernet ... output omitted ... 41:00.0 Ethernet controller: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection (rev 01) 41:00.1 Ethernet controller: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection (rev 01) 43:00.0 Ethernet controller: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection (rev 01) 43:00.1 Ethernet controller: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection (rev 01)
Make a note of your interface labeling, you’ll find the first two digits indicate PCI location (bus) and the trailing digit indicates the interface number on that card. Once you’ve confirmed your interfaces are operational, download and unpack TRex:
mkdir -p /opt/trex cd /opt/trex wget --no-cache http://trex-tgn.cisco.com/trex/release/latest tar -xzvf latest
You’ll notice we put this in the /opt/ folder to assure we have correct permissions set for use later with the TRex Stateless GUI. Next up we’ll perform loopback testing. For this I’ve plugged in a 10G DAC to both ports of a single NIC. If you’re like me and you have more NICs than you know what to do with, use “ip addr” to find the interfaces that are showing “state UP.” At this point, navigate into the TRex version folder you previously unpacked and execute the setup ports python script with the -s arg to view available interfaces for use with TRex:
cd v2.57/ ./dpdk_setup_ports.py -s
[admin@trex v2.57]$ sudo ./dpdk_setup_ports.py -s Network devices using DPDK-compatible driver ============================================ <none> Network devices using kernel driver =================================== ...output omitted... 0000:41:00.0 '82599ES 10-Gigabit SFI/SFP+ Network Connection' drv=igb_uio unused=ixgbe,vfio-pci,uio_pci_generic 0000:41:00.1 '82599ES 10-Gigabit SFI/SFP+ Network Connection' drv=igb_uio unused=ixgbe,vfio-pci,uio_pci_generic 0000:43:00.0 '82599ES 10-Gigabit SFI/SFP+ Network Connection' drv=igb_uio unused=ixgbe,vfio-pci,uio_pci_generic 0000:43:00.1 '82599ES 10-Gigabit SFI/SFP+ Network Connection' drv=igb_uio unused=ixgbe,vfio-pci,uio_pci_generic Other network devices ===================== <none>
Using this detail (or the output fro “lspci”) we can create a basic loopback config to assure all TRex services are operational. Update the TRex config, located at “/etc/trex_cfg.yml”, with (1) interface details and (2) port IP addressing.
[admin@trex v2.57]$ sudo vi /etc/trex_cfg.yaml
- port_limit : 2
version : 2
#List of interfaces. Change to suit your setup. Use ./dpdk_setup_ports.py -s to see available options
interfaces : ["43:00.0", "43:00.1"]
port_info : # Port IPs. Change to suit your needs. In case of loopback, you can leave as is.
- ip : 1.1.1.1
default_gw : 2.2.2.2
- ip : 2.2.2.2
default_gw : 1.1.1.1
A quick note here for those running Cisco UCS with VICs, you’ll need to assure you’re running the correct firmware and have a few settings tweaked within CIMC or UCSM to assure compatibility. Follow this guide and increase the Input/Output/Completion queues accordingly.
Now let’s run a test with the loopback config. To do so, we’re going to leverage one of the TRex provided yaml files that will simulate a few hundred clients attempting to perform DNS queries against a few hundred servers using the pcap file “cap2/dns.pcap”. There are also a few args here you should familiarize yourself with:
-f <file> // specifies source YAML file
-c <cores> // # of cores available to TRex for this test
-m <factor> // Multiplier AKA bandwidth factor - multiply the CPS (connections per second) by this number
-d <seconds> // duration in seconds
[admin@trex v2.57]$ sudo ./t-rex-64 -f cap2/dns.yaml -c 1 -m 1 -d 10 The ports are bound/configured. Starting TRex v2.57 please wait ... set driver name net_ixgbe driver capability : TCP_UDP_OFFLOAD TSO set dpdk queues mode to DROP_QUE_FILTER Number of ports found: 2 zmq publisher at: tcp://*:4500 wait 1 sec .
You’ll then be presented with a snapshot of port & global statistics as TRex runs:
-Per port stats table
ports | 0 | 1
-----------------------------------------------------------------------------------------
opackets | 5 | 5
obytes | 385 | 465
ipackets | 5 | 5
ibytes | 465 | 385
ierrors | 0 | 0
oerrors | 0 | 0
Tx Bw | 571.78 bps | 690.59 bps
-Global stats enabled
Cpu Utilization : 0.1 % 0.0 Gb/core
Platform_factor : 1.0
Total-Tx : 1.26 Kbps
Total-Rx : 1.26 Kbps
Total-PPS : 1.86 pps
Total-CPS : 0.93 cps
Expected-PPS : 2.00 pps
Expected-CPS : 1.00 cps
Expected-BPS : 1.36 Kbps
Active-flows : 0 Clients : 511 Socket-util : 0.0000 %
Open-flows : 5 Servers : 255 Socket : 5 Socket/Clients : 0.0
drop-rate : 0.00 bps
current time : 8.5 sec
test duration : 1.5 sec
Once the test completes, summary output will be provided:
summary stats -------------- Total-pkt-drop : 0 pkts Total-tx-bytes : 1530 bytes Total-tx-sw-bytes : 0 bytes Total-rx-bytes : 1530 byte Total-tx-pkt : 18 pkts Total-rx-pkt : 18 pkts Total-sw-tx-pkt : 0 pkts Total-sw-err : 0 pkts Total ARP sent : 4 pkts Total ARP received : 4 pkts
Assuming there were no dropped packets, you’re ready to move on to more interesting usage involving a DUT (device under test). The goal here is to understand the configuration necessary to pass TRex traffic through a network. I’m going to use a Catalyst 3850 for this step and will be referencing this guide from the TRex team. Using that guide, here is the topology we’ll configure:
One of the key things to note here is that you want an artificial subnet of clients and servers contained within your TRex server that will then traverse your network via an emulated router in TRex. To accomplish this config, we’ll need to set up the following yaml config within our TRex:
- port_limit : 2
version : 2
#List of interfaces. Change to suit your setup. Use ./dpdk_setup_ports.py -s to see available options
interfaces : ["43:00.0", "43:00.1"]
port_info : # Port IPs. Change to suit your needs. In case of loopback, you can leave as is.
- ip : 11.11.11.2
default_gw : 11.11.11.1
- ip : 12.12.12.2
default_gw : 12.12.12.1
generator :
distribution : "seq"
clients_start : "16.0.0.1"
clients_end : "16.0.0.255"
servers_start : "48.0.0.1"
servers_end : "48.0.0.240"
With that config set, we’ll set up the following on our C3850:
interface TenGigabitEthernet1/1/1 no switchport ip address 11.11.11.1 255.255.255.0 ! interface TenGigabitEthernet1/1/4 no switchport ip address 12.12.12.1 255.255.255.0 ! ip route 16.0.0.0 255.0.0.0 11.11.11.2 ip route 48.0.0.0 255.0.0.0 12.12.12.2
Easy enough, now let’s test the interface using the latency test argument:
-l <rate> // Runs latency test in parallel with test at rate/sec./t-rex-64 -f cap2/dns.yaml -m 1 -d 10 -l 1000
You’ll see the stats pop up followed by another summary screen, including the latency info this time around.
-Per port stats table
ports | 0 | 1
-----------------------------------------------------------------------------------------
opackets | 10115 | 10115
obytes | 667689 | 667833
ipackets | 10115 | 10115
ibytes | 667833 | 667689
ierrors | 0 | 0
oerrors | 0 | 0
Tx Bw | 527.67 Kbps | 527.80 Kbps
-Global stats enabled
Cpu Utilization : 0.2 % 1.4 Gb/core
Platform_factor : 1.0
Total-Tx : 1.06 Mbps
Total-Rx : 1.06 Mbps
Total-PPS : 2.00 Kpps
Total-CPS : 0.99 cps
Expected-PPS : 2.00 pps
Expected-CPS : 1.00 cps
Expected-BPS : 1.36 Kbps
Active-flows : 0 Clients : 511 Socket-util : 0.0000 %
Open-flows : 9 Servers : 255 Socket : 9 Socket/Clients : 0.0
drop-rate : 0.00 bps
current time : 12.5 sec
test duration : 0.0 sec
-Latency stats enabled
Cpu Utilization : 0.2 %
if| tx_ok , rx_ok , rx check ,error, latency (usec) , Jitter max window
| , , , , average , max , (usec)
----------------------------------------------------------------------------------------------------------------
0 | 10107, 10107, 0, 0, 10 , 44, 1 | 44 39 40 34 39 39 20 21 21 21 21 19 19
1 | 10107, 10107, 0, 0, 10 , 52, 1 | 52 42 45 42 42 44 19 21 19 21 20 13 19
summary stats -------------- Total-pkt-drop : 0 pkts Total-tx-bytes : 1354926 bytes Total-tx-sw-bytes : 1353396 bytes Total-rx-bytes : 1354926 byte Total-tx-pkt : 20524 pkts Total-rx-pkt : 20524 pkts Total-sw-tx-pkt : 20506 pkts Total-sw-err : 0 pkts Total ARP sent : 4 pkts Total ARP received : 4 pkts maximum-latency : 52 usec average-latency : 10 usec latency-any-error : OK
On the switch side, you’ll see the interfaces flap and will then find the ARP entries from TREX added to your ARP table (verified with “show arp”). With this basic config, you can now see how easy it would be to add static routes around at strategic points in your network to simulate some pretty impressive traffic.
Within the /cap2 folder, you’ll find a wealth of pre-made YAML configs and PCAPs to use with your testing or as starting points for your own flows. Some notable ones include: IMIX, HTTP browsing, HTTPS, NAT tests, Video Calls, and a bunch more!
Awesome stuff, right? It gets better, there’s a handy GUI available for stateless flows. The GUI is super simple to set up, first off you’ll need to start up your TRex server in “interactive” mode which essentially just starts up it and leaves it running for the stateless GUI.
-i // Runs TRex in interactive mode (for use with stateless GUI)[admin@trex v2.57]$ sudo ./t-rex-64 -i
-Per port stats table
ports | 0 | 1
-----------------------------------------------------------------------------------------
opackets | 0 | 0
obytes | 0 | 0
ipackets | 0 | 0
ibytes | 0 | 0
ierrors | 0 | 0
oerrors | 0 | 0
Tx Bw | 0.00 bps | 0.00 bps
-Global stats enabled
Cpu Utilization : 0.0 %
Platform_factor : 1.0
Total-Tx : 0.00 bps
Total-Rx : 0.00 bps
Total-PPS : 0.00 pps
Total-CPS : 0.00 cps
Expected-PPS : 0.00 pps
Expected-CPS : 0.00 cps
Expected-BPS : 0.00 bps
Active-flows : 0 Clients : 0 Socket-util : 0.0000 %
Open-flows : 0 Servers : 0 Socket : 0 Socket/Clients : -nan
drop-rate : 0.00 bps
current time : 11.0 sec
test duration : 0.0 sec
You’ll see TRex happily start up and wait for your command. From here, download the TRex Stateless GUI from the github release page. Launch the application and use the connect button to establish a session with your TRex server. Once connected, you’ll see a summary of your server and your configured ports on the left. Keep in mind, the ports shown here must be configured in your /etc/trex_cfg.yaml file.
From here, you’ll want to click on a specific port and press “aquire” to grant edit access to your GUI. You can navigate into the port details and see that your IP addressing has made it over. This can be a really handy way to tweak configs and do a quick test of ARP resolution.
Before we can start making things blink again, we must configure a port profile. For this activity you can either head into the “profile” sub menu of a port, or use the top level menu to the master “Traffic Profiles” menu. The TRex stateless GUI offers a phenomenal range of configuration options, down to the level of letting you hand craft the contents of packets in a wireshark-esque format.
Within the Traffic Profiles Menu, let’s “Create Profile” and then “Build Stream.” Now you can use simple mode to go through some easy menus to build the stream, or advanced to build using the packet editor.
When building a profile, you can also import pcaps. Try out the ones included in the TRex repo (note – the yaml profiles are not supported, as they are stateful and the GUI supports only stateless). Here you can see me import the DNS pcap and override with my desired source and destination ranges:
Once you have a traffic profile created, navigate into the profile sub section of an interface and select it from the drop down. From there, you can modify the amount of traffic you want sent across in either %L1, L1 bps, L2 bps, or pps. Here I am specifying the desire to run my traffic profile @ 2Gbps L2 bps which will only impact my continuous flow, as the single bursts will obviously only fire once. You can see the little bandwidth slider increases to 1/5th of my available 10G link capacity.
Time to get blinky, from here you can press the play button to initiate traffic generation. You’ll see the port LED indicator light up in the app, at which point you can open up the “Stats -> Dashboard” menu to monitor the action.
If you enabled “RX Stats” during the defintion of your stream, you’ll see graphs populated under “Charts”
You can also confirm the flow by checking out the interface counters on your DUT:
trex-sw#show interfaces counters Port InOctets InUcastPkts InMcastPkts InBcastPkts Te1/1/1 2211781824 34559094 0 0 Te1/1/2 0 0 0 0 Port OutOctets OutUcastPkts OutMcastPkts OutBcastPkts Te1/1/1 459 0 1 0 Te1/1/2 3297411904 51522066 0 0
You will also find your TRex server CLI provides you a summary of interface stats as well:
-Per port stats table
ports | 0 | 1
-----------------------------------------------------------------------------------------
opackets | 1605738738 | 0
obytes | 102767279282 | 0
ipackets | 3 | 1605738868
ibytes | 225 | 102767287889
ierrors | 0 | 0
oerrors | 0 | 0
Tx Bw | 1.76 Gbps | 0.00 bps
-Global stats enabled
Cpu Utilization : 74.2 % 4.7 Gb/core
Platform_factor : 1.0
Total-Tx : 1.76 Gbps
Total-Rx : 1.76 Gbps
Total-PPS : 3.43 Mpps
Total-CPS : 0.00 cps
Expected-PPS : 0.00 pps
Expected-CPS : 0.00 cps
Expected-BPS : 0.00 bps
Active-flows : 0 Clients : 0 Socket-util : 0.0000 %
Open-flows : 0 Servers : 0 Socket : 0 Socket/Clients : -nan
Total_queue_full : 266449762
drop-rate : 0.00 bps
current time : 3753.3 sec
test duration : 0.0 sec
With this basic setup, you can likely see the endless possibilities!
Go forth and push some packets!