This project is done for MEng in Electronics and Computer Engineering. The purpose of it is to investigate the possibilities to improve speed and processing network packets operating over high-speed modern network interfaces by monitoring the performance of packet processing in the Linux kernel and optimizing it.
Netfilter based solutions (iptables/nftables) became a de-facto standard firewall for Linux. From other hand, introduction of new high-speed ethernet standards introduce new performance requirements for packet processing that iptables/nftables are not always able to satisfy. In particular, for large number of filter rules. In parallel, several eBPF/XDP based approaches for input packet filtering have been emerged. In this research I investigate ability for XDP based approaches to provide comparable or even better performance than conventional netfilter solutions, especially for the case with big (up to 10K) number of stateless filtering rules. Two different solutions are considered: one is represented by pcn-iptables[2] and uses Linear Bit Vector Search (LBVS) algorithm underneath, while second adopts DPDK ACL packet classification method[5] based on multi-bit tries.
The repository consists of four directories; results, rules, scripts and xdp-acl.
The results directory consists of the results gained during the testing. There are two main test-cases; positive and negative.
- Positive test-cases: involves simply using wrk[4]/nginx[3] to stress the network stack.
- Negative test-cases: involves running wrk along with replaying pcap, to generate "malicious" traffic to simulate a DDoS attack on the server.
In addition, there is a run0 directory, which consists of results for the test involving no rules.
Each test-case of the two main directories consists of similar test-cases:
- XDP_ACL_(mod_for_pcn): consists of tests performed for rules using the XDP ACL program, using rules modified for pcn-iptables.
- XDP_ACL_(orig_rules): consists of tests performed for rules using the XDP ACL program, but with the original, non-modified rules.
- nftables_(mod_for_pcn): consists of tests performed for rules using Netfilter, using the rules modified for pcn-iptables.
- nftables_(orig_rules): consists of tests performed for rules using Netfilter, but with the original, non-modified rules.
- pcn-iptables: consists of tests performed for modified rules loaded into pcn-iptables within a virtual machine (VM), which acts as server.
Each directory in the list has 5 sets of results, with the exception of pcn-iptables, which has only 4 sets of results. Each set is based on the respective set of rules used for testing. The 'orig' in the names mean that the original rule-sets were used, while 'mod_for_pcn' means using modified rules.
test-case result structure:
- run1: 10 rules.
- run2: 91 rules.
- run3: 455 rules.
- run4: 8676 rules.
- run5: 5000 rules.
Each run directory consists of multiple files, containing the data observed during testing.
Here is the structure:
- wrk.outX: this file contains the output of wrk instance, in particular the number of queries per second. X is one of the four instances or wrk.
- wrkstat.out: contains the sum of all wrk instance outputs.
- mpstat0.out: contains CPU statistics collected during testing.
- irqstat.out: Information on the irqs assigned to CPUs for server and client. It is collected twice: before and after the testing.
Sometimes, there are additional files, depending on which of the 5 tests is performed. For XDP ACL, file bpfstat.out contains information on the BPF maps used.
The rules directory consists of the rule-sets used for testing. There are two categories of rules:
- Original: these are sets of synthetic rules generated by Classbench[1].
- Modified: these rules have been modified for the pcn-iptables, since pcn-iptables doesn't support rules with a range of source/destination L4 ports (i. e. only one port pair per rule). As the number of rules increases, the speed of adding new rule drops dramatically – in order to add around 10000 rules, it took around 22 hours for the VM to do so and then it hanged Due to this, it was required to limit pcn-iptables testing only with 5000 rules max.
For each category, there are 5 rule sets; 10, 91, 455, 5000 and 8676 rules. In addition, each rule set has its own trace file, a sequence of packet headers to perform functional testing and generate pcap files.
These trace files are used in the XDP ACL testing, both with original and modified. Modified rules are named X_mod_rule, where X is either 1,2,3,5k or 10k.
The scripts directory consists of Bash and python scripts that are used to perform tests on the network performance for various test cases such as using Netfilter (iptables/nftables), pcn-iptables (in a virtual machine), and XDP ACL.
Each test consists of three Bash scripts:
- The setup script that accesses the rule sets.
- The irq script is used to assign the irqs to the CPU cores that are used for server and client. Though, when testing with pcn-iptables, the irq script on host machine will do the same, but only for client, the server assignment is written in a separate script located on VM.
- The test script then performs the test itself, using nginx and wrk to simulate requests from 4 instances of wrk, to simulate requests from multiple clients.
Python scripts are used to read the rules from the files and express them in the required format for the tables.
As stated in the results section, there are 2 main test-cases:
- Positive test-cases: involves simply using wrk/nginx to stress the network stack.
- Negative test-cases: involves running wrk along with replaying pcap, to generate "malicious" traffic to simulate a DDoS attack on the server.
Positive test-case scripts are test_ip.sh, test_xdp.sh, test_pcn.sh, test_xdp_mod.sh, test_ip_mod.sh, with the first two are used for Netfilter and XDP ACL with original rules, while the other three use the modified rules. Scripts test_irq.sh and test_prf.sh assign irqs to the cores and perform the test itself respectively for Netfilter and XDP ACL. For pcn, there are separate scripts; tpcn_irq.sh and tpcn_pf.sh that access the virtual machine.
Scripts irqaft.sh, netflt.sh, nginx_kill.sh, nginx_start.sh, nginx_vm_setup.sh, perfcp.sh, perf_get.sh, servirq.sh, and xdp.sh are scripts used for testing with pcn-iptables, in which a virtual machine acts as the server. This scripts must be stored within the virtual machine and executed using ssh command. Scripts testrX.sh and pX_xdp.py are used to load rules into pcn-iptables and thus must be located on VM as well.
For negative test-cases, the first script for each test-case summons a pcap file required for the given rule-set. It is then used in the third script. Scripts for negative test-case end with "_pc.sh"; txdp_mod_pc.sh (XDP ACL for modified rules), txp_pc.sh, tip_mod_pc.sh, tip_pc.sh, with the scripts for irq assignment and testing are tpc_irq.sh and ts_pcap.sh. For pcn-iptables, the scripts are tpcn_pc.sh and tpcn_pcap.sh, the script for irq assignment is the same.
The xdp-acl directory contains sources and patch for XDP-ACL POC. It also contains a get_build.sh script which is used to clone DPDK, applying the patch in xdp-acl/patch and build it. It then compiles the XDP program that will be used for XDP ACL packet filtering.
Trace files have been generated using the Trace Generator tool from Classbench [1].
classbench-ng/trace_generator/trace_generator <num_packets> <start_id> <end_id> <input_file>
Where:
<num_packets>: The total number of packets to generate.
<start_id>: The starting identifier for the trace.
<end_id>: The ending identifier for the trace.
<input_file>: The path to the input file containing the ACL rules.
A generated trace file's name usually ends with a "_trace".
The script mk_pcap_u5.py, located in the scripts directory, is used to generate a pcap file. It works the following:
python3 mk_pcap_in_u5.py <path-to-trace-file> <path-to-pcap-file> <src-ether-addr> <dst-ether-addr> <size-of-packet>
For example;
python3 ../scripts/mk_pcap_in_u5.py ./rules/acl1v4_10k_trace ./rules/acl1v4_10k_trace_u1.pcap > 10:70:fd:30:43:c9 94:6d:ae:2e:79:f3 64
To send malicious traffic using the pcap file, use the following command:
ip netns exec ns2 time python3 send_pcap_u6.py ${PCAP} enp88s0 100000 20 > /dev/null 2>1 &
The send_pcap_u6.py script is also located in the scripts directory. PCAP is the pcap file. With this example, the script, using scapy[6], will send 100000 packets 20 times from pcap through device enp88s0.
Here is an example to run a test against XDP ACL:
/bin/bash -x test_xdp_mod.sh 5 5k
Where test_xdp_mod.sh is the script for XDP ACL with modified rules. The 5 parameter is the number of run directory where the collected data for this test will be stored, while 5k is the rule-set. In this example, the rule-set is the 5000 rules.
For the other test scripts, the command structure is the same, except for tests involving the Virtual Machine.
/bin/bash -x test_pcn.sh C xdp 3
Here, the xdp and 3 parameters mean that the rules from the third rule-set will be loaded into pcn-iptables within the VM. The collected data will be stored in directory runC.
[1] https://github.com/classbench-ng/classbench-ng
[2] https://polycube-network.readthedocs.io/en/latest/components/iptables/pcn-iptables.html
[5] https://doc.dpdk.org/guides/prog_guide/packet_classif_access_ctrl.html
[6] https://scapy.readthedocs.io/en/latest/introduction.html