Add Iperf3 Check to OpenWISP Monitoring : GSoC’22 Project Report
This summer has been one of the most productive above all I’ve ever had. For the past 3 months, I have been working on my GSoC project with OpenWISP under the guidance of my amazing project mentors, Federico Capaono (nemesisdesign), Gagan Deep (pandafy), and Oliver Kraitschy (okraits).
Let’s deep dive into what we have achieved this summer ☀
What is OpenWISP?
OpenWISP is a modular network management system built on top of OpenWRT (but designed to allow supporting multiple embedded operating systems) that allows managing and automating several aspects of IT network deployment, monitoring and management.
Before GSoC
I came to know about OpenWISP back in 2020. Many seniors and classmates are already part of this community. I started contributing to OpenWISP in mid-January this year. While contributing to OpenWISP, I also got the opportunity to participate in their release sprint, which further helped me to learn more about the community and the work they’ve been doing in the past. During that time, I made my own custom kanban board to manage my assigned issues, which are prioritised for the next release.
Overall, before GSoC, my aim was to touch every module of OpenWISP to learn how they’re interconnected to each other.
After Selection
After the results were announced, we were invited to attend a community meeting with the mentors, in which we introduced ourselves and I was asked to create all the necessary issues for my project according to the plan we mentioned in the project proposal.
In the beginning, I was a little confused about setting up my board. After a long discussion with my mentors, I concluded that:
“If the plan doesn’t work, change the plan, not the goal.”
The OpenWISP Iperf3 Check Kanban Board has more than 25+ cards, including issues and PR’s, in the done column after the program ends.
About Project
What is OpenWISP monitoring?
It is a network monitoring system written in Python and Django, designed to be extensible, programmable, scalable, and easy to use by end users. Once the system is configured, monitoring checks, alerts, and metric collection happen automatically. — docs
The goal of my project was to “Add an Iperf3 bandwidth monitoring check to OpenWISP monitoring” using its active check mechanism.
Currently OpenWISP monitoring has two active checks.
- Ping : This check uses an ICMP echo request to determine if a target host (OpenWRT device) is reachable or not and returns information about the device like uptime and RTT (round trip time) using the
fping utitily. - Configuration Applied: This check ensures that the OpenWISP config agent is running and applying configuration changes in a timely manner.
Similar to the two above checks, the Iperf3 check goal is to perform periodic network performance measurements such as maximum achievable bandwidth, jitter, datagram loss etc. of the OpenWRT device using the iperf3 utility.
My experience with the OpenWISP controller
What is OpenWISP controller?
Network and WiFi controller: provisioning, configuration management and updates, (pull via openwisp-config or push via SSH), x509 PKI management and more. Mainly OpenWRT, but designed to work also on other systems. — docs
Even before GSoC, I spent most of my time reading OpenWISP controller code. The check that I’m going to build uses OpenWRT SSH, which is already implemented in the openwisp-controller (connection module). Therefore, it is necessary for me to first understand how the controller provides this functionality. After spending a few more days on it, I was finally able to use the connection module to execute commands to the devices via SSH.
Starting off with an Iperf3 check class
After learning about how devices interact with the openwisp-controller, I implemented an Iperf3 check class which aims to execute the Iperf3 command with the --json flag on OpenWRT devices first in TCP mode and then in UDP mode.
After the execution of commands, it parses network performance results and stores them in a time series database. It also creates monitoring charts for the visualisation of Iperf3 data and alert settings for managing metric-related notifications.
Iperf3 Metric, Charts & Alert settings
Finally, after successful execution of the check and after parsing data, I get the following network performance results from the device:
{
"iperf3_result": 1,
"jitter": 0.01911334779014794,
"lost_packets": 0,
"lost_percent": 0.0,
"received_bps_tcp": 1584852657.4337416,
"received_bytes_tcp": 991821198,
"retransmits": 0,
"sent_bps_tcp": 1585856899.594464,
"sent_bps_udp": 29996024.07315705,
"sent_bytes_tcp": 991952896,
"sent_bytes_udp": 18747860,
"total_packets": 12841
}Here, iperf3_result denotes that the check has been successfully executed in both modes (i.e., TCP & UDP) without any failure.
Now this data is being stored inside a timeseries database (Influxdb) and used by openwisp-monitoring charts for visualization.
Like other existing checks, Iperf3 Check also provides the ability to send alerts to notify users. In the metric configuration, I’ve added the alert_field key so that users can easily reconfigure their system to send alerts for any metric field they want, or they can directly change field_name through the OpenWISP device page as shown below.
Here is the example that shows how to reconfigure the system for Iperf3 check to send an alert if the measured TCP bandwidth has been less than 10 Mbps for more than 2 days.
Pull requests 🏷
Set up celery beat for Iperf3 check
In contrast to the ping and configuration checks, which are performed on the device every 5 minutes, the Iperf3 check’s objective is to gather information on network performance, and the majority of users do not wish to do it regularly.
Therefore, in order to prevent interfering with regular traffic, we advise users to schedule this check to occur during times when there is low traffic.
Here is an illustration of how the Iperf3 check was set to run at midnight with a certain interval.
Pull request 🏷
Iperf3 authentication support
This is a very interesting feature that I have worked on during my GSoC.
By default, Iperf3 Check doesn’t support any kind of authentication, but with this enhancement, users can configure their system to use RSA authentication between the client (OpenWRT device) and the Iperf3 server to restrict connections to the server only to authenticated clients.
After mentioning all the required steps to build this feature in this issue, I got approval from the mentors to work on this. The one thing which I have been told to keep in mind is that after executing an Iperf3 check on OpenWRT devices, we should not leave any traces like public RSA keys etc. inside the device.
Pull request 🏷
Iperf3 charts enhancements
- As we already know, Iperf3 Check does not run very frequently, so the Iperf3 Check monitoring charts will give us blank spaces between different data points that look weird and make the purpose of data visualisation somewhat useless. To avoid this, I have added the
connect_pointsproperty to openwisp-monitoring charts, which allows us to connect those individual data points together.
2. The second improvement is related to leveraging other contributors’ work, which is limited to showing adaptive data size only for monitoring traffic charts. So I have done some research and came up with the findings mentioned in the issue.
I improved the previous adaptive feature to work with any chart with just a simple tweak in the monitoring charts configuration.
# Traffic is measured in 'B' (bytes).
# Allow adaptive_prefix to determine the unit (K/M/G/T)bytes automatically.'traffic': {
'unit': 'adaptive_prefix+B',
},# Bandwidth is measured in 'bps'(bits/sec).
# Allow adaptive_prefix to determine the unit (K/M/G/T)bps automatically.'bandwidth_tcp': {
'unit': 'adaptive_prefix+bps',
},
Pull request 🏷
Support for configuring more Iperf3 parameters
When we talk about the Iperf3 utility, we all know it has so many parameters which almost cover all network performance measurements. We want to do the same with Iperf3 Check. Currently, Iperf3 Check supports 15 parameters, including RSA authentication. It can be configured easily with simple settings as shown below.
Pull request 🏷
Support for multiple Iperf3 servers
“One Iperf server is not enough for a big system.”
The primary goal of this feature is to enable the execution of multiple Iperf3 checks on different devices at the same time. If the current available server is busy running a check, we should keep other checks back in queue until the previous check finishes.
For this, I have to implement a cache lock which ensures that no two devices will use the same Iperf3 server at the same time. It is very easy to configure. You just need to add the Iperf3 server in the host parameter of the Iperf3 check configuration.
OPENWISP_MONITORING_IPERF3_CHECK_CONFIG = {
# public iperf servers are also available
# https://iperf.fr/iperf-servers.php#public-servers
'a9734710-db30-46b0-a2fc-01f01046fe4f': {
'host': ['iperf.openwisp.io', '2001:db8::1', '192.168.5.2'],
'client_options': {
'port': 5209,
'udp': {'bitrate': '20M'},
'tcp': {'bitrate': '0'},
},
}
}
Pull request 🏷
Make the Iperf3 check resilient to failures
Failure can happen at any time. Robust software should handle it gracefully. But most failures occur when Iperf3 Check runs in an unsupported environment, so to make Iperf3 Check run properly on their system, users should take care of some points mentioned below:
- Make sure your OpenWRT device has the iperf3 package installed. If you need authentication, use iperf3-ssl instead.
- Ensure SSH access from OpenWISP is enabled on your devices with the right update strategy, i.e. OpenWRT SSH.
- Finally, the Iperf3 check configuration should be as specified in the documentation.
If all the above mentioned points are followed correctly by users, then there will be a very low chance that your device logs an error related to Iperf3 check.
Pull requests 🏷
Add “Checks” and “Alert settings” from the device page
This feature reduces a lot of stress for users who want to configure checks and alert settings directly from the device tab of OpenWISP. Non-superusers are currently unable to add checks and alert settings directly from the device tab, but with this enhancement, they will be able to do so after obtaining some special permissions known as “Inline permissions” for the respective tabs.
Pull request 🏷
Experience
This was one of the best summers for me, a summer packed with learning, developing, and networking. It was a pleasure to work under the guidance of my mentors, Federico Capaono (nemesisdesign), Gagan Deep (pandafy), and Oliver Kraitschy (okraits). I would like to extend my humble gratitude to my mentors, without whom my project wouldn’t be possible.
This time, OpenWISP provided access to their own demo instance for GSoC through which we were able to test our code in a real-world scenario. This gives us an idea of how our work actually performs in the real world. I believe this is the best way to test our work because, through this, we are able to find bugs early on, and it will lead us to develop more robust and secure code at the end of the program.
Through contributing to OpenWISP, I learned new technologies like Ansible, OpenWRT, and Docker and programming practices like TDD.
I believe that contributing to open source is the best way to learn and grow as a developer. Contributing to open source software like OpenWISP feels like being a part of software that brings value to people’s lives ✨
What’s Next?
Now that I have a pretty good understanding of the OpenWISP codebase, I’ll keep contributing to OpenWISP and would love to maintain my project.
