Fixing your first bug in Neutron: 101

by Elvira

This post is a really basic introduction on how to start developing at Neutron. The goal is to provide an idea of what the workflow is and which tools you need when fixing a bug in the project. Hopefully, I can help anyone interested in making a first contribution to the project! Please use this together with Neutron’s official Contributor’s Guide.

This blog post assumes you are familiar with git and python coding.

Intro to Neutron

Openstack is cloud software that has multiple use cases and is widely used all over the world. It is a set of multiple components, and not every component is needed for every deployment (See this page for some examples).

There are some components that must be present regardless of the use case. Neutron, also known as the Openstack Networking Service, is one of them. It manages all of the virtual network infrastructure, both in the control and the data plane of Openstack. The codebase is big, so it can get a bit complex at first, but there is a community of contributors that are very active on IRC, where you can ask any questions. Note that most developers are usually busy people and getting answers might not be something immediate, but everyone will do their best to help new devs to understand Neutron better. You can also go to the weekly neutron team meeting on IRC.

Check the Contributing section of Neutron documentation for up to date information!

Neutron’s Launchpad is the tool to track bugs from the project. If you have already spotted a problem in the code, don’t hesitate to report it and fix it, but if you don’t know where to start and it’s your first time in the project, search for bugs tagged as “low-hanging-fruit”.

If you need some help with network basics, Neutron Documentation also has an informative section on networking concepts that are widely used within the project.

Deploying an environment using Devstack

Once you have chosen your first bug, it’s time to replicate it. Because of the complexity of most Openstack deployments, we usually start reproducing and checking the bugs using Devstack. Devstack is a development version of the Openstack environment, and it should not be used as a production environment. It is based on the latest versions of every component, and it’s also used as the base for functional tests. It is very important to run this environment in a VM since it will make important changes in the system during the installation.

The configuration file that you will need to tune, depending on the needs of your testing, is local.conf. I use the configuration file that comes with neutron for OVN, since that’s the kind of environment that I usually want to replicate, but this part is really customizable depending on what you need. If the bug you want to debug is related to OVN I recommend compiling OVN from source instead of downloading it as a package. That way you will get the latest master version and not the last stable packaged one. You can check the local.conf I used while writing this blog post in case it helps.

This guide for deploying Devstack + OVN from the documentation is the one that has guided me during the process of installation of Devstack.

Once you finish running ./stack.sh and have your VM with Devstack installed, you are ready to start using the env.

Some tips regarding this environment:

If stack.sh finishes successfully, you should be able to see this:

This is your host IP address: 172.16.189.6
This is your host IPv6 address: ::1
Horizon is now available at http://172.16.189.6/dashboard
Keystone is serving at http://172.16.189.6/identity/
The default users are: admin and demo
The password: password
2021-03-09 11:02:12.001 | stack.sh completed in 1998 seconds.

That means your deployment was successfully deployed!

Adding some workload to the deployment

By workload, I’m referring to networks, routers, VM instances, security groups… Any actual component that you need to replicate the bug you are working to solve. There are some cases (i.e. working with Network Agents) where no additional objects need to be spawned, but it is always good to know how to spawn some basic resources.

Although there is a Dashboard available (Horizon), not every Neutron feature is supported there. It is important to learn how to use the Openstack CLI from the terminal so as to get a real understanding of how it works.

First of all, check that you have images and flavors already there. You will need them to spawn VM instances:

$ openstack flavor list
$ openstack image list

If you need to add an image, bear in mind that it needs to be a cloud version. For developing purposes we will use Cirros. I usually download it with:

$ curl -k -L [image_url] > [image_name.img]
$ openstack image create [image_name.img] cirros

The flavor I create for cirros is 1 GB of disk, and 128 of RAM.

$ flavor create m1.small --disk 1 --vcpus 1 --ram 64

After that, I create a basic network topology:

$ openstack network create net1
$ openstack subnet create --subnet-range 192.168.100.0/24 --network net1 subnet1
$ openstack router create r1
$ openstack router add subnet r1 subnet1
$ openstack router set --external-gateway public r1 #public is the external network
$ openstack security group create secgroup1
$ openstack security group rule create --protocol tcp --dst-port 22 secgroup1
$ openstack security group rule create --protocol icmp secgroup1
$ openstack server create --nic net-id=net1 --flavor m1.small --image cirros \
--security-group secgroup1 server1

The result will be a topology like this: schema

The security group associated to the VM allows TCP traffic on port 22 and also icmp traffic on the server. By default, the servers are completely isolated from traffic. To access the VM you will need to associate a floating ip to the VM.

First, we create the floating IP

$ openstack floating ip create nova

We look for the floating IP ID

$ openstack floating ip list

And for the ID of the port associated to the server

$ nova interface-list server1

Finally, we associate the port to the floating IP

$ openstack floating ip set --port 5f798f30-d78b-4da2-95a4-7785aeeb6016 \
c617ec29-3b64-45b3-b32b-4b74b6a1804d

After this, you should be able to ping and access the VM from your undercloud using its floating IP.

Some basic debugging tools

There is not a single way to debug Neutron. It depends on your preferences. Depending on the complexity, you might want to check PDB or remote PDB, a wrapper that uses a tcp socket to comunicate with the external world.

Sometimes, adding logs might be enough for debugging purposes. If you want to check the logs in Devstack, use:

$ journalctl -u devstack@q-svc.service

If you are running the unit tests and want to debug those, you should enter the virtual environment created and execute stestr from there, otherwise you won’t be able to correctly work with the debugging tool:

source .tox/py38/bin/activate
stestr run -n <test path>

I won’t go into the details regarding the code because the codebase is huge and therefore the explanation would change depending on the fix you need to do. Nevertheless, there are some tools that will make your life easier when confronting the code at first:

Running the tests

Once you think you have the solution to the problem, you must ensure the code is correct before uploading it. If possible, also create at least a unit test that ensures your change will stay functional over time. Before writing your unit tests, read this post by Otherwiseguy on how NOT to write Python unit tests. It will help you creating better code on the long run.

The official documentation will provide up-to-date information on how to check your code. Tox is the main tool that provides a compilation of syntax, unit and functional tests. To ensure the syntax of your code is correct, you must run tox -e pep8. Unit tests must also pass successfully: tox -e py38 (the number next to py indicates the python version you want to use for testing). Finally, it is also nice to ensure functional tests are working as expected. In the Docs you will find functional and dsvm-functional. The latter is the one we need to check tox -e dsvm-functional. As advised, please use Devstack within a VM to execute those.

If you want to check only a certain group of tests, Tox supports filtering, so you can do something like this for unit and functional tests:

tox -e py38 -- neutron.tests.unit.services.ovn_l3

Once you upload your changes, your code will also go through the Zuul gates, and further testing will be performed.

Uploading your change!

All projects in Openstack are managed using Gerrit, an extension to git that works a bit differently than what you might be used to in GitHub/GitLab. See more on how to set up your Gerrit account.

Instead of pull request, commits are reviewed individually (although they can be chained). Use git review to upload your changes. You do not need to have a branch dedicated, but it is a good practice to make the changes in a branch named after the bug you are solving. It is important that your commit message is clear and that the changes that are uploaded stick strictly to your intended changes.

Your commit will be then reviewed by the community, where each person can decide to +1 or -1 your commit. If your commit is -1’d, you will get information on why that person decided your commit was not yet ready to be merged. Core reviewers can give you +2, and when at least 2 core reviewers give your commit +2, your code will be ready to be merged!

Back To Top