OpenStack Mitaka is out the door and it is time to start looking at OpenStack Newton. OpenStack Newton brings changes to how Neutron is configured out of the box, for example, of_interface driver will switch from ofctl to native. To read more about how things have changed visit, Unreleased Notes. We want to measure how these changes could impact Neutrons API Performance.
Our methodology behind Neutron API testing was to use OpenStack TripleO to deploy OpenStack Mitaka and OpenStack Newton on the same exact hardware, and execute the same set of Browbeat-Rally scenarios at the same times and concurrencies. With this methodology we were able to quickly deploy on baremetal and capture API Performance metrics from these two different releases.
Our Neutron scenarios we defined in Browbeat:
- name: neutron enabled: true concurrency: - 50 times: 2500 scenarios: - name: create-list-network enabled: true file: rally/neutron/neutron-create-list-network-cc.yml sla_max_seconds: 30 sla_max_failure: 0 - name: create-list-port enabled: true file: rally/neutron/neutron-create-list-port-cc.yml - name: create-list-router enabled: true file: rally/neutron/neutron-create-list-router-cc.yml - name: create-list-security-group enabled: true file: rally/neutron/neutron-create-list-security-group-cc.yml - name: create-list-subnet enabled: true file: rally/neutron/neutron-create-list-subnet-cc.yml
To view the entire browbeat-config : browbeat-neutron-performance-test.yml
This Scenario will run every described test at a concurrency of 50 neutron objects at a time until it reaches 2500 neutron objects — a neutron object could be a port/network/router/etc. Browbeat will execute each of the scenarios 3 times to get a distribution of results.
Reviewing out of the box Neutron API Performance 99th percentil results of both releases:
In OpenStack Newton, Neutron changed how it handles
api_workers=0 in neutron.conf, and instead of deploying with # of cores == # of api workers, Neutron deploys with a single API worker. This caused a huge performance regression in Newton when deployed with TripleO, see below.
The metrics depicted above shows a limited set of tests that completed, this is because the OpenStack Newton cloud ended up failing most tests due to overloading the single worker with too many requests. This highlights the importance of properly configuring the service, and having tooling to help identify possible performance regressions due to incorrect configurations.
Reviewing the results when Mitaka and Newton are configured with 40 Neutron API workers ( ignoring the case when
api_workers=0 for Newton), the top 10 API calls reveal that OpenStack Newton is under-performing for almost every single API call. This isn’t an order of magnitude difference, for example the port-list operation is ~2 seconds off comparing OpenStack Newton to OpenStack Mitaka.
It is important to point out, using OpenStack Browbeat we identified the performance regression when
api_workers=0. This is why we feel that it is important to get OpenStack Browbeat pulled into CI infrastructure. Once we have OpenStack Browbeat in the CI workflow, we can index results into ElasticSearch and visualize via Kibana, to quickly identify build to build regressions or release to release regressions. In the case of Mitaka to Newton Neutron API Performance, there is a slight regression that we will be doing more investigations on.
Be looking out for another blog post going into our vision of Performance CI.