OpenStack Newton vs Mitaka : Neutron API Performance

openstack_logoOpenStack Mitaka is out the door and it is time to start looking at OpenStack Newton. OpenStack Newton brings changes to how Neutron is configured out of the box, for example, of_interface driver will switch from ofctl to native. To read more about how things have changed visit, Unreleased Notes. We want to measure how these changes could impact Neutrons API Performance.

Our methodology behind Neutron API testing was to use OpenStack TripleO to deploy OpenStack Mitaka and OpenStack Newton on the same exact hardware, and execute the same set of Browbeat-Rally scenarios at the same times and  concurrencies. With this methodology we were able to quickly deploy on baremetal and capture API Performance metrics from these two different releases.

Our Neutron scenarios we defined in Browbeat:

- name: neutron
  enabled: true
  concurrency:
    - 50
  times: 2500
  scenarios:
    - name: create-list-network
      enabled: true
      file: rally/neutron/neutron-create-list-network-cc.yml
      sla_max_seconds: 30
      sla_max_failure: 0
    - name: create-list-port
      enabled: true
      file: rally/neutron/neutron-create-list-port-cc.yml
    - name: create-list-router
      enabled: true
      file: rally/neutron/neutron-create-list-router-cc.yml
    - name: create-list-security-group
      enabled: true
      file: rally/neutron/neutron-create-list-security-group-cc.yml
    - name: create-list-subnet
      enabled: true
      file: rally/neutron/neutron-create-list-subnet-cc.yml

To view the entire browbeat-config : browbeat-neutron-performance-test.yml

This Scenario will run every described test at a concurrency of 50 neutron objects at a time until it reaches 2500 neutron objects — a neutron object could be a port/network/router/etc. Browbeat will execute each of the scenarios 3 times to get a distribution of results.

Reviewing out of the box Neutron API Performance 99th percentil results of both releases:

screenshot-from-2016-09-15-11-26-43

In OpenStack Newton, Neutron changed how it handles api_workers=0 in neutron.conf, and instead of deploying with # of cores == # of api workers, Neutron deploys with a single API worker. This caused a huge performance regression in Newton when deployed with TripleO, see below.

screenshot-from-2016-09-15-11-38-05

The metrics depicted above shows a limited set of tests that completed, this is because the OpenStack Newton cloud ended up failing most tests due to overloading the single worker with too many requests. This highlights the importance of properly configuring the service, and having tooling to help identify possible performance regressions due to incorrect configurations.

Reviewing the results when Mitaka and Newton are configured with 40 Neutron API workers ( ignoring the case when api_workers=0 for Newton),  the top 10 API calls reveal that OpenStack Newton is under-performing for almost every single API call.  This isn’t an order of magnitude difference, for example the port-list operation is ~2 seconds off comparing OpenStack Newton to OpenStack Mitaka.

screenshot-from-2016-09-15-13-45-12

It is important to point out, using OpenStack Browbeat we identified the performance regression when api_workers=0. This is why we feel that it is important to get OpenStack Browbeat pulled into CI infrastructure. Once we have OpenStack Browbeat in the CI workflow, we can index results into ElasticSearch and visualize via Kibana, to quickly identify build to build regressions or release to release regressions. In the case of Mitaka to Newton Neutron API Performance, there is a slight regression that we will be doing more investigations on.

Be looking out for another blog post going into our vision of Performance CI.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s