Network Performance : Rally-Pbench-uperf integration

Browbeat ships with a Rally plugin which integrates PBench, this marriage of tools allows us to develop many low-level tests. Integrating PBench into Browbeat opens the doors for the potential to orchestrate NFV workloads in the future.

Today, we have pbench-uperf integrated, and we will discuss what this work actually does, first off here is a friendly image depicting what is built in the OpenStack cloud.

nova-create-pbench-uperf.png

There are some requirements we should discuss when it comes to running the pbench plugin. First, use a CentOS7 image that has removed notty, so we can install via cloud-init (like this one). We recommend using the pbench-helper scripts, this will install dependencies for pbench and pbench-uperf in the instance, to use the helper-script.

nova boot --nic net-id=<netid> --flavor m1.small --image centos-notty --user-data ./pbench-user.file pbench-image

The network which the instance launches on must be able to reach the outside world, ie via provider networks, or a Neutron router. This is needed because the instance needs to pull down the PBench packages to install.

Verify the install went through by viewing the console-log of the host that is building. You should see:

centos-require login: [ 9.907660] cloud-init[1977]: Cloud-init v. 0.7.5 running 'modules:config' at Tue, 21 Mar 2017 18:47
:45 +0000. Up 9.83 seconds.
[ 10.278948] cloud-init[1998]: Cloud-init v. 0.7.5 running 'modules:final' at Tue, 21 Mar 2017 18:47:45 +0000. Up 10.20 sec
onds.
[ 10.330330] cloud-init[1998]: # Template file to be used with ansible playbook "pbench-repo.yml"
[ 10.331473] cloud-init[1998]: ###########################################################################
[ 10.332561] cloud-init[1998]: [pbench]
[ 10.333179] cloud-init[1998]: name=Pbench 7Server - x86_64
[ 10.333905] cloud-init[1998]: baseurl=http://www/repo/7Server
[ 10.334883] cloud-init[1998]: arch=x86_64
[ 10.335511] cloud-init[1998]: enabled=1
[ 10.336121] cloud-init[1998]: gpgcheck=0
[ 10.336716] cloud-init[1998]: skip_if_unavailable=1
[ 10.337416] cloud-init[1998]: # External COPR repo
[ 10.338111] cloud-init[1998]: [copr-pbench]
[ 10.338724] cloud-init[1998]: name=Copr repo for pbench owned by ndokos
[ 10.339556] cloud-init[1998]: baseurl=https://copr-be.cloud.fedoraproject.org/results/ndokos/pbench/epel-7-x86_64/
[ 10.341110] cloud-init[1998]: skip_if_unavailable=True
[ 10.341814] cloud-init[1998]: gpgcheck=1
[ 10.342434] cloud-init[1998]: gpgkey=https://copr-be.cloud.fedoraproject.org/results/ndokos/pbench/pubkey.gpg
[ 10.343561] cloud-init[1998]: enabled=1
[ 10.344174] cloud-init[1998]: enabled_metadata=1
[ 10.344832] cloud-init[1998]: skip_if_unavailable=1
[ 10.639529] cloud-init[1998]: Loaded plugins: fastestmirror
[ 10.687422] cloud-init[1998]: Cleaning repos: base copr-pbench extras pbench updates
[ 10.688405] cloud-init[1998]: Cleaning up everything
[ 10.811089] cloud-init[1998]: Loaded plugins: fastestmirror
[ 246.895727] cloud-init[1998]: Determining fastest mirrors
[ 247.070033] cloud-init[1998]: * base: www.gtlib.gatech.edu
[ 247.070794] cloud-init[1998]: * extras: repo1.ash.innoscale.net
[ 247.071566] cloud-init[1998]: * updates: ftp.linux.ncsu.edu
[ 248.959033] cloud-init[1998]: Resolving Dependencies
[ 248.961414] cloud-init[1998]: --> Running transaction check
[ 248.962132] cloud-init[1998]: ---> Package pbench-agent.noarch 0:0.40-1 will be installed
...
[ 260.783961] cloud-init[1998]: Running transaction test
[ 260.784693] cloud-init[1998]: Transaction test succeeded
[ 260.785383] cloud-init[1998]: Running transaction
[ 260.857900] cloud-init[1998]: Installing : pbench-uperf-1.0.4-1.x86_64 1/1
[ 260.933543] cloud-init[1998]: Verifying : pbench-uperf-1.0.4-1.x86_64 1/1
[ 260.934854] cloud-init[1998]: Installed:
[ 260.935479] cloud-init[1998]: pbench-uperf.x86_64 0:1.0.4-1
[ 260.936247] cloud-init[1998]: Complete!
[ 260.970737] cloud-init[1998]: disable_root: 0

On the OpenStack Controllers where Nova and Neutron are running we need to update the policy.json files.

Neutron’s policy.json:

create_router:external_gateway_info:enable_snat": "rule:regular_user",

Nova’s  policy.json:

"os_compute_api:servers:create:forced_host": "",

After the policies are updated, there is no need to restart services.

For the Browbeat configuration, we need to capture a couple of important details. First, we need to define the hypervisors, you can pass a single hypervisor if you only have one. You can capture this from running.

nova hypervisor-list

Next we need to provide the pbench image name, which has pbench installed into it. In our case this is pbench-image. If you are using Elasticsearch to warehouse results, provide the cloudname, elastic_host and elastic_port.

 protocols: "tcp"            # Can be: tcp,udp
 num_pairs: 1                # Number of pair of guests
 message_sizes: "1024,16384" # Message sizes for uperf
 instances: 1                # Number of uperf instances running
 test_types: "stream"        # stream, rr
 samples: 3                  # Number of samples for each message size

Note: Today we do not expose the message sizes, we run with the default. This could be improved upon. ( https://review.openstack.org/#/c/446515/ )

Note: providing a password is unnecessary since we use ssh-key exchange.

benchmarks:
 - name: plugins
 enabled: true
 concurrency:
 - 1
 times: 1
 scenarios:
 - name: nova-create-pbench-uperf
 enabled: true
 hypervisor_server: "nova:overcloud-compute-1.localdomain"
 hypervisor_client: "nova:overcloud-compute-0.localdomain"
 image_name: pbench-image
 flavor_name: m1.small
 cloudname: "openstack-pbench"
 elastic_host: "elastic.com"
 elastic_port: 9200
 user: "root"
 password: "password"
 external_network: "public"
 protocols: "tcp"
 num_pairs: 1
 message_sizes: "1024,16384"
 instances: 1
 test_types: "stream"
 samples: 1
 send_results: True
 test_name: "browbeat-rally"
 file: rally/rally-plugins/nova-create-pbench-uperf/nova-create-pbench-uperf.yml

Note: The password field here is not used, since we exchange ssh-keys. There will be a patch to make this optional. ( https://review.openstack.org/#/c/446515/ )

Once the Browbeat configuration is updated, execute Browbeat. Once Browbeat completes, you should should see PBench Uperf Stats in the log file. Example Below:

--------------------------------------------------------------------------------
Task 2b2c9df2-116b-4fad-91d0-3c11d95022e2 has 0 error(s)
--------------------------------------------------------------------------------

+---------------------------------------------------------------------------------------------------------------------------------+
| Response Times (sec) |
+------------------------------+-----------+--------------+--------------+--------------+-----------+-----------+---------+-------+
| Action | Min (sec) | Median (sec) | 90%ile (sec) | 95%ile (sec) | Max (sec) | Avg (sec) | Success | Count |
+------------------------------+-----------+--------------+--------------+--------------+-----------+-----------+---------+-------+
| neutron.create_router | 1.271 | 1.271 | 1.271 | 1.271 | 1.271 | 1.271 | 100.0% | 1 |
| neutron.list_networks | 0.378 | 0.378 | 0.378 | 0.378 | 0.378 | 0.378 | 100.0% | 1 |
| neutron.create_network | 0.129 | 0.129 | 0.129 | 0.129 | 0.129 | 0.129 | 100.0% | 1 |
| neutron.create_subnet | 0.366 | 0.366 | 0.366 | 0.366 | 0.366 | 0.366 | 100.0% | 1 |
| neutron.add_interface_router | 1.316 | 1.316 | 1.316 | 1.316 | 1.316 | 1.316 | 100.0% | 1 |
| nova.boot_server | 13.418 | 13.418 | 13.418 | 13.418 | 13.418 | 13.418 | 100.0% | 1 |
| vm.attach_floating_ip | 2.361 | 2.361 | 2.361 | 2.361 | 2.361 | 2.361 | 100.0% | 1 |
| nova.boot_server (2) | 11.811 | 11.811 | 11.811 | 11.811 | 11.811 | 11.811 | 100.0% | 1 |
| nova.boot_server (3) | 8.107 | 8.107 | 8.107 | 8.107 | 8.107 | 8.107 | 100.0% | 1 |
| vm.wait_for_ping | 0.012 | 0.012 | 0.012 | 0.012 | 0.012 | 0.012 | 100.0% | 1 |
| vm.wait_for_ssh | 0.252 | 0.252 | 0.252 | 0.252 | 0.252 | 0.252 | 100.0% | 1 |
| vm.run_command_over_ssh | 0.049 | 0.049 | 0.049 | 0.049 | 0.049 | 0.049 | 100.0% | 1 |
| vm.run_command_over_ssh (2) | 0.05 | 0.05 | 0.05 | 0.05 | 0.05 | 0.05 | 100.0% | 1 |
| vm.run_command_over_ssh (3) | 0.66 | 0.66 | 0.66 | 0.66 | 0.66 | 0.66 | 100.0% | 1 |
| total | 910.907 | 910.907 | 910.907 | 910.907 | 910.907 | 910.907 | 100.0% | 1 |
+------------------------------+-----------+--------------+--------------+--------------+-----------+-----------+---------+-------+

PBench UPerf Stats
+---------------------------------+-----------+--------------+--------------+--------------+-----------+-----------+-------+
| Action | Min (sec) | Median (sec) | 90%ile (sec) | 95%ile (sec) | Max (sec) | Avg (sec) | Count |
+---------------------------------+-----------+--------------+--------------+--------------+-----------+-----------+-------+
| aggregate.tcp_stream-1024B-1i | 3.965 | 3.965 | 3.965 | 3.965 | 3.965 | 2.482 | 1 |
| aggregate.tcp_stream-1024B-64i | 5.296 | 5.296 | 5.296 | 5.296 | 5.296 | 3.148 | 1 |
| aggregate.tcp_stream-1024B-8i | 5.162 | 5.162 | 5.162 | 5.162 | 5.162 | 3.081 | 1 |
| aggregate.tcp_stream-16384B-1i | 3.623 | 3.623 | 3.623 | 3.623 | 3.623 | 2.311 | 1 |
| aggregate.tcp_stream-16384B-64i | 5.095 | 5.095 | 5.095 | 5.095 | 5.095 | 3.047 | 1 |
| aggregate.tcp_stream-16384B-8i | 5.631 | 5.631 | 5.631 | 5.631 | 5.631 | 3.315 | 1 |
| aggregate.tcp_stream-1B-1i | 0.017 | 0.017 | 0.017 | 0.017 | 0.017 | 0.508 | 1 |
| aggregate.tcp_stream-1B-64i | 0.016 | 0.016 | 0.016 | 0.016 | 0.016 | 0.508 | 1 |
| aggregate.tcp_stream-1B-8i | 0.015 | 0.015 | 0.015 | 0.015 | 0.015 | 0.508 | 1 |
| aggregate.tcp_stream-64B-1i | 0.769 | 0.769 | 0.769 | 0.769 | 0.769 | 0.885 | 1 |
| aggregate.tcp_stream-64B-64i | 0.753 | 0.753 | 0.753 | 0.753 | 0.753 | 0.876 | 1 |
| aggregate.tcp_stream-64B-8i | 0.758 | 0.758 | 0.758 | 0.758 | 0.758 | 0.879 | 1 |
| single.tcp_stream-1024B-1i | 3.965 | 3.965 | 3.965 | 3.965 | 3.965 | 2.482 | 1 |
| single.tcp_stream-1024B-64i | 5.296 | 5.296 | 5.296 | 5.296 | 5.296 | 3.148 | 1 |
| single.tcp_stream-1024B-8i | 5.162 | 5.162 | 5.162 | 5.162 | 5.162 | 3.081 | 1 |
| single.tcp_stream-16384B-1i | 3.543 | 3.543 | 3.543 | 3.543 | 3.543 | 2.272 | 1 |
| single.tcp_stream-16384B-64i | 5.092 | 5.092 | 5.092 | 5.092 | 5.092 | 3.046 | 1 |
| single.tcp_stream-16384B-8i | 5.634 | 5.634 | 5.634 | 5.634 | 5.634 | 3.317 | 1 |
| single.tcp_stream-1B-1i | 0.017 | 0.017 | 0.017 | 0.017 | 0.017 | 0.508 | 1 |
| single.tcp_stream-1B-64i | 0.016 | 0.016 | 0.016 | 0.016 | 0.016 | 0.508 | 1 |
| single.tcp_stream-1B-8i | 0.015 | 0.015 | 0.015 | 0.015 | 0.015 | 0.508 | 1 |
| single.tcp_stream-64B-1i | 0.769 | 0.769 | 0.769 | 0.769 | 0.769 | 0.885 | 1 |
| single.tcp_stream-64B-64i | 0.753 | 0.753 | 0.753 | 0.753 | 0.753 | 0.876 | 1 |
| single.tcp_stream-64B-8i | 0.758 | 0.758 | 0.758 | 0.758 | 0.758 | 0.879 | 1 |
+---------------------------------+-----------+--------------+--------------+--------------+-----------+-----------+-------+

 

In the future we hope to integrate PBench-FIO and any NFV workloads that PBench integrates in the future. The work here should pave the way for any workload that PBench integrates, we can quickly prototype to run in OpenStack.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s