Neutron VXLAN multicast – disable IGMP snooping

openstack-ansible liberty 6e3815d, l2pop=off

At first I encountered this issue: https://bugs.launchpad.net/openstack-ansible/liberty/+bug/1563448 , but after deleting vxlan interfaces and restarting neutron agents, there are still problems.

  1. Unicast packets in VM tenent networks works, which means from agent namespaces you can ping VM with their IP in that subnet.
  2. Broadcast packets in VM tenant networks doesn’t work, which means VM can’t get DHCP reply, nor ARP reply, and nor can the DHCP agents. but strangely in my case L3 agent can get VM’s ARP reply.
  3. After further investigation, I found that vxlan-X interface on compute and agent containers doesn’t receive broadcast packets that others sent.
  4. Further, on physical interface (in my case `eth1.30`, slave of  `br-vxlan`), I can see packets going out to `vxlan_group` (239.1.1.1 by default), but not packets from 239.1.1.1
  5. Packet capturing on the switch found that the switch is not forwarding the packets.
  6. On #openstack-ansible @jimmdenton pointed me this article: http://movingpackets.net/2013/06/04/multicast-problems-juniper-ex/ , suggest to disable IGMP snooping
  7. It works!

Thanks to Rackspace guys at #openstack-ansible, who spend a lot of time helping me debug the issue.

This article helped me understand how VXLAN and L2 population works: https://kimizhang.wordpress.com/2014/04/01/how-ml2vxlan-works/ . (Don’t use L2 population! Neutron developers suggest against it!)

Multiple Neutron external network on one node

In order to achieve multiple external networks in Neutron, where each external network has different subnet and default gateway, we need multiple L3 agents. Note: this is only for Havana and below, since Change Id260a239 (developed in Icehouse cycle) multiple external networks can be handled by single l3-agent.

Official documentation has cover on this, I will cover much detail in this article.

Besides br-ex you originally have, we need to create a new OVS bridge br-ex-2, then add the physical interface connected to that external network to br-ex-2.

If there is IP address already associated with that physical interface (eth3), you need to clear it with:

# ip addr del x.y.z.w dev eth3

Create the bridge and add the port:

# ovs-vsctl add-br br-ex-2
# ovs-vsctl add-port br-ex-2 eth3

Same as when you configured br-ex , configure /etc/network/interfaces like this:

# We need the section below or eth3 will not be up on boot
auto eth3
        iface eth3 inet manual
        up ifconfig $IFACE 0.0.0.0 up
        up ip link set $IFACE promisc on
        down ip link set $IFACE promisc off
        down ifconfig $IFACE down
# br-ex-2 should have a IP belongs to that external subnet, default gateway should not be needed, since you already have br-ex
auto br-ex-2
iface br-ex-2 inet static
        address x.y.z.w
        netmask 255.255.255.0
        network x.y.z.0
        broadcast x.y.z.255

Each l3-agent has respective l3_agent.ini:

# l3_agent.ini
[DEFAULT]
host = network
external_network_bridge = br-ex
metadata_port = 8775
gateway_external_network_id = 9de13fee-fdcf-4343-9a8b-9f04deaf0d41 # Your external network id obtained via neutron net-list
router_id = 046c79ed-64f3-40ce-9328-f8b6329e81f4 # Your external router ID obtained via neutron router-list
# OVS
interface_driver = neutron.agent.linux.interface.OVSInterfaceDriver
# l3_agent-2.ini
[DEFAULT]
#debug = True
host = network-psuedo2
external_network_bridge = br-ex-2
metadata_port = 8775
handle_internal_only_routers = False
gateway_external_network_id = cc4ad4e0-e86e-43cf-bcd8-36de71be8b01
router_id = 6e9cc927-73ba-40e4-8d2e-e2a8fca24280
# OVS
interface_driver = neutron.agent.linux.interface.OVSInterfaceDriver

The host directive distinguishes multiple l3-agents, or they’ll be recognized as the same agent in Neutron.
Also add network-psuedo2 to /etc/hosts , point to the IP address you set on br-ex-2:

x.y.z.w      network-psuedo2

Now just start the agent in commandline:

# /usr/bin/python /usr/bin/neutron-l3-agent --config-file=/etc/neutron/neutron.conf --config-file=/etc/neutron/l3_agent-2.ini --log-file=/var/log/neutron/-agent-2.log

Note: I’m not sure if there’s any risk running l3-agent as root, but I have yet to figure out a way to run it as the neutron user. You can also combine nohup tool.

Deploy OpenStack with Chef, Part 2: Network

In this part we’ll set up the network node, Neutron plugins and create base Neutron networks. Rackspace’s documentation to set up Neutron was confusing to me, so I checked the official OpenStack Documentation and figured out some things need to be done differently from Rackspace’s documentation.

Set up br-ex

Execute commands in this section on the network node.

After the chef-client run on network node finishes, br-ex is created, we need to manually add our external interface to br-ex, during the procedure the external network connectivity will be lost so it’s a good idea to SSH into network node via its private IP (In my case 172.17.1.5), also just in case that external network never get back up.

Remove IP address on eth0 because we want to add it to a bridge. Network connectivity on eth0 will be lost. I learned this command here.

# ip addr flush dev eth0

Add eth0 to the bridge br-ex

# ovs-vsctl add-port br-ex eth0

Assign public IP to br-ex and add default gateway

# ifconfig br-ex inet x.y.54.54 netmask 255.255.255.0
# route add default gw x.y.54.254

Now you can verify if external connectivity works. If not, check your routing table, mine looks like this:

0.0.0.0         x.y.54.254      0.0.0.0         UG        0 0          0 br-ex
140.113.54.0    0.0.0.0         255.255.255.0   U         0 0          0 br-ex
172.17.1.0      0.0.0.0         255.255.255.0   U         0 0          0 eth1

Edit /etc/network/interfaces as my example to make these settings persistent.

Create base Neutron networks

Execute commands in this section on the controller node.

Mostly the same as official OpenStack documentation.

Read credentials:

# source ~/openrc

Create net network:

# neutron net-create ext-net --router:external=True \
  --provider:network_type gre --provider:segmentation_id 2 --shared

Create subnet for ext-net:

# neutron subnet-create ext-net --allocation-pool start=x.y.54.1,end=x.y.54.10 \
  --gateway=x.y.54.254 --enable_dhcp=False x.y.54.0/24

Create router:

# neutron router-create ext-to-int

Connect the router to ext-net by setting the gateway for the router as ext-net:

# neutron router-gateway-set EXT_TO_INT_ROUTER_ID EXT_NET_ID

Create internal network and subnet:

# neutron net-create admin-net
# neutron subnet-create admin-net 192.168.2.0/24 --gateway 192.168.2.1

Connect the internal subnet to router:

# neutron router-interface-add EXT_TO_INT_ROUTER_ID INTERNAL_SUBNET_ID

Update internal subnet’s DNS information:

# neutron subnet-update INTERNAL_SUBNET_ID \
  --dns_nameservers list=true 8.8.8.8 8.8.8.7

Now you can boot up an instance, select the interface to admin-net, and see if it gets IP and DNS settings correctly.

Deploy OpenStack with Chef, Part 3: Storage

Cinder needs a LVM volume group to create volumes. Make sure you have free space on the VG (on storage node):

# vgdisplay
  --- Volume group ---
  VG Name               storage-vg
  System ID
  Format                lvm2
  Metadata Areas        1
  Metadata Sequence No  5
  VG Access             read/write
  VG Status             resizable
  MAX LV                0
  Cur LV                3
  Open LV               3
  Max PV                0
  Cur PV                1
  Act PV                1
  VG Size               546.21 GiB
  PE Size               4.00 MiB
  Total PE              139831
  Alloc PE / Size       13824 / 54.00 GiB
  Free  PE / Size       126007 / 492.21 GiB
  VG UUID               MfihR0-9350-o1T9-OQhU-aL2E-RtwA-JO9I7d

If your have no free space left, you can follow this guide to shrink existing partitions (LV).

In the environment file, you can specify which VG cinder should use:

    "cinder": {
      "storage": {
        "lvm": {
          "volume_group": "<VG_name>"
        }
      }
    },

Then on chef server, add the cinder-all role:

# knife node run_list add storage.openstack.xxxx.tw 'role[cinder-all]'

Run chef-client on storage node:

# chef-client -E <environment_name>

Run chef-client on controller node again, since cinder-api and cinder-scheduler run on it:

# chef-client -E <environment_name>

Problems

I encountered a weird problem that cinder services on storage node cannot connect to MySQL on controller. Investigation found that /etc/cinder.conf has wrong mysql password. I copied over mysql password for user cinder from controller’s /etc/cinder.conf and it worked. This is just a temporary fix, I guess it would be overwritten on the next chef-client run, not sure why chef-client gets the wrong password.

Deploy OpenStack with Chef, Part 0: Basic Concepts & Planning Environment

In this series of post I will demonstrate deploying OpenStack with Chef, using Rackspace Private Cloud’s cookbooks. This guide largely follows Rackspace’s documentation (Getting Started Guide, list of documentation), with notes by myself.

One thing to clarify is that Chef doens’t do the bootstrapping of nodes for you. I misunderstood this at first. You need to install and configure the OS, networking, network bridges required by Neutron manually.

My setup will be

  1. controller node: x.y.54.50, 172.17.1.1
  2. compute node: x.y.54.51, 172.17.1.2
  3. network node: x.y.54.54, 172.17.1.5, runs Neutron, with OVS plugin and GRE tenant network
  4. storage node: x.y.54.53, 172.17.1.4, runs Cinder

Each node has 2 interfaces, first one (not necessarily eth0, since there will be br-ex on network node) connects to public network x.y.54.0/24, each node has a public IP address; the second interface connects to a private subnet 172.17.1.0/24 .

First install Ubuntu 12.04 on each node, set up public IP addresses on eth0 so that you can access by SSH, we’ll reconfigure some part of them later.

Chef Architecture

chef-server stores the environment file, cookbooks, roles for each node.

Environment file describes your environment by overriding default attributes.

Cookbooks are chain of actions to bring the node to certain “state", cookbook are collections of attributes and recipes.

Role is a role for a machine, a role can be composed of many recipes and certain attributes or even other roles.

chef-server needs not to be in the same subnet as the nodes, it only needs to be accesible by all nodes. Putting chef-server on the Internet is also entirely fine.

chef-client needs to be installed on all nodes, whenever you run chef-client, it contacts chef-server and see if there’s any change of the environment or the role of the node. It updates the node to desired state by executing recipes.

Also take a look at http://www.rackspace.com/knowledge_center/article/rackspace-private-cloud-installation-prerequisites-and-concepts

Install chef-server and bootstrap the nodes

I highly recommend setting up FQDN for all nodes and chef-server, and set up SSH passwordless login for root on each node.

Put id_rsa in the chef-server, copy id_rsa.pub to all nodes.

chef-server should be acessible by all nodes at port 443 and 80.

Make sure all nodes and chef-server’s hostname are correctly configured. /etc/hostname should contain <hostname> and /etc/hosts should have entry like this:

x.y.54.50   controller.openstack.example.com        controller

Then just follow Rackspace’s guide: Install Chef Server, Cookbooks, and chef-client

Environment file

{
  "name": "swarm1",
  "description": "OpenStack Demo : Swarm 1",
  "cookbook_versions": {
  },
  "json_class": "Chef::Environment",
  "chef_type": "environment",
  "default_attributes": {
  },
  "override_attributes": {
    "nova": {
      "network": {
        "provider": "neutron"
      }
    },
    "neutron": {
      "ovs": {
        "network_type": "gre"
      }
    },
    "horizon": {
      "endpoint_type": "publicURL",
      "endpoint_scheme": "http"
    },
    "mysql": {
      "allow_remote_root": true,
      "root_network_acl": "%"
    },
    "cinder": {
      "storage": {
        "lvm": {
          "volume_group": "storage-vg"
        }
      }
    },
    "osops_networks": {
      "nova": "x.y.54.0/24",
      "public": "x.y.54.0/24",
      "management": "172.17.1.0/24"
    }
  }
}

The horizon part is the fix for this bug. Additionaly, patch chef-cookbooks/cookbooks/horizon/recipes/server.rb like this patch. More information on this bug

Provider network settings like below is not needed in GRE setup. (They had it on Getting Started Guide p.33 so I misunderstood that I need it)

 "provider_networks": [
    {
        "label": "ph-eth1",
        "bridge": "br-eth1",
        "vlans": "1:1000"
    }
]

The osops_networks section defines networks that services should listen on (endpoints). Following list is copied from Rackspace Private Cloud Installation Prerequisites and Concepts.

Network Services
nova
  • keystone-admin-api
  • nova-xvpvnc-proxy
  • nova-novnc-proxy
  • nova-novnc-server
public
  • graphite-api
  • keystone-service-api
  • glance-api
  • glance-registry
  • nova-api
  • nova-ec2-admin
  • nova-ec2-public
  • nova-volume
  • neutron-api
  • cinder-api
  • ceilometer-api
  • horizon-dash
  • horizon-dash_ssl
management
  • graphite-statsd
  • graphite-carbon-line-receiver
  • graphite-carbon-pickle-receiver
  • graphite-carbon-cache-query
  • memcached
  • collectd
  • mysql
  • keystone-internal-api
  • glance-admin-api
  • glance-internal-api
  • nova-internal-api
  • nova-admin-api
  • cinder-internal-api
  • cinder-admin-api
  • cinder-volume
  • ceilometer-internal-api
  • ceilometer-admin-api
  • ceilometer-central

nova network should be accessible by future Horizon user, since the VNC proxy (nova-novnc-proxy) listens on that network, it must be accessible by clients so that their VNC console in Horizon will work. In nearly all cases you should put nova and public on the same network.

Chef will not set up these networks for you. During chef-client run of each node, it searches for the node’s IP on that network, and bind (some) services to that IP. For example, my management network is on 172.17.1.0/24, when deploying mysql on the controller node, chef-client searches for the node’s IP on that network, the node actually have 2 IPs, x.y.54.50 and 172.17.1.1, so it binds mysql to 172.17.1.1 .

Run chef-server and OpenStack controller on same node

Update: This did not end up well so I gave up, actually you can put Chef server anywhere as long as it can be reached by clients, even in another VM would work after some patches to RackSpace scripts.

Chef’s own rabbitmq-server will conflict with OpenStack’s, so I had to configure Chef to use the system rabbitmq.

Use this guide: https://community.rackspace.com/products/f/45/t/1669

Also add
bookshelf['url'] = "https://#{node['ipaddress']}:4000"
to /etc/chef-server/chef-server.rb.

ref: https://groups.google.com/forum/#!topic/opscode-chef-openstack/0DnbYhxQT90