This document describes how to use Open Virtual Networking with Kubernetes 1.8.0 or later. This document assumes that you have installed Open vSwitch by following [INSTALL.rst] or by using the distribution packages such as .deb or.rpm. ## Setup OVN provides network virtualization to containers. In the "overlay" mode, OVN can create a logical network amongst containers running on multiple hosts. In this mode, OVN programs the Open vSwitch instances running inside your hosts. These hosts can be bare-metal machines or vanilla VMs. Installing Open vSwitch is out of scope of this documentation. You can read [INSTALL.rst] for that. That documentation in turn links to platform specific installations. If you use packages to install OVS, you should install both OVS and OVN related packages. You can also read the following quick-start guide for Ubuntu that installs OVS and OVN from source and packages: [INSTALL.UBUNTU.md] The following guide explains setting up an OVN overlay network on Openshift running on RHEL/Centos/Fedora: [INSTALL.OPENSHIFT.md](docs/INSTALL.OPENSHIFT.md) On each node, you should also install the 'ovnkube' utility that comes with the [ovn-org/ovn-kubernetes repo]. To compile it, you will need golang installed. You can install golang with: ``` wget -nv https://dl.google.com/go/go1.9.2.linux-amd64.tar.gz sudo tar -C /usr/local -xzf go1.9.2.linux-amd64.tar.gz ``` You need to set your GOPATH, clone the ovn-kubernetes repo, compile and install it. ``` export GOPATH=$HOME/work mkdir -p $GOPATH/github.com/ovn-org cd $GOPATH/github.com/ovn-org git clone https://github.com/ovn-org/ovn-kubernetes cd ovn-kubernetes/go-controller make sudo make install ``` Note that, you need not do the above compilation on every host. You can copy over the compiled binaries to other hosts. We will have the option to run OVN via kubernetes daemonsets soon. ### Kubernetes networking requirements: OVN is a network virtualization solution. It can create logical switches and routers. One can then interconnect these switches and routers to create any network topologies. Kubernetes (k8s) in its default mode has the following networking requirements. 1. All containers should be able to communicate with each other over IP address. OVN is capable of having overlapping IP addresses (as part of different network topologies). But with the k8s requirement that all containers should be able to talk over IP address, it makes sense to have a simple network topology with one logical switch created per k8s node and then inter-connect all those logical switches with a single logical router. Any containers created in these nodes get connected to the logical switch associated with that node as a logical port. 2. All nodes should be able to communicate with the containers running inside that node over IP address. k8s has this requirement for health-checking. To achieve this goal, we will create an additional logical port for the node (represented by a OVS internal interface). 3. All containers should be able to communicate with the k8s daemons running on the master. This can be achieved by running OVN gateways on the nodes. With at least one OVN gateway, the pods can reach the k8s central daemons with NAT. ### Master initialization * Start the central components on a k8s master. OVN architecture has a central component which stores your networking intent in a database. Start this central component on one of the hosts where you have started your k8s central daemons and which has an IP address of $CENTRAL_IP. (For HA of the central component, please read [HA.md]) Run the following commands to open up the TCP ports for OVN Northbound and Southbound ovsdb-server. The clients will connect to the respective database at its port. ``` ovn-nbctl set-connection ptcp:6641 ovn-sbctl set-connection ptcp:6642 ``` Start ovn-northd daemon. This daemon translates networking intent from k8s stored in the OVN_Northbound database to logical flows in OVN_Southbound database. ``` /usr/share/openvswitch/scripts/ovn-ctl start_northd ``` Also start ovn-controller on this node. ``` /usr/share/openvswitch/scripts/ovn-ctl start_controller ``` Now start the ovnkube utility on the master. The below command expects the user to provide * A cluster wide private address range of $CLUSTER_IP_SUBNET (e.g: 192.168.0.0/16). The pods are provided IP addresses from this range. * $NODE_NAME should be the same as the one used by kubelet. kubelet by default uses the hostname. kubelet allows this name to be overridden with --hostname-override. * $SERVICE_IP_SUBNET is the same as the one provided to k8s-apiserver via --service-cluster-ip-range option. An e.g is 172.16.1.0/24. ``` nohup sudo ovnkube -k8s-kubeconfig kubeconfig.yaml \ -loglevel=4 \ -k8s-apiserver="http://$CENTRAL_IP:8080" \ -logfile="/var/log/ovn-kubernetes/ovnkube.log" \ -init-master=$NODE_NAME -init-node=$NODE_NAME \ -cluster-subnets="$CLUSTER_IP_SUBNET" \ -k8s-service-cidr=$SERVICE_IP_SUBNET \ -nodeport \ -init-gateways -gateway-local \ -k8s-token="$TOKEN" \ -nb-address="tcp:$CENTRAL_IP:6641" \ -sb-address="tcp:$CENTRAL_IP:6642" 2>&1 & ``` Note: Make sure to read /var/log/ovn-kubernetes/ovnkube.log to see that there were no obvious errors with argument passing. Also, you should only pass "-init-node" argument if there is a kubelet running on the master too. If you want to use SSL instead of TCP for OVN databases, please read [INSTALL.SSL.md]. ### Node initialization. On each host, you will need to run the following command once. In addition to the same variables explained in the previous step, if you are using ingress controllers with L7 load-balancing to enter into the k8s cluster, you can skip providing the '-nodeport' option with the below command. ``` nohup sudo ovnkube -k8s-kubeconfig kubeconfig.yaml -loglevel=4 \ -logfile="/var/log/ovn-kubernetes/ovnkube.log" \ -k8s-apiserver="http://$CENTRAL_IP:8080" \ -init-node="$NODE_NAME" \ -nodeport \ -nb-address="tcp:$CENTRAL_IP:6641" \ -sb-address="tcp:$CENTRAL_IP:6642" -k8s-token="$TOKEN" \ -init-gateways \ -k8s-service-cidr=$SERVICE_IP_SUBNET \ -cluster-subnets=$CLUSTER_IP_SUBNET 2>&1 & ``` Note: Make sure to read /var/log/ovn-kubernetes/ovnkube.log to see that there were no obvious errors with argument passing. Notes on gateway nodes: * Gateway nodes are needed for North-South connectivity in OVN. OVN has support for multiple gateway nodes. In the above command, since '-init-gateways' has been provided as an option, a OVN gateway will be created on each node. * Just providing '-init-gateways', will make OVN choose the interface in your node via which the node default gateway is reached. * If you want to chose the interface for your gateway, you should provide '-gateway-interface' and '-gateway-nexthop' as options. For e.g: ``` -init-gateways -gateway-interface=enp0s9 -gateway-nexthop="$NEXTHOP" ``` For both master and node initializations above, ovnkube will create an OVS bridge on top of your physical interface and move the IP address and route information from the physical interface to OVS bridge. If you are using Ubuntu and OVS startup scripts are systemd (e.g: there is a file called /lib/systemd/system/ovsdb-server.service) , you will have to add the following line to /etc/default/openvswitch ``` OPTIONS=--delete-transient-ports ``` For more control on the options to ovnkube, please read [config.md] ## Debugging Please read [debugging.md]. ### Overlay mode architecture diagram: The following digaram represents the internal architecture details of the overlay mode. ![alt text](https://i.imgur.com/i7sci9O.png "Overlay mode diagram") ## Installing Kubernetes Installing k8s is out of scope of this documentation. You can read [README.K8S.md] for that. For a quick start, the vagrant in this repo, does install kubernetes in its simplest form. [INSTALL.rst]: http://docs.openvswitch.org/en/latest/intro/install [INSTALL.UBUNTU.md]: docs/INSTALL.UBUNTU.md [README.K8S.md]: https://github.com/kubernetes/kubernetes/tree/master/docs [README.RHEL.rst]: https://github.com/openvswitch/ovs/blob/master/rhel/README.RHEL.rst [debugging.md]: docs/debugging.md [vagrant]: vagrant/README.md [INSTALL.SSL.md]: docs/INSTALL.SSL.md [config.md]: docs/config.md [HA.md]: docs/ha.md [ovn-org/ovn-kubernetes repo]: https://github.com/ovn-org/ovn-kubernetes