For CI, OVN-Kubernetes runs the Kubernetes E2E tests and some locally defined tests. GitHub Actions are used to run a subset of the Kubernetes E2E tests on each pull request. The local workflow that controls the test run is located in ovn-kubernetes/.github/workflows/test.yml.
The following tasks are performed:
Checks out the Kubernetes source tree and compiles some dependencies
Runs a matrix of End-To-End Tests using KIND
Below describes how to update the set of tests that run and how to run these tests locally.
Updating CI Test Suite¶
The tests are broken into a set of shards, which is just a grouping of tests, and each shard is run in a separate job in parallel. Below is an example of the shards (which may change in the future):
All E2E tests that match
[sig-network] Nand do NOT have P as their sixth letter after the N (i.e. Roughly all
[sig-network] Networking ...tests.), and all other tests that don’t match the rule below
All E2E tests that match
[sig-network] Nand DO have a P as their sixth letter after the N. (i.e. Roughly all
[sig-network] NetworkPolicy ...tests.)
Single E2E test that matches the name of the test specified with a regex. See bottom of this document for an example.
All locally defined tests.
The regex expression for determining which E2E test is run in which shard, as well as the list of skipped tests is defined in ovn-kubernetes/test/scripts/e2e-kind.sh. The local tests are controlled in ovn-kubernetes/test/scripts/e2e-cp.sh and the actual tests are defined in the directory ovn-kubernetes/test/e2e/.
Each of these shards can then be run in a matrix of:
HA setup (3 masters and 0 workers) and a non-HA setup (1 master and 2 workers)
Local Gateway Mode and Shared Gateway Mode. See: Enable Node-Local Services Access in Shared Gateway Mode
IPv4 Only and IPv6 Only
To reduce the explosion of tests being run in CI, the test case run are limited
exclude: statement in
Running CI Locally¶
This section describes how to run CI tests on a local deployment. This may be useful for expanding the CI test coverage or testing a private fix before creating a pull request.
Download and Build Kubernetes Components¶
Older versions of Kubernetes do not build with newer versions of Go, specifically Kubernetes v1.16.4 doesn’t build with Go version 1.13.x. If this is a version of Kubernetes that needs to be tested with, as a workaround, Go version 1.12.1 can be downloaded to a local directory and the $PATH variable updated only where kubernetes is being built.
$ go version go version go1.13.8 linux/amd64 $ mkdir -p /home/$USER/src/golang/go1-12-1/; cd /home/$USER/src/golang/go1-12-1/ $ wget https://dl.google.com/go/go1.12.1.linux-amd64.tar.gz $ tar -xzf go1.12.1.linux-amd64.tar.gz $ PATH=/home/$USER/src/golang/go1-12-1/go/bin:$GOPATH/src/k8s.io/kubernetes/_output/local/bin/linux/amd64:$PATH
Download and Build Kubernetes Components (E2E Tests, ginkgo, kubectl):¶
Determine which version of Kubernetes is currently used in CI (See
and set the environmental variable
K8S_VERSION to the same value.
K8S_VERSION=v1.17.2 git clone --single-branch --branch $K8S_VERSION https://github.com/kubernetes/kubernetes.git $GOPATH/src/k8s.io/kubernetes/ pushd $GOPATH/src/k8s.io/kubernetes/ make WHAT="test/e2e/e2e.test vendor/github.com/onsi/ginkgo/ginkgo cmd/kubectl" rm -rf .git cp _output/local/go/bin/e2e.test $GOPATH/bin/. sudo cp /usr/bin/kubectl /usr/bin/kubectl.bak sudo cp _output/local/go/bin/kubectl /usr/bin/kubectl-$K8S_VERSION sudo ln -s /usr/bin/kubectl-$K8S_VERSION /usr/bin/kubectl popd
If you have any failures during the build, verify $PATH has been updated to point to correct GO version. Also may need to change settings on some of the generated binaries. For example:
chmod +x $GOPATH/src/k8s.io/kubernetes/_output/bin/deepcopy-gen
Kubernetes in Docker (KIND) is used to deploy Kubernetes locally where a docker container is created per Kubernetes node. The CI tests run on this Kubernetes deployment. Therefore, KIND will need to be installed locally.
Instructions for installing and running KIND can be found at: https://github.com/ovn-org/ovn-kubernetes/blob/master/docs/kind.md
To run the tests locally, run a KIND deployment as described above. The E2E tests look for the kube config file in a special location, so make a copy:
cp ~/admin.conf ~/.kube/kind-config-kind
To run the desired shard, use the following (each shard can take +30 mins to run):
$ cd $GOPATH/src/github.com/ovn-org/ovn-kubernetes $ pushd test $ make shard-n-other $ make shard-np $ GITHUB_WORKSPACE=$GOPATH/src/github.com/ovn-org/ovn-kubernetes make control-plane $ popd
To run a single test instead, target the shard-test action, as follows:
$ cd $GOPATH/src/github.com/ovn-org/ovn-kubernetes $ pushd test $ make shard-test WHAT="should enforce egress policy allowing traffic to a server in a different namespace based on PodSelector and NamespaceSelector" $ popd
To skip the IPv4 only tests (in a IPv6 only deployment), pass the
KIND_IPV6_SUPPORT=true environmental variable to
$ cd $GOPATH/src/github.com/ovn-org/ovn-kubernetes $ pushd test $ KIND_IPV6_SUPPORT=true make shard-n-other $ popd
Github CI doesn´t offer IPv6 connectivity, so IPv6 only tests are always skipped. To run those tests locally, comment out the following line from ovn-kubernetes/test/scripts/e2e-kind.sh
# Github CI doesn´t offer IPv6 connectivity, so always skip IPv6 only tests. SKIPPED_TESTS=$SKIPPED_TESTS$IPV6_ONLY_TESTS