Till Fedora 22 release we have tested our Cloud images only with manual help. The amazing Fedora QA team organized test days, and also published detailed documentation on the wiki about how to test the images. People tried to help as when possible, as not having access to a Cloud was a problem for many. The images are also big in size (than any random RPM), so that also meant only people with enough bandwidth can help.
During Fedora 23 release cycle, we worked on a change for Automated Two Week Atomic release. A part of it was about having automated testing, which we enabled using Autocloud project. This automatically tests every Cloud Base and Atomic qcow2 images, and libvirt, and Virtualbox based Vagrant images.
This post will explain the state of currently activated tests for the same. These tests are written as Python3 unittest cases.
At first we run 5 non-gating tests, failing any of these will not do a release blocking, but at the end of each run we get the summary for all of these non-gating tests. These include bzip2, cpio, diffutils, Audit.
Then we go ahead in testing the basics test cases, things like journald logging, package install (only on base image), SELinux should be enforcing, and no service should fail during startup of the image.
The next test is about checking that /tmp should be world writable. We then move into testing the mount status of the /tmp filesystem. If it is a link, then also it should behave properly. Our next test checks that for Vagrant images we are using predictable naming convention of network devices, basically checking the existing of eth0.
We then disable crond service, after a reboot we make sure that the service is still in disabled state, and do the rest of service manipulations. We also check that the user journald log file exists on that reboot, this comes from a regression test. Then we reboot the instance again.
After this second reboot, we test the status of the crond service once again, and then we move to our special tests cases related to Atomic host. Means these tests will run only on Atomic hosts (both cloud, and Vagrant ones).
In the first test we check if the package docker is at all installed or not. Yes, if you guessed that this is a regression test, then you are correct. We once had an image without the docker package inside :) Next, we test the docker storage setup, that should be up, and in running state. We then run the busybox image, and see that it should be able to run properly. The atomic command is used next to start the same container. Pulling in the latest Fedora image, and running it is the next test. As our next test, we try to mount / as /host in the container. The next test is about having /bin, /sbin, and /usr mounted as read only in the Atomic host. In our final test, we check that a privileged container should be able to talk to the host docker daemon.
We also have a github wiki page explaining how to write any new test case. Feel free to ping if you want to contribute, and make Fedora flying high in the Cloud.