Scality provides as part as its product an NFS server and a SMB server on Linux (Ubuntu/RHEL/CentOS). To handle connection balancing and fail-over we have developed a piece of software which can either be run as a stand-alone process or used as library from an NFS / SMB server There's a few commands associated with the Virtual Server for getting status and performing management actions.
It basically assigns IP aliases (aka Virtual IP's) to machines, and reshuffles these aliases when nodes appear, leave, fail... It's written in C and we’re currently focusing mostly on the fail-over usage of this Virtual Server. It also monitors NFS instances and react when an NFS server dies. There are different policies to associate an alias to a new customer (NFS or SMB client).
We do need to extend the suite of automated tests we have for this software. This will include dealing with multiple (virtual) machines for the (NFS/SMB/...) servers and their Virtual Servers instances. We need to insure aliases move as expected from one machine to the other upon configuration changes (machine joining, leaving, process or communication failures). Hence this will include triggering such events and injecting failures, and checking the result is fine. For the admin part, we use salt to ease the management of the software on the set of machines.
The tests produced must be based on pytest, to ease further integration with our internal framework.
Details / Inputs
Scality will provide a test environment based on a set of virtual machines providing:
- a Scality storage deployment already set up and ready to use (it's basically a set of 6 virtual machines with some storage capacity. This can be ignored from the test developer stand point)
- A set of virtual machines where NFS/SMB servers will run, as well as the virtual server software. It will be up to the tests to configure and start these servers and the Virtual Server Daemon according to the tests scenarios.
- A virtual machine where the PyTest will be run.
- NFS/SMB clients will be run on the virtual machines used to provide Scality's Ring.
The Scality Virtual Server relies on a set of 3 ZooKeeper instances. These ZooKeeper instances will be running on the virtual machines provided for the Scality RING.
The existing tests scenarios will be provided as examples.
Milestone 1) Implement one additional pytest scenario:
- Provision the virtual IP addresses in ZooKeeper using the Virtual Server management tool.
- Install and start NFS servers and their companions Virtual Servers.
- Start one or more NFS client on some of the NFS servers, with some real traffic.
- Force reboot (not shutdown) of one of the machines hosting one of the NFS servers.
- Check that the status of the VIP's among surviving NFS servers is OK
- Check that NFS requests are still served correctly from the failed-over NFS server.
Milestone 2) Take the existing scenarios involving the NFS servers and adapt them to SMB.
- A working configuration with SMB servers will be provided, to avoid wasting time on Scality SMB servers configuration.
Deliver the pytest files implementing the above scenarios.
Demonstrate a successful run of these tests within the provided environments.
Code provided will be reviewed. Requested modifications resulting from the review must be applied to provided code.
Further missions / jobs will be possibly defined as a result of this mission.
Python, Salt, Tests, NFS, SMB, Network administration, Virtual Machines