Kubernetes storage validation by Ansible test automation framework

Ansible is mainly used for software provisioning, configuration management, and application-deployment tool. We have used it for developing test automation framework to validate Kubernetes storage. How we used ansible as a test automation tool to validate Kubernetes storage is explained in this post.

Why we used ansible?

Kubernetes is a clustered environment where we will have 2 or more worker nodes and one or more master node. We have to create CSI driver volumes in it to validate our storage box.

So, the test environment will consist of multiple hosts. The volume may be mount on any of the pod created in any of the worker nodes. So dynamically, we need to validate any of the worker nodes. If we use any programming/scripting languages, then we need to handle remote code execution. We worked in a couple of automation projects using PowerShell and python. But remote code execution library needs a lot of work. But in ansible, the heavy lifting of remote execution is taken care of by itself. So, we can only concentrate only on core test validation logic

How ansible is used?

As part of the Kubernetes storage validation, there are many features to be validated.
Features such as Volume group, Snapshot group, Volume mutator, Volume resize need to be validated. Each feature will have many test cases.
For each feature, we created a role. Each test is covered in tasks file under role.
In main.yml in roles will call all the test tasks file.

Structure of ansible automation framework roles

	roles
	Feature_test
	  volumegroup_provision
        Tasks
          Test1.yml
          Test2.yml
          Main.yml
	  volumesnapshot_provision
	  volume_resize
	  basic_volume_workflow
	Lib
	  resources
	    (library files sc,pvc,pod and IO inside Pod)
volgroup_play.yml
volsnaphost_play.yml
volresize_play.yml
basic_volume_play.yml
Hosts

In the above framework, test1.yml and test2.yml are tasks file where test cases would be written.
Each feature will have its own play file—for example, Volgroup_play.yml. So if we execute volgroup_play.yml, then tests reside in test1.yml and test2.yml will be executed.

Below command will execute the play

ansible-playbook -I hosts volgroup_play.yml -vv

Challenges:

Problem:

In ansible, if a task is failed, then execution will be stopped. So, if 10 test cases are there in a feature, and if a second test is failed, then remaining 8 test cases will not be executed.

Solution:

Each test case is written inside block and rescue. So, when testing is failed, it will be handled by a rescue block. In the rescue block, we will clean up the testbed so that the next test case will be executed without any issues.

Sample test file.

	- Block:
  
 - include: test_header 
  vars:
Test_file: ‘test1.yml’
Test_description: ‘volume group provision basic workflow’

  < creation of SC,PVC and POD and validation logic>
  <After validation ,do cleanup>
  
 - include: test_footer
  vars:
Test_file: “test1.yml”
Test_description: ‘volume group provision basic workflow’
Test_result: “Pass”

 rescue:
  - include: test_footer 
   vars:
Test_file: “test1.yml”
Test_description: ‘volume group provision basic workflow’
Test_result: “Fail”

  < Cleanup logic> 

Problem:

Some of the tasks which can be done easier in a programming language are tough in ansible.

Solution:

Write custom ansible module using python.

Pros of using ansible as automation framework:

1. Ansible is very simple to implement.
2. It takes care of heavy lifting of remote code execution
3. For clustered environment, speed of automation development is considerably higher.

Cons:

1. Though it is simple, still ansible is not programming language. When straightforward commands are written, it will be easier. but when we write logic, few lines of programming language will do what 100 lines of ansible does.
2. When multiple tasks need to be executed in nested loop passion, it will be very hard to implement that in ansible. (we have to use ‘include’ module with loops then again use ‘include’ modules. It is not very intuitive)

Conclusion:

Ansible can be used as a test automation framework for Kubernetes storage validation. Wherever heavy programming logic is required , it is better to use custom ansible module using python which will make life easier.