Presenter: Silvio Orsini, Pietro Tiberi, and Marco Capotosto
Non-Functional Tests (NFT) are key activities that must be carried out during the implementation phase of a project.
They play a substantial role in verifying that infrastructure, application and configurations meet the non-functional requirements of the IT service.
For H24/7D service, the non-functional resilience tests are executed under operating conditions, for this reason the NFT design phase must also provide the measures to mitigate the risk of destroying the service while continuing to guarantee the effectiveness of the test.
In this paper, we present a concept on automated NFT running on a H24/7D application and focus on a couple of implemented cases—Disaster Recovery, Chaos Monkey—with the aim of building a predictive and repeatable model for operating conditions under disturbances; for instance, multiple and simultaneous hardware and software failures, high traffic conditions, unusual load profiles, and so on.
Our results support the practicality of an automated approach by showing the effectiveness and the advantages of the automation philosophy. In fact, the tests need to be repeated several times because the service infrastructure is periodically updated and changes are applied. For this reason, automation techniques for testing and producing the relevant documentation are essential. The results of the test cases showed that different executions produce comparable outputs (a comparative analysis among test executions has been carried out).
We use Ansible to build the test cases and each run has three phases. First, to check the infrastructure health to ensure all the preconditions are met. Second, execute the test itself such as rolling update, disaster recovery, HW/SW failures, high traffic conditions, etc. Third, to collect data and store it in a GitLab repo and to restore the application to the status ex-ante.
All of the scripts used for the NFT, the result and the analyses are available on a GitLab repo.
The automated approach applied on the NFT of a H24/7D application showed significant benefits in terms of understanding the infrastructure and the application, reducing human intervention and human errors, saving time and resources in test execution (that could be scheduled at any time without the human presence) and it sets best practice on the design, the execution, and the analysis of the tests themselves.
About the Presenters
Marco Capotosto has been working at Banca d’Italia since 2013 in the IT Directorate General where he works as system engineer with particular focus on the service continuity and business continuity topics. Capotosto has represented Banca d’Italia in serveral working group at Eurosystem level and has taken part in international programs. Capotosto graduated cum laude in Electronic Engineering.
Pietro Tiberi has been working at Banca d’Italia since 2013 in the IT Directorate General after 13 years of experience in the Wind Telecomunicazioni Network and ICT Department. P. Tiberi has represented Banca d’Italia in several working groups at Eurosystem level and has taken part in international programs. In particular he was one of the coordinators of the technical team supporting TARGET2, TARGET2 Securities and TIPS operations. Tiberi graduated in Electronic Engineering and Space Physics.
Silvio Orsini has been working at Banca d’Italia for around 30 years in the IT Directorate General after 7 years of experience in the ENEL ICT Department. Currently he is Deputy Head of IT Operations Directorate. S. Orsini has represented Banca d’Italia in several bodies and working groups at Eurosystem level and has taken part in international programs. In particular he was the infrastructure coordinator in the 3CB (BBk, BdF, BdI) – 4CB (BBk, BdF, BdI, BoE) cooperation for designing and implementing the TARGET2 and the TARGET2 Securities projects. At National level he was, inter alia, board member of the Computer Measurement Group ITALIA in the period 1992-1994 and board member of the Guide Share Europe in 1999-2003. Orsini graduated cum laude in Electronic Engineering and in Aerospace Engineering. He also achieved a Master in Controlling and Computing Engineering Systems. He was Teaching Assistant and Assistant Professor at the University of Rome from 1994 to 2001.