Software Testing

Presently we are in the age of DevOps, a framework that integrates every step of the life-cycle of software development, bringing all the operations together, collaboratively, without any organizational silos, under one umbrella. A Continuous Integration and Continuous Delivery (CI/CD) pipeline empower a DevOps-driven SDLC.

Although Continuous Integration is the process of integrating each developer's developed code into the main source code, Continuous Delivery allows fully automated, low-risk, and low-cost releases to be deployed with one click. Automation thrives on a CI/CD pipeline, which is the justification for the quick deliveries and quicker feedback loops. But, test automation and continuous testing (CT) are the keys to achieving truly effective CI/CD. Although Continuous Integration and Continuous Delivery are essential for product releases to be accelerated, Continuous Testing is what brings in quality to these accelerated releases.

Continuous Testing within a CI/CD pipeline

A CI pipeline will have automated unit test cases that can be run quickly. This gives the first level of feedback to developers if their code has broken any of the unit test cases.

The CI pipeline can also contain automated integrations tests that look at API based integrations between modules of the system and/or API based integration of the system with surrounding applications. These API based integration tests may use test stubs for surrounding applications. This gives the second level of feedback to developers if their code has broken any of the integration tests.

Projects also have GUI based automated regression tests. These tests take longer to run and are typically not run as part of the CI pipeline, as the objective of CI is quick feedback to developers on the impact of their code changes. However, automated smoke tests for each build deployed to environments bring a lot of RoI by eliminating manual smoke test efforts and providing faster feedback. These smoke tests can be included in the pipeline to ensure that the system under test works as a whole. To increase the speed of these tests, they can be run in a headless browser (unless your application does not support a headless browser, or the focus of the test is on real world conditions). This gives the third level of feedback to developers if their code has broken any key flows.

The Pipeline should also integrate end-to-end (E2E) tests to be run on an integrated environment so that all the E2E tests covering both internal and external integrations are executed as part of an automated pipeline.

Each of the above levels of feedback can have intelligence built in to proceed to the next stage only if all tests have passed. This gives the developers the opportunity to identify and fix issues quickly.

Organizations use different approaches to run the entire GUI based automated regression tests. Some run them periodically as a batch (outside the CI/CD pipeline) at certain times of the day or is manually triggered. Some run them as part of the CI/CD pipeline by splitting these tests into suites which are run in parallel to decrease the time taken to run all tests and to provide feedback faster – this requires multiple hardware for running these automated tests in parallel. The need for multiple hardware can be met by dynamically creating and destroying them in the cloud from the pipeline (this approach can also be used for smoke tests mentioned above if the number of tests is high). Irrespective of the approach, the key is to run the automated tests frequently to identify & fix issues early. Again, as in the case of smoke tests, one can explore running these tests in a headless browser to increase the speed of testing.

Performance tests can also be integrated into the CI/CD pipeline. Performance tests as part of the pipeline are appropriate for applications that are not fit for purpose if they do not meet certain minimum performance thresholds. Early white box performance engineering activities like Unit Performance Profiling (Code and system) for End to End Stack can be made part of the pipeline along with unit and integrations tests. Performance Tests including Load, Stress, Endurance and Soak tests can also be integrated with Automated CI/CD Pipeline. Since performance tests will involve creating user load, the DevOps pipeline will need to set up the performance test environment, run the tests, copy the test results and then destroy the environment for optimal use of the test infrastructure. The scenarios that are covered in the performance should be functionally stable and can be run as part of the pipeline in parallel to regression test automation or post the regression test automation, based on infrastructure availability.

Security test automation is similar to automated functional tests, but targeted at verifying that security features from authentication to logout work as expected. They can be automated and integrated with the CI/CD pipeline. Specific security testing includes testing known weaknesses and mis-configurations such as lack of the http Only flag on session cookies or the use of known weak SSL suites and ciphers, etc. These are particularly well suited for automation because the weaknesses are known upfront from the threat modeling done by the security team. To successfully automate application scanning, one should ensure that all the content to be scanned is navigated and populated in the security scanning tool, before starting to spider and scan the application through the CI/CD pipeline. Post the scanning, one will have to manually analyze the detailed security test reports/logs from the tool to verify false positives (vulnerabilities).

The tests mentioned above will need test data. Typically, unit tests do not run against a real database. But smoke, regression, and performance tests will need a real test database. So, the CI/CD pipeline will need to set up the test data before the execution of these automated tests. At the end of the test, the test data should be reset to avoid dependencies between different test stages.

The Benefits and Importance of CI/CT/CD

Underpinned by automation, CI/CT/CD has become core to DevOps for delivering software in short cycles of build-configure-
deploy-test-release. The following are some of the many benefits that businesses reap from this.

• Early testing to identify and fix defects early
• Speedy innovation in the industry for a competitive advantage
• Reduced manual effort and increased efficiency of delivery
• Quicker revenue generation

How Sonata can help?

With a wealthy involvement in testing and Implementing of DevOps we assist our clients to promote a Continuous Integration (CI) / Continuous Testing (CT) framework to integrate automated software testing with deployment and operations, to improve code quality and time to market. By doing so, we accelerate the release of applications into production, improve the efficiency and utilization of resources, and streamline the CI/CD pipeline.

Testing of ETL Projects – Why is it crucial than ever?

Testing of business-critical applications that are data and process-intensive is a laborious process owing to varied factors. Processing of huge amount of data from multiple sources involves complex calculations, transformations that necessitate frequent and elaborate data loading and reloading activities.

A systematic approach towards data validation is necessary owing to high throughput, processing volumes and business sensitivity. Further, frequent and increased number of enhancements rolled out, demand additional levels of validation besides the frequent regression checks.

Accelerating Data Validation with Automation

The elaborate and time-consuming nature of data extraction, loading and validation of ETL implementations, calls for automation of repetitive activities involved.

A comprehensive validation of ETL-processes requires multiple validations at various interface points to establish if the ETL process is effective. Furthermore, ensuring data integrity requires testing of data warehouse as a whole, with multiple services between the ETL processes, the middleware, dashboard visualizations, complex interfacing logic, and messaging across the several layers.

Adopting a continuous testing approach requires automating the testing of different source and target data sets that helps rationalize testing effort when handling multiple and diverse data sources and volumes.

Leveraging the right tool for ETL Automation

While several advanced tools for test automation for various technology streams and platforms are available, tools that can automate the comprehensive testing activity of the ETL process model are limited.

The fundamental check in any ETL Testing is validating if all the data records have been loaded correctly into the data warehouse. This activity could be as simple as comparing a record count of the source and the target table or as complex as validating the data being transformed to a completely different structure through a data modelling technique. When there are multiple data source and targets, the data validation becomes more complex and crucial to the effectiveness of the implementation. 

While SQL Based ETL Testing is the initial step to ETL Testing, the very basic nature of the query-driven validation falls short when data arises from multiple sources and formats to be handled.  The queries would be limited to a single system and format that can be connected to. Furthermore the constant transformation rule changes, typical in large ETL implementations, would require constant maintenance efforts to update the queries to reflect the changes.

It is clear, that the need of the hour is a tool or a framework that would, apart from providing extensive automation support for a multi-technology and data source landscape, should be able to integrate with the test management tools available in the market.

Scenario: Data Consolidation for a Multi-Technology, Multi-application, Multi-Geo Landscape

Let us consider a scenario with one of our customers in the Travel IT space. The scenario involved consolidation of Travel Revenue Systems, of different source markets (geographies) with different instances of the backend revenue management systems (RMS). These different instances had to be merged into a single instance supporting all source markets where each source market can access the data and features relevant to them.

The backend revenue management system had to process the inventories by analyzing the sales of multiple source markets individually. It also had to provide yield adjustments to reservation system at source market level; support numerous currencies and source market data across geographies.

The challenges that transpired during data testing for an ETL implementation of this scale were:

• Virtualization, manual effort and errors, support for large dataset volumes, performance monitoring
• Support for (Extraction and Validation) different source (CSV/parquet/json) and target (Hive/presto/s3/Hdfs) systems
• Comparison of data frames
• Development, Synchronization, and Maintenance of the Test Cases & Scripts
• Integration and synchronization with CI-CD pipelines

Building Custom Frameworks: Relevance and Maintenance

Given the limitations of the SQL Query based validation of data, the python Unit test framework was evaluated and recommended for the end-to-end data validation.  The python test framework was customized to automate tests, tie in with CI-CD (Continuous Integration, Continuous Delivery) pipeline, and integrate with the qTest Tool for End to End Test Automation and Management.

This python Unit Test Framework –

•Supported data comparisons across multiple data sources (CSV, parquet, json, table)

• Supported configuration file-based connectivity to multiple data sources and targets

•Was used to connect to multiple sources using apache spark and python, supported comparison with data frames

• Supported development and execution of unit test cases, integration test cases, functional and comparison test cases

• Supported continuous execution of test cases on-demand or on the trigger of changes within the delivery pipeline

• Supported integration of automation test cases into qtest using Jenkins

• Tracked automation test cases execution daily, helped to generate reports and programmed to send to the test team and business

Customization on Python Unit Test Framework

The customization of the framework allowed extension of the automation across the spectrum of testing, while integrating the test management tools for a comprehensive test management coordination. Tests that were done with the framework are listed below:

• Test Automation, Execution & Maintenance
  • Monitored End-to-end system using with Automation test cases
  • Implemented Integration test cases, functional test cases, performance test cases and Unit test cases
  • Integrated all automated test cases into qtest using with Jenkins pipelines
  • Generated reports on a scheduled basis
  • Comparison of different data sources and targets
  • Connected to Hadoop ecosystems
  • High Frequency of Regression Test Cycles
• Support for Multiple Data Sources
  •  Support for data comparison across multiple data sources
  •  Reading from different source and target systems with multiple hops
  • Data Integration and Data Sync Verification
  • Data format conversion & loading
• Test Maintenance
  • Maintenance of test cases, re-usability of test cases, minimize manual test cases
• Test Execution & Maintenance
  • Scheduled Execution of Automated Tests
  • Auto Publishing of Results and Reports to Business Teams
  • Maintenance of the automation suite

The Test Automation function is one of the most apt solution to overcome the existing manual testing snag. Automation testing can save you both time and money.

If you want to know how we can assist you in your path to automation, reach out to us at info@sonata-software.com

We have seen a sea of change in the way software development projects are approached, however, the underlying factors defining a successful project still remains the same, as the delivery teams continuously jostle to get the right balance among time, cost and quality. Quality assurance is an integral part of any successful software application development process. By having a robust software testing mechanism in place, enterprises ensure delivery of successful software projects.

Software testing since its formal inception in the 1980s has evolved. From being a manual and human resources intensive process it has moved towards automation, with technology helping QA engineers automate a bulk of the repetitive tasks.

As per the “State of Software Testing Report 2017”, by PractiTest, software companies are adopting DevOps in a more holistic manner, and QA engineers are now seeing more involvement in CI& CD process, playing a larger part in the software development cycle. With more and more companies adopting DevOps & agile methodology, involving continuous software development, leaving testing to the end is no longer an option.

Two factors, which determine the success in the continuous delivery model, DevOps, are

• Quality of testing – ensuring the highest level of testing is of prime importance to DevOps.
• Speed of development and testing – to ensure software release pipeline is not stalled, speed of testing should match the speed of software development. Any delay in testing will slow down the release of new development.

This change in approach requires continuous testing process, which ensures seamless code change in the software release and quality at speed, throughout the software development cycle.

However, managing continuous testing manually is a challenge as it presents key issues to address

• Siloed automation
• A lack of end to end visibility of requirements
• The high volume of test

As per industry reports in 2018, 90% of software testing is performed manually and it consumes a large chunk of the software development cycle. Manual continuous testing slows down the release of new development and thereby preventing the realization of DevOps. To address this challenge, the industry needs to move towards autonomous testing, which ensures zero-touch quality assurance.

Autonomous Testing

In the era of digital disruption, autonomous testing, made possible by the application of digital technologies such as AI (Artificial Intelligence), ML (Machine Learning), and NLP (Natural Language Processing), is becoming mainstream.

Analysts predict that Automation Testing Market will be worth $19.27 Billion by 2023. Autonomous testing has the potential to revolutionize the testing as we know today.

It has enhanced the speed of automation. With autonomous testing, one can automatically create test scripts. It makes creation and maintenance of test scripts easier. It helps in reducing the cost of operation by bringing down the number of test automation & manual testers.

Benefits of Autonomous Testing:

• Significant reduction in time to automate: Autonomous testing helps automate large test suite faster, resulting in the increase of delivery speed, which in turn provides more time to the QA team for extensive test coverages. Autonomous testing helps in early identification of defects in the software development process
• Saves cost on building skills: Autonomous testing eliminates the need for coding and expertise on the test automation tool. Test scripts are generated from the test cases written by domain experts in plain English. This is a huge cost saver as companies invest a significant amount of time and money either to train new resources or hire costly resources, who are familiar with test automation skills.
• Easier maintenance of test scripts created from plain English test suites.
• Enables participation and collaboration of business analysts and subject matter experts, who may not have technical expertise, in the autonomous testing model.

Advancement in technologies is helping businesses transform themselves digitally. Autonomous testing, helps enterprises accelerate their digital transformation journey with

• Improved business and delivery agility
• Quicker go to market
• Reduced total cost of ownership & cost of operations
• Better ROI
• Increased throughput

To know more about our Testing Platforms and Solutions click here.