Skip to content

Case study: One project QA approach

June 1, 2011


In this post I’d like to describe QA approach for one project I participate. The post will describe techniques for testing high availability  and reliability for high scalable applications. There will be some examples to show how to apply this techniques.

Project overview

The main goal of the project was to build a core framework for deployment of Windows-specific applications in the cloud. So it was implemented for Amazon EC2 cloud.

On high level the architecture is according to the scheme (I’m not allowed to publish real architecture).

So there is monitoring/management subsystem (actually cluster in cloud) that controls all work flow of Agents (many nodes in cluster). Also there is Web tier for managing and status check. All communication between nodes are implemented using Rabbit MQ API.

Test phases

All project testing was split into 3 major parts so during these phases I need to write all test documentation, implement automated tests for system high availability and reliability using C# and Power Shell and also automate build and test process via continuous integration system.

Phase I.  Component level

On this phase there were a lot of investigations, mocking and so on. So I wrote documentation according to functional design.

As system was not implemented yet but there was communication protocol I wrote subsystems emulators using Rabbit MQ. After emulators was done I implemented test for it (some kind TDD).  All tests were done as unit tests in MS VS 2010.

After this huge part of coding I started investigation about Web UI test tool. The major criteria were AJAX support and project compatibility. Below is table describing this search:

Languages Project
Watir No Ruby n/a n/a
elerity Yes Ruby Partly Popup windows troubles. Actually Watir extension for javascript support
Selenium IDE + RC Yes C#, Java, PHP, Ruby Yes Can be integrated into the MS VS 2010

The choice was obvious – Selenium HQ IDE + RC. It was immediately integrated on out build servers under Cruise Control .NET. All Web UI test were done using Selenium API.

Phase II. “Complete” system

Some times past and developers finished core system and I started to move tests from mocks and emulators to real system.  And started system reliability and HA testing. Below there are sample approach I use for testing:
  • Create CS using Amazon API
    • Verify CS created
  • Launch ECM instance using system API
    • Verify ECM instance created
  • Reboot (N times) Controller Server machine
  • Ensure ECM instance changed status to “running”
  • Terminate created instances
    • Verify instances terminated
So the aim was to terminate subsystems for a while and ensure that system will recover all information without data loss.
High availability:
  • Launch CS(A) using Amazon API
    • Verify CS(A) started properly
  • Create ECM instance using system API
    • Verify ECM instance created
  • Create CS(B) using system API
    • Verify CS(B) started properly
  • Terminate CS(A) using Amazon API
    • Verify CS(A) terminated
    • Verify CS(B) recovered additional CS instead of CS(A)
    • Verify ECM instance changed status to “running”
CS cluster form some kind of cycle where one CS watch for another (like in closed linked list). The goal was to see that one CS can restore others. This test can be modified for any amount of CS and ECM instances.
CS – controller server that contains Management/Monitoring subsystems;
ECM – calculation node;
“running” – internal ECM instance state (e.g. staring, provisioning, running, deleted);

Phase III. Integration and CI

As CI we choose Cruise Control .NET. And implemented all preparation/testing/building processes using its tasks.

The work flow was the following:

I mentioned in my blog earlier about IIS Web Deply Tool and in more details about some of it.


This was short overview about applying QA for the project. Where I needed to implement tests for HA, reliability, automate testing and deploy.

From → Automation

Leave a Comment

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: