Archive for the ‘QA’ Category

SOA Testing, Emergent Behavior and Ants?

In order to understand the importance of the link between SOA and software testing, one needs to understand what makes an SOA environment so different from other architectures.

SOA uses services as the primary means of getting things done through the delivery of SOAP object (aka XML documents) over an HTTP protocol; but so what? Why is testing in this type of environment any different than testing in a 2 tier, 3 tier, N tier, or peer-to-peer distributed architecture? The answer can found in understanding that in service-oriented architectures independent services can discover and interact with other services, across disparate computing spaces, not contemplated by the originating developer; producing non-deterministic behavior, otherwise known as Emergent Behavior.

Emergent Behaviors is a self-organizing principle of growing importance in large distributed systems. In these new systems, central control and management is exceedingly difficult and nearly impossible to achieve. This is akin to an ant colony in that each ant is relatively primitive but when acting together they can collectively regulate (control and manage) the temperature of a hive to within 1 degree C. The best digital household thermostat can not do that today! That “programming” can not be found in the individual “service” constructs of each ant. It is emergent from the interaction of the collective as a whole.

Because of similar types of interactions betweens services, our next generation SOA-based systems will experience similar non-deterministic emergent behavior. Some computer scientists and futurists believe this will evolve to a point where system evolution will not only occur through the hands of the developer, but also through these internal self-organizing principles of the system itself.

Think this is Sci-Fi, think again! A study of software related trouble reports in just 10 SOA-based enterprise architectures revealed that some of the root causes for reported problems was not just with the unit level software, but as a result of the unintended expression of behavior via two or more services that were never thought to have interacted in the original architectural design (aka emergent behavior). Individual services found other services thought useful in the pursuit of their solution, only to have the enterprise (collective) behavior in a manor inconsistent with the end user expectations.

Traditionally testing the interaction between entities has it limits in service-based distributed system. Think about it for a moment. Let’s say we want to thoroughly test the interaction between 3 entities, each one being able to do two things. How many different tests would we have to perform? That’s right, 6 (2*2*2 = 2^3). Now, let’s say the system was composed of 100 such entities; how many tests would you need to complete validate the system? Excellent, you are right again; we would need 1.26×10^30 tests (a lot). The point is that we would not have enough time or resources to perform such exhaustive testing. So what do we do?

The next generation of distributed service-oriented architecture-based systems will be described as “statistically behaving to within 99% of it nominal baseline.” Easy for you to say; right? What this means is that we, as a community, will move towards statistical based tests built on experimental designs used to give us assurances that the systems is operating as expected. It is the same way a doctor examines you during a check up. The human body is so complex that it is impractical to exhaustively evaluate everything, so the doctor tests a limited set to seeing how you are performing and then declares your healthiness based on that. Statistical-based software testing for SOA-distributed systems will eventually evolve to the same place.


Read Full Post »