Thus far we’ve made it pretty clear we have devoted a big part of our process to test automation  for both Ignite UI and in general. We strive to make the most out of it by incorporating Behavior-driven development methodology and tools like Selenium 2.0 (WebDrivers) and SpecFlow (Cucumber for .NET). While the latter can solve some of the common challenges, there’s still some we haven’t touched yet – the creation of test beds (which in this context boil down to the actual pages under test). Even though the previous post was more of an overview of some key tools and methods, I’m hoping it gave you an idea how the test process goes and I will add some more details now to fill some missing pieces. Should come as no surprise that at this stage we have a growing collection of Cucumber step definitions, all of them bound to implementations using Selenium WebDriver. It only made sense to take it further by adding additional features and utility and centralize things a bit and so we did!

 

Ignite UI BDD Test Framework

As a source of many useful step definitions and other utility the framework’s main benefits are the ease of use, the fact that it lets us write tests quickly and most importantly reuse a large amount of the test logic.

Far from being a simple repository for step definitions, the framework also provides the ability to generate test beds directly based on the scenario or background’s “Given” steps and tags. The framework uses SpecFlow’s methods for sharing data (like context injection and scenario context) mechanism to inspect the context and assert scenario tags for meaningful ones to the framework itself. Remember the snippet with Gherkin? The one that contains a “@generate-testpage” tag? Can you guess what it’s for? Yes, that would be the tell-tell sign to create new test bed. They are generated based on predefined templates (also part of the framework) and a model of the Ignite UI widgets you have defined in your Given steps, any features/options you specified and so on. The framework uses various hooks to check tags and the steps themselves to progressively build what you can call a model of the control(s) involved in the scenario. The framework is capable of both HTML pages and full blown ASP.NET MVC test bed generation! Moreover, it manages browser instances, various other test settings, shared widget information though SpecFlows context injection mechanism and pretty much any other utility we deemed necessary for the execution of our BDD tests.

A simple representation of the whole process:

Additional WINs for us from using this framework are that the feature files can be the actual test plan itself, the tags add nice semantics to how a scenario is run (which template, data source, category) and automating new features is greatly simplified in the process. Moreover, for the top level intended for everyone, the framework abstracts away the entire technology standing behind test execution – if it’s Selenium running that step, is it MSTest or NUnit based, what browser will it run with – you get steps that clearly state ‘what’ they do without bothering you with the ‘how’.

But wait, there’s more!

Unit tests and headless... awesomeness

You can never have enough, right? While the interaction side is covered by Selenium, API and functionality is, as always, verified by Unit Tests.  And what better tool than QUnit? But with our established process practices, besides keeping it easy for developers to write and execute those locally, we want integration with everything else.

In comes the quite awesome Awesomium – It’s as they describe it a “Web UI bridge for native applications”. Besides the many interesting additional things it can do, it is a windowless WebKit that is ideal for executing headless test – such that only run on a layout engine and JavaScript engine. As you might be guessing it does have a very nice .NET API that allows for our QUnit tests to be nicely integrated with builds and also requires no browser windows to be open. That gives us a lean, fast suite even developers can run with their local changes very easily. Oh, and it’s a very mature (read: advanced and stable) project that we found is faster than alternatives such as PhantomJS.

Execution

So, if you remember, every CI build executes the BDD Cucumber tests. Now add to that a task that runs our project incorporating all the QUnit unit tests through Awesomium for both build types. What you get is a constant process of ensuring quality and monitoring for regression. How do you imagine we run all this (considering CI builds are constantly rolling)? Well it does take some muscle – both from hardware and our Engineering Services team :) - and Team Foundation Server and Microsoft Test Manager deliver both on utility, flexibility and integration. While it would be all too much to into greater detail I could point you to this article about Automated Tests.

We are also using System Center Virtual Machine Manager (SCVMM) environment that allows for easy management of multiple virtual machines to perform the roles of controllers and agents. What’s also great about this is that the controller will assign separate tests to run on different agents achieving parallel execution of the suite (that’s why I liked the separate tests SpecFlow generates for scenario outlines so much!).

Sharing behaviours between platforms

A.k.a the Infragistics Cross-Platform BDD Framework. Since the Ignite UI one separates underlying implementation from actual scenario definition (or connects them, however you want to look at it) it wasn’t far from thought to consider such approach for other platforms and even more so a special case of controls – Infragistics’ Data Visualization controls. I mean we have been translating our powerful XAML controls to jQuery for a while now and the range on controls that are shared keeps on growing. Take the Data Chart or Radial Gauge for example – they are essentially the same control so they should have identical behaviour down to smaller details. Therefore, it makes perfect sense for such controls to actually share test steps and some of them do! Guess what other platform also have the very same controls? Windows UI and iOS! And, yes, that means a separate cross-platform framework targeted at our translated controls. While there are some preprocessor directives involved, the main part is introducing a whole additional layer of abstraction where the framework doesn’t really contain the step definitions, but actually sits between them and the actual platform-specific implementations. To further explain this – the framework actually presents a number of assemblies (for each platform), each providing identical utility classes – such that manage control properties, data, SpecFlow argument transformations and so on. That way you get a unified API for step definitions to use, write once and share those very same steps (quite literally by linking the files) across platform test projects and merely assign the respective assembly with helpers and managers:

Naturally, it’s not that simple and there are always platform quirks to iron out, but overall it actually is turning out pretty great!

Closing

I’m sure some insight (with a promise for more) got your attention! There’s still an announcement to make that should happen fairly soon so subscribe, follow and all that good stuff to stay in the loop. Useful links are below:

• The first blog - Infragistics Test Automation: Ignite UI with info on Selenium, BDD and SpecFlow.
• Application Lifecycle Management with Visual Studio and Team Foundation Server
• QUnit: A JavaScript Unit Testing framework.
• Awesomium
• Come see us at ISTA Conference 
• Try out the Data Visualization brilliance of  Ignite UI and then get yours:
  

I’d love to hear some thoughts, so leave a comment down below or @DamyanPetev.

And as always, you can follow us on Twitter @Infragistics and stay in touch on Facebook, Google+ and LinkedIn!