Blog

Ensuring Quality and Efficiency in a World of Multi-Platform Application Development

Alexander Todorov / Tuesday, November 12, 2013

     As a component vendor, we are building software that satisfies the same business requirements and runs on multiple platforms. Many of our customers would like to have a chart or a grid on both WPF and Web. What happens in reality, though, is that we usually build this component for a specific platform first - like Win Forms. Then our customers like it a lot, and start asking - is this available for Web, we have to go “web” soon? Do you also have an iOS version of it? So what we can do is ask our development teams to build it for the new platform. Our grids, for example, usually require a couple of cycles to get into market with a decent number of features. Assuming there is a team (or more) of at least 3-4 engineers actively testing the grid, this means we need to double the number of engineers in order to satisfy the new platform requirements. We've been struggling with this for a while, and we have experimented with various approaches. In this blog post, I would like to outline one solution to this problem.

   Using this solution, we have automated most of our Data Chart component, and you all know that the chart is a huge beast in terms of functionality and performance requirements. And that’s just the beginning. This blog post is about writing automated tests for software developed for multiple platforms. It’s also about achieving consistency and transparency of the required features. Having a set of business scenarios upfront, it should be obvious what the feature coverage is on every platform.  One can argue that a native application creates the best UX, and there are many big companies that follow this approach. I am not going to discuss whether going native is better than doing only web & adaptive CSS. That’s a different topic. Web apps written with responsive design in mind can actually create great user experiences. But, sadly, sometimes it’s those 5 or 10 % difference which actually makes the difference between “Great” and “wow!”.

    We all know that native development comes at a big price. You need skillful engineers for every different platform. You need to support multiple IDEs, multiple build and test infrastructures and use multiple tools for everything. There is a nice diagram, courtesy of Scott Hanselman, which depicts the effort vs. user experience when it comes to building apps with different approaches:

   

It’s interesting to see how complex the multi-platform development can be. It’s not just about the number of platforms, but also about the number of devices and the number of different screen sizes. You quickly realize how many combinations you may end up with.

    We have some obvious and easy choices when it comes to multi-platform testing. For example, we can use each platform’s testing framework to write tests. That is, Selenium Web Driver for Web or UIAutomation for WPF. But this requires separate knowledge about those frameworks. It’s also hard to maintain, and you end up with a lot of technical debt to carry on. There is no single place which can serve as a reference point about the core user stories / scenarios that the app implements. There are different IDEs, different tools, and different teams using those tools. Requirements documents may become quite different if they aren’t kept up to date. And even if they are – which is not an easy thing to do – it’s just not as transparent as you’d probably like it to be. We may end up covering only 70% of our intended functionality in iOS, and at the same time have features that we haven’t even planned for in Android, without even knowing it.

    What we could do, though - and this is the main concept I would like to convey – is to abstract the test definitions by using a more high level language. We could borrow from the concepts of Behavior-Driven Development and use Cucumber, and then write our business requirements in Gherkin format. And we could easily reuse those requirements, as well as the implementation of our scenario steps on all platforms. Then we could have platform-specific drivers that implement the core high-level user interactions on every platform. Take an “Add Task” interaction which happens in a TODO Application. The Android TODO Driver will implement the “Add Task” differently than the iOS TODO Driver. Then those drivers will need to use some automation framework, specific to the platform, to execute some core platform interactions such as clicking a button, touching elements, entering text char by char, and so on. The idea of using this approach is not to cover everything - you may still have many unit tests written for the target platform. Unit tests serve a different purpose, and that's OK. We want the highest quality, that's why combining several testing approaches is great.

When I talk about high-level interactions, I usually mean an end user “transaction” that consists of multiple platform-specific actions – and when combined, those actions perform some useful business task. You can immediately see the main benefit of this approach – we end up with the same high level codebase, and use the same development environment, bundled with the cucumber implementation for that Dev environment and platform. Hmm, looks like we could also use the same test runner, and the same Continuous integration tools -;)  In this blog I am going to show how this works using Visual Studio and SpecFlow.NET, but you can apply the same approach to any IDE running on any platform – such as Eclipse & Java.

Some of you may argue that we don’t actually need to have all features implemented on all platforms. That’s a valid argument, but I would like to mention two things:

  • It’s good to write a cucumber scenario, and have it fail on platforms where it’s not implemented. That’s a good thing. It gives you transparency and you can even use the pass/fail rate in order to get some notion of feature coverage
  • Looking at the diagram below, it’s interesting that apps which were developed a couple of years ago already have almost the same functionality on all platforms that they target. Take Facebook, for example. As an end user of mobile apps, you do want the same feature present on web, Android, and iOS, don’t you? When I saw the update which gave one the ability to add pictures as part of comments, I was like “Finally!”

For those of you who are not very familiar with Cucumber/Gherkin, here are two short definitions:

Gherkin - Business Readable, Domain Specific Language that lets you describe software’s behavior without detailing how that behavior is implemented

Cucumber - a tool for running automated acceptance tests written in a behavior-driven development (BDD) style. Cucumber is written in the Ruby programming language.[6][7] Cucumber projects are available for other platforms beyond Ruby.

Sounds pretty natural to what we are trying to achieve. Here is an example of a cucumber scenario:

Scenario: Add new task

    Given The TODO Application has loaded successfully

    And The task list is empty

    When I add a new task with the text 'go to work‘

    Then The text of the last task in the list should be 'go to work'

Those Given/When/Then statements have different purpose – Givens are used to put the system in a specific state, Whens are used to model user interactions with the system (such as adding a new task), and Thens are used to verify the result of user interactions. Those step definitions are backed up by concrete code which calls into the step implementation. The step implementations can be written in any language. If we use a .NET implementation of cucumber, such as SpecFlow.NET, we can write them in C#.

In the context of multi-platform testing, it’s very important to write good step definitions. Can you see an issue with the following scenario?

When I click the button with ID button1

Yes, there are several obvious ones:

  • We don’t have “click” on mobile, we use touch events instead
  • Button is a platform specific UI component. We don’t  need to use a button to implement parts of a UI, on a particular platform
  • Using IDs to specify components as part of the scenario is wrong – those are implementation details.

Actually, clicking a button with some ID may be part of the implementation of the “Add Task” interaction for a desktop platform like Web/WPF. And the Add Task user interaction is described by the following cucumber scenario step:

When I add a new task with the text 'go to work‘

Writing cucumber tests for features that are implemented in multiple platforms actually forces you to write good Gherkin code that can be reused.

In order to have our tests do useful and real checks on the target platform app, we also need an API to talk to the target platform app. That’s the purpose of the platform drivers – each one of them depends on a native platform automation framework, which could be:

  •  Selenium Web Driver for Web
  • UIAutomation for WPF
  • The Calabash project – to automate and interact with iOS and Android apps remotely

In my demo app I have implemented four platform drivers for a simple TODO app and those cover WPF, Web, iOS and Android. The TODO App is also implemented on those four platforms.

It’s important to note that a platform driver has two main parts – the part which implements the user interactions for our app, and that’s specific to the platform. Then there is the platform automation “Bridge”, which allows us to execute clicks, touches, keyboard entry, query for components on the visual tree, and so on. For some platforms, I am using existing technologies such as UIAutomation and Selenium Web Driver. For other platforms, like Android and iOS, I’ve written my own drivers (HTTP Clients), which talk to the native instrumented app. It’s instrumented with Calabash, and Calabash is a multi-platform technology which allows you to run user interactions remotely via HTTP. That’s not documented, though. They have a console Ruby app which sends commands with HTTP, and I had to do some packet sniffing / reverse engineering in order to understand what JSON data to send and what to expect in the response. The Calabash guys promise a C# .NET client API in the future, but I just couldn't wait to experiment with this great technology.

Below is the structure of my Visual Studio solution - I have installed SpecFlow.NET via NuGet. I have two apps in the solution, which implement the TODOApp in Web (ASP.NET MVC) and WPF, respectively. Then I have four framework projects, one for each platform, and one common project. Then I also have four test projects, one for every platform. Every test project for platform X, references the framework project for the same platform X. So, when the TODOAppPlatformDriver is referenced from framework X’s DLL, even though the implementation is specific to that platform, the test projects can share the step implementations for the cucumber scenarios because the platform driver class name is the same on all platforms.

Let’s look at the implementation for one of our scenario steps:

[When(@"I add a new task with the text '(.*)'")]
public void WhenIAddANewTaskWithTheText(string text)
{
    TodoAppPlatformDriver driver = (TodoAppPlatformDriver)FeatureContext.Current["platformDriver"];
    driver.AddNewTask(text);
}

This code, along with the .feature files is basically the same (it’s linked) for all test projects. We can add it to an arbitrary test project, and then add it as link in other projects.

Let’s look at some of the implementation of the AddNewTask method for a specific platform driver, such as Web:

public void AddNewTask(string text)
{
    // enter text in the todo_text input box
    this.driver.FindElement(By.Id("todo_text")).SendKeys(text);
    // click the Add Task button
    this.driver.FindElement(By.Id("addtask")).Click();
}

You can see that we are using Selenium WebDriver API here. When you compile the solution, since all our tests use the same test runner – MSTest, you end up with a nice Test Explorer view of all your tests, separated by platform – assuming every test project contains tests for the specific platform:

The test runner can be set in a single test project’s App.config:

<?xml version="1.0" encoding="utf-8"?>
<configuration>
  <configSections>
    <section name="specFlow" type="TechTalk.SpecFlow.Configuration.ConfigurationSectionHandler, TechTalk.SpecFlow" />
  configSections> <specFlow>  For additional details on SpecFlow configuration options see http://go.specflow.org/doc-config --> <unitTestProvider name="MsTest.2010" /> specFlow> configuration>

It’s not necessary to set it in every test project’s App.config,  because our .feature file, the generated code which backs it up – AddingTasks.feature.cs, as well as the step implementation code – AddingTasksSteps.cs, is all shared. The “AddingTasks.feature.cs” is actually generated by SpecFlow.NET’s VS design time support, and it has the correct method attributes based on the selected test runner. So the AddNewTask generated method in it will have a Microsoft.VisualStudio.TestTools.UnitTesting.TestMethodAttribute set.

The project which I have attached to this blog post contains everything you need to compile and run the tests, but you will also need the respective Android and iOS environments set up. I am attaching the XCode (iOS) as well as the Eclipse (Android) projects to this blog post. I have used the following versions of these environments:

iOS:

  • XCode 4.6.3
  • iOS 6.1

Android:

  • Eclipse ADT (Android Developer Tools) Build: v22.2.1-833290
  • Build Target – Android 4.3, API Level 18

Apart from importing those projects, you should also install Calabash-Android and Calabash-iOS:

https://github.com/calabash/calabash-android/blob/master/documentation/installation.md

https://github.com/calabash/calabash-iOS

I used latest.

When you import the project and build it & run it on the emulators, you can start the Calabash console in the following way:

Note that on Android you will need to start the test HTTP server for the target app from the console, while for iOS this isn’t necessary because it’s bundled in the app itself:

irb(main):001:0> start_test_server_in_background

nil

irb(main):002:0>

After the test server is started, you can execute queries or perform interactions directly from the console. This URL describes the query format in detail:

http://blog.lesspainful.com/2012/12/18/Android-Query/

Note that the Calabash Android and iOS Platform Drivers implement the Calabash protocol via HTTP by sending JSON in HTTP POST requests. When you run the Android and iOS tests, you can use a packet sniffer like RawCap.exe & WireShark for Windows, or HttpScoop for OS X and listen on the loopback interface and observe the packets that the test driver sends. The HTTP commands are based on the Frankly protocol, which is described here:

http://www.testingwithfrank.com/frankly.html

Additionally, the iOS version uses a number of extra endpoints for playback of interactions (/play), as well as keyboard entries (/keyboard).

Keep in mind that you cannot use WireShark on its own on Windows – you will need to use RawCap first and then open the dump in WireShark - because it doesn’t support listening to loopback interfaces. If your android environment is on a different machine, you can listen on the respective network adapter’s interface.

Here is an example of a command that performs touching a button – it is send from the Android Calabash driver to an Android app running in the emulator. I use RawCap.exe to capture the loopback traffic, since both Visual studio and my android emulator are running on the same machine. Then I open the RawCap saved dump file with WireShark and look at the HTTP POST Requests:

So you can see that the Android Calabash HTTP driver queries the button first, and then sends a touch_coordinate command by passing the center_x and center_y values as arguments. You can also see that the query command is sent to the /map endpoint, why the touch_coordinate command is sent to “/”.

You can also see that the Android HTTP instrumentation server which is running on the device is running on port 34777. This port is different for Calabash iOS, but the default one is encoded in the driver which is part of the Framework.Common project in my VS solution.

In order to change the IP/Host for the iOS and Android devices/emulators, you can modify the TodoAppPlatformDriver.cs files in the Framework.iOS and Framework.Android, respectively:

this.driver = new IOSFranklyDriver("http://10.14.1.137:37265");

The following blog posts describe in more detail the architectures of Calabash-Android and Calabash-iOS:

http://blog.lesspainful.com/2012/03/07/Calabash-iOS/

http://blog.lesspainful.com/2012/03/07/Calabash-Android/

The Calabash HTTP clients (platform protocol drivers) which I have implemented – namely, the AndroidFranklyDriver.cs and the IOSFranklyDriver.cs - have full support for querying elements, sending keyboard keys, and touching platform components like buttons.

The protocol works a bit differently for Calabash-iOS, you need to send keyboard characters to the /keyboard HTTP endpoint, and any touch events to the /play HTTP endpoint. Then if you do that, you also need an “event” value as part of your HTTP request POST data. The value of this “event” property is a base64 string which encodes the exact touch command that you are executing on the device/simulator. Here are all the currently supported commands, and their base64 strings, respectively:

https://github.com/calabash/calabash-ios/blob/master/calabash-jvm/src/resources/events/

Here is an example of a Calabash-iOS HTTP request/response to tap on a button (touch event):

URL: http://10.0.1.10:37265/play

Content-Type: application/json

Request POST data:

{

  "events": "YnBsaXN0MDCiARLVAgMEBQYHCAkOEVRUaW1lVERhdGFeV2luZG93TG9jYXRpb25YTG9jYXRpb25UVHlwZRMAADAlq7xUgk8QQAEAAAABAAEAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAABAAICAgAAAIA/AACAPwAALEMAAH5DAECKQAAAAADSCgsMDVFYUVkjQGWAAAAAAAAjQG/AAAAAAADSCgsPECNAZYAAAAAAACNAb8AAAAAAABELudUCAwQFBhMUFRgREwAAMCWuVOtDTxBABgAAAAEAAQAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAEAAgIBAAAAgD8AAIA/AAAsQwAAfkMAAAAAAAAAANIKCxYXI0BlgAAAAAAAI0BvwAAAAAAA0goLGRojQGWAAAAAAAAjQG/AAAAAAAAACAALABYAGwAgAC8AOAA9AEYAiQCOAJAAkgCbAKQAqQCyALsAvgDJANIBFQEaASMBLAExAToAAAAAAAACAQAAAAAAAAAbAAAAAAAAAAAAAAAAAAABQw==",

  "query": "button"

}

Response:

{"results":["<UIRoundedRectButton: 0x7487840; frame = (29 50; 239 44); opaque = NO; autoresize = TM+BM; layer = >"],"outcome":"SUCCESS"}

(the “query”: “button” key/value means querying for controls of type “button”)

Here is an example of a Calabash-iOS HTTP request/response to send a keystroke to an input field – after the keyboard is opened:

URL: http://10.0.1.10:37265/keyboard

Content-Type: application/json

Request POST data:

{

  "key": "y",

  "events": "YnBsaXN0MDCiARLVAgMEBQYHCAkOEVRUaW1lVERhdGFeV2luZG93TG9jYXRpb25YTG9jYXRpb25UVHlwZRMAACxOMOM7BE8QPAEAAAABAAEAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAABRAICAgAAAIA/AACAPwAAAEIAgLBDAdE4ANIKCwwNUVhRWSNAQAAAAAAAACNAdhAAAAAAANIKCw8QI0BAAAAAAAAAI0B2EAAAAAAAEQu51QIDBAUGExQVGBETAAAsTjJSfK5PEDwGAAAAAQABAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAZECAgEAAACAPwAAgD8AAABCAICwQwEAdUPSCgsWFyNAQAAAAAAAACNAdhAAAAAAANIKCxkaI0BAAAAAAAAAI0B2EAAAAAAAAAgACwAWABsAIAAvADgAPQBGAIUAigCMAI4AlwCgAKUArgC3ALoAxQDOAQ0BEgEbASQBKQEyAAAAAAAAAgEAAAAAAAAAGwAAAAAAAAAAAAAAAAAAATs="

}

Response data:

{"results":["<UIKeyboardAutomatic: 0x72d0610; frame = (0 0; 320 216); opaque = NO; layer = >"],"outcome":"SUCCESS"}

The above sends the “y” character to the keyboard in an iOS app. Visually, this will mean that the “y” character is tapped in the app, in a visible keyboard. If the keyboard is not visible, the response will have a FAILED “outcome” value.

You need to keep in mind several things when compiling and running the attached zip files:

  • All projects are placed in a single, multi-platform.zip
  •  Each folder represents a project for a different platform
  • The TODOApp folder contains the iOS XCode project
  •  I’ve deleted the calabash.framework library from the XCode project, because it’s around 27 MB. If you have troubles compiling and running the project, please refer to this guide, you can run the commands and re-initialize the project (by setting it up again), so that the lib reference is added again:

https://github.com/calabash/calabash-ios => “Fast Track”

  • Note that Calabash follows a different approach with iOS7 and XCode5. The iOS7 support is actually experimental, and I suggest waiting a bit until it’s stabilized, if you really want to use iOS7. This link talks about the differences and the issues with iOS7/Calabash:

https://github.com/calabash/calabash-ios/wiki/A0-UIAutomation---instruments-problems

  •  The TODOApp folder contains the whole Visual studio solution, which contains the apps for WPF and Web, the 4 test projects, as well as the 4 framework projects; it also contains a Framework.Common project which contains the base Frankly protocol driver for the AndroidFranklyDriver and the iOSFranklyDriver
  •  You need to run Visual Studio as an Administrator, otherwise some of the automation framework & code may fail
  • The contents of the bin and obj folders in the VS sln are emptied on purpose, because they make the zip file huge. The same applies to the contents of the NuGet packages folders, you will need to add the missing assembly references either via NuGet, or by hand
  • The Android project is in the Android folder – the actual project is a subfolder called TODOApp
  • If you have any issues running / compiling the android project, please re-initialize the calabash dependencies in the following way:

https://github.com/calabash/calabash-android/wiki/Running-Calabash-Android

Now that you’ve setup everything, you can develop and run the tests from Visual studio – on 4 different platforms!

As a conclusion, I would like to sum up the advantages of this multi-platform testing approach:

  • You achieve complete reuse of both test definitions, as well as test (step) implementations
  • You and your teams are focused on business requirements
  • The multi-platform nature forces test definitions to be good (remember the example I gave you)
  • You use one IDE and all tooling has common usage -  intellisense, autocomplete, etc.
  • You use the same test runner for all your platform tests (MSTest, in my case)
  • Because you use the same test runner, you can easily use the same Continuous Integration framework/platform

Let's see how these advantages could directly translate to ROI. If developing an automated test cost $ 50, and I mean all the costs associated with the development and maintenance, and you have 4 platforms to cover and $1,000 scenario tests in total, this means your organization needs to spend roughly $ 200,000 on those tests. Now, assume that the cost of writing and maintaining a platform driver approximates $ 0, as we are adding more and more tests for our app. Then, since we are writing once, running in  one place, and maintaining once, we need to spend just $ 50,000. Now think about all the extra goodness and reuse you get from being able to easily reuse cucumber steps for your scenarios - just because they are so isolated and self-contained ... oh yeah!

I hope you have enjoyed this content – please let me know if you have any suggestions, questions or comments! If you encounter any issues with the environment setup please let me know – it can be quite overwhelming considering the mere number of different frameworks, platforms and versions involved.

multi-platform.zip