Pattern for running multiple Cucumber projects on CI server

Background:

One of the most common folder structures of Cucumber projects is as below

Cucumber project stucture

 

However, often while working with large companies, the projects consist of multiple teams and each team would like to work on their own cucumber projects while “in-sprint”, however at some point they would like to integrate and run all the acceptance-tests (automation scripts) projects on CI server.

As an example, there could be multiple projects each having the structure as above.

How do we ensure that individual teams continue with their work as usual adding code, at the same time have flexibility to run projects (by picking and choosing) at run time. One of the solutions is to have a top level Rakefile where we define tasks to be able to execute features by looping through the projects. The parameters generally for such a task would be:

1.  tag                 #Cucumber tag to run eg. @smoke tests, @slow tests, @fast tests and so on.
2. target         # where do you want to run the tests ? locally OR on a Selenium grid
3. environment # which environment to run eg. integration, sandbox, qa , pre-production etc
4. browser   # choice of browser viz. chrome, firefox, ie and so on
5. project(s)  # select which project features to run

At the top folder the following commands will achieve what we want

1. rake smoke[local,int,chrome] => runs the features with @smoke tag
2. rake smoke[local,int,chrome,”appium_page_object|calabash_keyword_driven “]=> runs the features with @smoke tag on local, on int environment and on chrome browser on the projects “appium_page_object” and “calabash_keyword_driven”

Rakefile Example:

require 'rubygems'
require 'cucumber'
require 'cucumber/rake/task'
require 'rake'

# rake smoke[local,int,chrome] => runs the features with @smoke tag # on local , on int environment and on chrome browser

# rake #smoke[local,int,chrome,"appium_page_object|calabash_keyword_dri#ven "] => runs the features with @smoke tag on local, on int #environment and on chrome browser
# on the projects "appium_page_object" and #"calabash_keyword_driven"

# I have three projects each running appium, Calabash and Frank.

@projects = %w[appium_page_object calabash_keyword_driven Frank_data_driven]

def taskbuilder

@projects.each do |project_name|
Cucumber::Rake::Task.new(project_name.to_sym) do |t|
t.cucumber_opts = "TARGET=#{ENV['target']} BROWSER=#{ENV['browser']} --tags #{ENV['tags']} "
t.profile = ENV['env']

end
end

end

 

# The task below runs all features with @smoke tag

desc "Run all features with the @smoke tag"
task :smoke, [:target, :env, :browser, :projects] do |t, args|
execute_tests('@smoke',args[:target],args[:env],args[:browser], args[:projects])
end

 

# This method builds and runs the rake tasks with specified tag, #target, environment and browser

def execute_tests(tag,target,env,browser,projects={})

ENV['tags'] = tag
ENV['target'] = target
ENV['env'] = env
ENV['browser'] = browser

results = []

#If projects are specified , then the global array @projects is reset #with the values supplied by user, otherwise it remains

if(not projects.nil? and not projects.empty?)
@projects.clear
@projects = projects.split "|"
end

# Build the cucumber tasks
taskbuilder

@projects.each do |project_name|
Dir.chdir(project_name)
results << collate_results(project_name)
Dir.chdir(File.expand_path(File.dirname(__FILE__)))
end
is_failure results
end

# This method reports any failures

def is_failure(results)
results.each do |result|

unless result
puts "Scripts had failures"
raise BuildFailure
end
end
end
# This method invokes the cucumber rake task for the specified #project_name

def collate_results(project_name)
begin
Rake::Task[project_name].invoke
rescue Exception => e
return false
end
true
end

 

Name(required)

Email(required)

Website

Comment(required)

Submit

Δ

Mobile web browser Automation

Background: There is often a need in many companies to test web applications on mobile web browser for eg. launch safari browser on an iphone, hit the url and test the application (It might be for functional testing or Responsive web design testing). This article talks about function/automation testing on mobile web browser and how Selenium + Appium lets us achieve the same by being able to re-use the selenium scripts written for browsers on a PC/MAC et al.

Instant Karma: Think of Appium as a Selenium GRID on the mobile test automation side. Selenium GRID works using the principle of JSON wire protocol, where the selenium scripts send GET/POST requests to the Selenium GRID (hub and nodes) and those get converted into commands through webdrivers that ultimately do the set,get, click operations and so on. Just like how hub and node needs to be started to be able to use selenium GRID, on the appium side, you will just need to start the Appium server (On Mac, just download the appium.dmg file , extract and run it — similar on the windows side too)

Step1: Run Appium server and lets say it is listening on 127.0.0.1:4724

Appium 1.0

Step2:

Copy the below code into Eclipse or any IDE and run it as Java Application. When run on Mac, you should see the iOS/iphone/ipad simulator pop up, open safari browser and hit the url (Assuming that you have Xcode, command line tools installed)

Java Code:

package test
import java.net.MalformedURLException;
import java.net.URL;
import org.openqa.selenium.By;
import org.openqa.selenium.WebElement;
import org.openqa.selenium.remote.DesiredCapabilities;
import org.openqa.selenium.remote.RemoteWebDriver;
import org.testng.Assert;
public class MobileWebTest {
public static void main(String[] args) throws MalformedURLException {
DesiredCapabilities desiredCapabilities = new DesiredCapabilities();
desiredCapabilities.setCapability("browserName", "Safari");
URL url = new URL("http://127.0.0.1:4724/wd/hub");
RemoteWebDriver remoteWebDriver = new RemoteWebDriver(url, desiredCapabilities);
// Navigate to the page print html source
remoteWebDriver.get("http://www.google.com");
System.out.println(remoteWebDriver.getPageSource());
//close the app.
remoteWebDriver.quit();
}
}

Eclipse Java Code

Ruby Code (should have selenium-webdriver gem installed):

## test.rb file. Run it as "ruby test.rb" or run from an IDE like ##RubyMine/Eclipse

require 'selenium-webdriver'

caps = Selenium::WebDriver::Remote::Capabilities.safari
driver = Selenium::WebDriver.for(:remote, :url => "http://127.0.0.1:4724/wd/hub",:desired_capabilities => caps)

driver.get('http://www.google.com')

puts driver.page_source

driver.quit

RubyMineIDE

Closing Thoughts:

I had a large framework built for a company that uses Selenium Grid for parallel/cross browser testing that uses cucumber, ruby. When the company decided to go mobile and started investing in mobile apps, the first effort was to launch the same application in a mobile browser. All that I had to change was the above start up code in hooks.rb file [If you know the standard layout of a cucumber project, the driver is instantiated in hooks.rb].

For Java based projects, if using TestNG, you would have to change the code where driver is instantiated as per above [@before annotation in TestNG]

Note: The same works on Windows with Android too. Just that the AVD and Android SDK needs to be installed correctly and started

Name(required)

Email(required)

Website

Comment(required)

Submit

Δ

Cucumber Tags

Introduction

Tags are a great way to organise your features and scenarios. Consider this example:

@featuretag

Feature: Verify billing

@ScenarioTag

Scenario: Missing product description

Scenario: Several products

A Scenario or feature can have as many tags as you like. Just separate them with spaces:

@feature1 @scenario1 @release1

Feature: Verify billing

 

Tag Inheritance

Any tag that exists on a Feature will be inherited by Scenario, Scenario Outline or Examples

 

Running a subset of scenarios

You can use the –tags option to tell Cucumber that you only want to run features or scenarios that have (or don’t have) certain tags. Examples:

cucumber –tags @featuretag            # Runs both scenarios

cucumber –tags @Scenariotag          # Runs the first scenario

cucumber –tags ~@Scenariotag         # Runs the second scenario (Scenarios without @Scenariotag)

cucumber –tags @featuretag –tags @Scenariotag    # Runs the first scenario (Scenarios with @featuretag AND @Scenariotag)

cucumber –tags @featuretag,@Scenariotag          # Runs both scenarios (Scenarios with @featuretag OR @Scenariotag)

Another way to “filter” what you want to run is to use the file.feature:line pattern or the –scenario option

Logically ANDing and ORing Tags

As you may have seen in the previous examples Cucumber allows you to use logical ANDs and ORs to help gain greater control of what features to run.

Tags which are comma separated are ORed:

Example: Running scenarios which match @featuretag OR @Scenariotag

cucumber –tags @featuretag,@Scenariotag

Tags which are passed in separate —tags are ANDed

Example: Running scenarios which match @featuretag AND @Scenariotag

cucumber –tags @featuretag –tags @Scenariotag

Reference:https://github.com/cucumber/cucumber/wiki/Tags

Name(required)

Email(required)

Website

Comment(required)

Submit

Δ

How to Choose a Test Automation Solution

Background:
Often many of us hear lot of jargon, words, language around Test Automation that confuses us especially when we have to make a decision to choose a Test Automation Solution. Some of the jargon as I heard from many clients: Selenium, Cucumber, Ruby, Java, QTP, vbscript, gherkin, test-driven, data-driven, page-object and so on.

• How do we put a structure around the jargon and how do NOT end up comparing apples and oranges?
• How do I as a director make a case for budgeting a Test Automation solution?
• How much money do I put in and what is the justification for the ROI?
• What is the skill set of people I need to look for for staffing up on a project like this?
• What is the process and what are the technologies? (On a funny note, I used to detest people who emphasized process importance at one time, until I had to lead a department and realized how critical processes are, sometimes more critical for a trade-off decision than technology)
• How do I measure the progress (aka. what metrics) and pivot and reset the direction if diverging?

Few Thoughts:
Test Automation skill sets are broadly measured across programming languages, across libraries and across frameworks and across software development models.

• Programming languages can be Java, Ruby, C#, groovy, vbscript, php and so on.
• Some of the libraries can be Selenium, Cucumber, testNG, Watij, Apachi-poi and so on.
• Frameworks can be Data-Driven, Keyword-driven, Page-object, Hybrid and so on
• Development models can be waterfall, spiral, Agile, TDD, BDD, ATDD and so on

As with software development in general, Test Automation IS A DEVELOPMENT EFFORT. By that I mean, test Automation is writing code to accelerate the feedback loop (build-measure-learn). For some of us who come from waterfall, spiral models of working, it is to say that Test Automation helps us in Regression testing and save time and money.

What is TDD/ATDD?

I do not want to get into the differences at this point, but by and large both of them have many similarities and for this blog and Test Automation in general, both the models are similar.

ATDD or acceptance-test driven development is a model of software development that brings the 3 amigos aka. developer, tester and business analyst collaborate on acceptance criteria constantly and consistently until the software is delivered. Typically starts with defining acceptance-criteria (how software should behave and use cases) and using the acceptance-criteria as a single source of truth by the tester and developer to advance in their day-day work.

Combine this working model with Agile and Continuous Integration/Continuous Deployment/Continuous Execution. That is when the power of ATDD really shows up. Agile teams working in Sprints, team collaborates as often as possible on where each team is on the tasks and have daily-stand ups , sprint reviews, retrospects and pivot to adapt to business changes.

Continuous Integration/ Continuous Deployment is the concept of integrating and deploying code as often as possible so that failures are detected as early as possible. Continuous Execution is to be able to run acceptance-tests (or test cases) with the same velocity as continuous integration and deployment happens.

The power of Test Automation really shows up during Continuous Execution in sync with CI/CD.

In an Agile environment that follows CI/CD/CE, I cannot stress more that Test Automation is as quintessential as heart-beat for a human body.

There is often a misconception in teams when they first adopt ATDD, that this is a new working model only for testers — It is NOT. ATDD is a practice that needs to be adopted and followed by everyone on an Agile team. A typical use case of ATDD is as follows:

1. Business Analyst writes acceptance-tests from a high level perspective
2. Testers expand on the acceptance-tests in that they cover all possible scenarios and boundary conditions
3. Acceptance-tests should have automation code/scripts associated to run the scenarios
4. Before the source code is developed, the automation scripts fail
5. Developers run the acceptance-tests and it fails the first time as in Step 4)
6. Developers have to write source code to make the acceptance-tests pass (this will be in iterations and testers keep on adding more acceptance-tests and automation code)
7. End of every sprint (generally 2 weeks), we assess how many acceptance-tests have passed – that measures how close are we to the delivery. We also assess end of every sprint if we need to pivot as per changing business needs

As you can see the steps 1 through 7 happen every sprint and the cycle Build-measure-learn is a very powerful concept.

What about defects?

Since the feedback loop aka. build-measure-learn happens as often as possible, the defects or bugs are fixed as early as possible ultimately resulting in huge cost savings in time and money.

How to choose Test Automation Solution: So coming back to our discussion on how do we choose a Test Automation Solution given the landscape of so many moving parts and changes at organizational level? It is NOT easy, however taking baby steps and following some of the practices which many start-ups and innovative companies have done helps. Think about how nimble these companies and if that brings a glean in your eye, keep reading. There is no one size fits all, however asking the right questions helps align with the overall goals and directions. People: Do I have the right skill sets in the team members ? What kind of training do they need ? Do I bring in external consultants to get started? Or do I train change agents within the company ? Do I have the buy-in of the team members to experiment on new practices ? How do I NOT disrupt the existing operations? Processes: One implementation of Agile believes in decentralization and every team members answers to a team, not necessarily to one person on the team. Of course that also requires a change in mindset of all the stake holders, since there is relinquish of control and more transparency, responsibility and accountability power shifts. Every person team should feel responsible. So what happens if something doesn’t get fixed ? Does it result in the circular discussions of blames ? How do we build a culture where all team members think positive and work together to address issues and not point fingers ? What processes should exist if you are a financial company like auditing, compliance etc ? How do we build a quality driven culture and that ‘Quality’ is the responsibility of everyone on the team and NOT just testers ? What are the dependencies on technologies and does processes align with those technologies ? Technologies: What softwares/tools do I choose for Test Automation? How to resolve the transitive and n-level dependencies of the choice of each tool over others ? How do I decide which programming language, which libraries and which Test Automation Frameworks ? Can I choose technologies that most of the team members (aka developers) are familiar with, so that pair programming becomes a norm and developers help testers ? On maintenance projects, since these decisions have already been made, can I choose upcoming and relatively much simpler/readable coding languages like Ruby? What Gap-Analysis and attributes need to be taken into account while choosing languages, libraries and automation frameworks.

Closing Thoughts:

It really depends on each situation and context to decide and approach and choosing a Test Automation Solution, however I have observed over years that Test Automation is a development project, meaning the more closer a tester gets to becoming a developer, the more ROI is realized from Test Automation. That does NOT mean, there are NO solutions for patrons who want to take baby steps without being comfortable with programming language.That is where “Test Automation Framework” plays an extremely critical role in alleviating the concerns and challenges of team members. As cliched as it may sounds, the context, company, domain, industry determines if we have to choose a top-down or bottom-up or both approaches when designing, developing and rolling out Test Automation Frameworks. In my next post, I will expand on the attributes I have used to build Test Automation frameworks and that immensely helped mid-sized and large clients to become effective and efficient as teams and organization.

 

Name(required)

Email(required)

Website

Comment(required)

Submit

Δ