Background:
Often many of us hear lot of jargon, words, language around Test Automation that confuses us especially when we have to make a decision to choose a Test Automation Solution. Some of the jargon as I heard from many clients: Selenium, Cucumber, Ruby, Java, QTP, vbscript, gherkin, test-driven, data-driven, page-object and so on.
- How do we put a structure around the jargon and how do NOT end up comparing apples and oranges?
- How do I as a director make a case for budgeting a Test Automation solution?
- How much money do I put in and what is the justification for the ROI?
- What is the skill set of people I need to look for for staffing up on a project like this?
- What is the process and what are the technologies? (On a funny note, I used to detest people who emphasized process importance at one time, until I had to lead a department and realized how critical processes are, sometimes more critical for a trade-off decision than technology)
- How do I measure the progress (aka. what metrics) and pivot and reset the direction if diverging?
Few Thoughts:
Test Automation skill sets are broadly measured across programming languages, across libraries and across frameworks and across software development models.
- Programming languages can be Java, Ruby, C#, groovy, vbscript, php and so on.
- Some of the libraries can be Selenium, Cucumber, testNG, Watij, Apachi-poi and so on.
- Frameworks can be Data-Driven, Keyword-driven, Page-object, Hybrid and so on
- Development models can be waterfall, spiral, Agile, TDD, BDD, ATDD and so on
As with software development in general, Test Automation IS A DEVELOPMENT EFFORT. By that I mean, test Automation is writing code to accelerate the feedback loop (build-measure-learn). For some of us who come from waterfall, spiral models of working, it is to say that Test Automation helps us in Regression testing and save time and money.
What is TDD/ATDD?
I do not want to get into the differences at this point, but by and large both of them have many similarities and for this blog and Test Automation in general, both the models are similar.
ATDD or acceptance-test driven development is a model of software development that brings the 3 amigos aka. developer, tester and business analyst collaborate on acceptance criteria constantly and consistently until the software is delivered. Typically starts with defining acceptance-criteria (how software should behave and use cases) and using the acceptance-criteria as a single source of truth by the tester and developer to advance in their day-day work.
Combine this working model with Agile and Continuous Integration/Continuous Deployment/Continuous Execution. That is when the power of ATDD really shows up. Agile teams working in Sprints, team collaborates as often as possible on where each team is on the tasks and have daily-stand ups , sprint reviews, retrospects and pivot to adapt to business changes.
Continuous Integration/ Continuous Deployment is the concept of integrating and deploying code as often as possible so that failures are detected as early as possible. Continuous Execution is to be able to run acceptance-tests (or test cases) with the same velocity as continuous integration and deployment happens.
The power of Test Automation really shows up during Continuous Execution in sync with CI/CD.
In an Agile environment that follows CI/CD/CE, I cannot stress more that Test Automation is as quintessential as heart-beat for a human body.
There is often a misconception in teams when they first adopt ATDD, that this is a new working model only for testers — It is NOT. ATDD is a practice that needs to be adopted and followed by everyone on an Agile team. A typical use case of ATDD is as follows:
- Business Analyst writes acceptance-tests from a high level perspective
- Testers expand on the acceptance-tests in that they cover all possible scenarios and boundary conditions
- Acceptance-tests should have automation code/scripts associated to run the scenarios
- Before the source code is developed, the automation scripts fail
- Developers run the acceptance-tests and it fails the first time as in Step 4)
- Developers have to write source code to make the acceptance-tests pass (this will be in iterations and testers keep on adding more acceptance-tests and automation code)
- End of every sprint (generally 2 weeks), we assess how many acceptance-tests have passed – that measures how close are we to the delivery. We also assess end of every sprint if we need to pivot as per changing business needs
As you can see the steps 1 through 7 happen every sprint and the cycle Build-measure-learn is a very powerful concept.
What about defects?
Since the feedback loop aka. build-measure-learn happens as often as possible, the defects or bugs are fixed as early as possible ultimately resulting in huge cost savings in time and money.
How to choose Test Automation Solution: So coming back to our discussion on how do we choose a Test Automation Solution given the landscape of so many moving parts and changes at organizational level? It is NOT easy, however taking baby steps and following some of the practices which many start-ups and innovative companies have done helps. Think about how nimble these companies and if that brings a glean in your eye, keep reading. There is no one size fits all, however asking the right questions helps align with the overall goals and directions. People: Do I have the right skill sets in the team members ? What kind of training do they need ? Do I bring in external consultants to get started? Or do I train change agents within the company ? Do I have the buy-in of the team members to experiment on new practices ? How do I NOT disrupt the existing operations? Processes: One implementation of Agile believes in decentralization and every team members answers to a team, not necessarily to one person on the team. Of course that also requires a change in mindset of all the stake holders, since there is relinquish of control and more transparency, responsibility and accountability power shifts. Every person team should feel responsible. So what happens if something doesn’t get fixed ? Does it result in the circular discussions of blames ? How do we build a culture where all team members think positive and work together to address issues and not point fingers ? What processes should exist if you are a financial company like auditing, compliance etc ? How do we build a quality driven culture and that ‘Quality’ is the responsibility of everyone on the team and NOT just testers ? What are the dependencies on technologies and does processes align with those technologies ? Technologies: What softwares/tools do I choose for Test Automation? How to resolve the transitive and n-level dependencies of the choice of each tool over others ? How do I decide which programming language, which libraries and which Test Automation Frameworks ? Can I choose technologies that most of the team members (aka developers) are familiar with, so that pair programming becomes a norm and developers help testers ? On maintenance projects, since these decisions have already been made, can I choose upcoming and relatively much simpler/readable coding languages like Ruby? What Gap-Analysis and attributes need to be taken into account while choosing languages, libraries and automation frameworks.
Closing Thoughts:
It really depends on each situation and context to decide and approach and choosing a Test Automation Solution, however I have observed over years that Test Automation is a development project, meaning the more closer a tester gets to becoming a developer, the more ROI is realized from Test Automation. That does NOT mean, there are NO solutions for patrons who want to take baby steps without being comfortable with programming language.That is where “Test Automation Framework” plays an extremely critical role in alleviating the concerns and challenges of team members. As cliched as it may sounds, the context, company, domain, industry determines if we have to choose a top-down or bottom-up or both approaches when designing, developing and rolling out Test Automation Frameworks. In my next post, I will expand on the attributes I have used to build Test Automation frameworks and that immensely helped mid-sized and large clients to become effective and efficient as teams and organization.
AutomatorWorld 