Often while designing test automation frameworks, we get overwhelmed with the # of requirements that needs to be satisfied. We need speed, accuracy, modularity, flexibility and so on. Below are a few attributes to talk about when we talk to higher management in convincing them to invest in developing a test automation framework. Also bear in mind that test automation is very tightly coupled with build, release and development these days. The reason is because quality is not responsibility of one person, but the whole team and test automation is something which any person in the team should be able to run at look at the results. So the context in which I am talking is an Agile team following one of BDD,TDD or ATDD and using a page-object test automation pattern.
- Maintainability: How do we ensure maintenance of scripts is optimal? By that we mean, as we evolve by churning out feature files, step definitions, data and page objects, we need to have a strategy to be able to diagnose, version control and audit the artifacts produced. There should also be a tight coupling between the released version of source code and the corresponding set of acceptance tests which tested that released version.
- Re-usability: What are the specific artifacts that can be re-used across feature teams? This is particularly helpful during integration testing when components/functions have to talk to each other. For example, data and page objects are re-usable artifacts
- Scalability: How do we scale the acceptance tests and ultimately accelerate the feedback loop to verify/validate the software to release and market per deadlines?We can use Selenium GRID for parallel execution of tests. The necessary infrastructure required for a distributed parallel execution of tests needs to be discussed too. The technology needs for the same in the Technology section and any exceptions and dependencies on infrastructure (hardware/software) should be laid out.
- Configurability: How do we configure the test automation framework and how much of it is convention vs. configuration?Environment configuration (switch environments),Logging (Detailed log to help debug)
- Auditability: How do we keep track of work done and roll up metrics to serve QS dashboards or any other management required metrics. Reports (Historical vs. current), Auditability (System of record)
Popular Test Automation Framework Designs
1) Keyword Driven : Keywords are essentially blocks or functions that are named so that they can be consumed for a given functionality. At a simplest level, keyword takes in parameters and throws back some output. Keywords are written to abstract the complexity of repeatedly writing code. Think of Keywords as API’s that serve you (or your application) when called. In one of the frameworks I designed for a large company, Keywords.java was a Class file with list of Java methods with defined inputs and outputs.
However Keywords can get complicated too based on data structures and layers in your test automation framework
2) Data Driven: If your applications is NOT too deep with pages , however each page can have scenarios that need to be tested with large datasets, you would want to write automation scripts with a focus on test data aka. data-driven. Tools like QTP already have excel sheet parsing etc that loops through rows and the same test case is executed for each data-set. It helps me to think in an easier way when I talk about excel for data driven, however please DO NOT think excel is the most optimum way to handle test data. In fact, excel is heavier than xml, yml or other data exchange formats. If you are at the beginning of designing a data-driven framework, go ahead and use a spreadsheet to manage your data, however as you mature and advance, try doing away with spreadsheets.
3) Page-object: This is a very popular pattern used these days for Test Automation Frameworks. If you think of a web based application as a network of pages that interact with each other AND there are large number of pages in your application that can change nature frequently, this pattern is the most favorable one to use.
Page-objects (or classes) are defined for each page, where a page contains the html locator information and also the methods that can operate on those page elements. The advantage of this pattern from the decision making point is:
- If a hmtl locator information changes for an element, changing it in the page-object reflects across all the scripts – nice right
- Gives us ability to define keywords too as we can define methods inside page-objects
- We can define navigation logic between the pages too as a way to tie the links between the pages
Tip: If your application has very few pages (maybe <15), page-object pattern is probably an over kill. You may use it, however I would suggest to use a Keyword-driven or data-driven framework
4) Hybrid: A Hybrid model is permutations and combinations of the above three frameworks described
See Ruby support from tooling perspective