Automation Coverage: Best Practices, When to Avoid, and When to Stop

When thinking about testing automation the popular movement in technology today is to automatically (forgive the pun) go full all in on a testing automation investment. However, not every technology or project necessarily needs a large-scale automation investment. The following represents some of our experience around when and how the decision to invest should be made.

First, we need to ask ourselves if we need automation in the first place. To give us a sense of this initial qualification, consider the following preconditions that will yield a favorable indication for investment.

• When you have a lot of regression testing on the project to do.
• When you have the need for verification of the same functionality frequently.
• When you need to make a lot of small changes or improvements simultaneously.
• When the same issues reappear again and again after “fixing” or developing new and adjacent functionality.
• When you know your product well. Deep knowledge will yield quick results.

The question of when a testing automation investment should be made is of equal importance. Frequently IT professionals think about how to automate everything but forget about when to stop. It’s important to have clear answers to the following questions such as:

• How many test cases should be added?
• How much effort will it take to reach this or that?
• How much does it cost for me, for the company?
• Do we have enough (or too much) test coverage?

As with most things in life, the effort around automation should have a certain balance. Very often people think that they should to do as much as possible and even more, but this approach doesn’t work in some cases. For example:

• Trying to automate everything. A realistic goal is to automate 70-80% of your test; 100% is just a pipe dream. Not all test cases can be or need to be automated.
• Start with test cases that cover high risk areas.
• Treat the manual and automation efforts as independent teams.
• Don’t waste too much time on automation – that last 20% of effort will cost you the same as the first 80% of effort.

From this experience we can draw some conclusions that are red flags as to indicate when you shouldn’t start:

• It is a short-term project (why bother with automation if the project will be finished in couple of months?)
• When manually testing cycles take a less amount of time than the automated.
• When you don’t have any Documentation/Specification about the project or any knowledge base how it should work

When you don’t have access to Engineers who have deep understanding of the technology.

And finally, here are a set of questions that you can use to understand when to stop. This is a very important concept that will help with both time and costs of a project.

• When you have reached the basic project goals (defined on planning stage).
• When you are trying to re-invent a bicycle (it means when you have reached a basic goal, but still trying to add more tests. It is no longer bringing a value, but is only resulting in a headache)
• When your tests are Independent and Isolated (each test shouldn’t depend on previous test, they should be run separately to don’t affect each other). This point can help you to run your tests in parallel and save more time

Best Practices

The following represent some of our best practices that can help to reach desired goals, based on our own experience:

• Define the goals for automation – likely the most important thing when taking this journey.
• Hire proper Engineers that fit your needs and invest in training them.
• Be clear that automation WILL NOT replace manual testing. It is worth it usually, but there is no magical cure to testing.
• Understand that automation is a separate work component to the project and do not make the mistake of including it in the estimation of code development.
• Pick the right tools/frameworks that best fits your needs on the project.
• Think about Continuous Integration. Test run/reports should be done without human interaction.
• Be sure to create good documentation and hand-off procedures to ensure sustainability and growth.

Conclusion

To start an automation project and maintain it properly requires a high level of planning and preparation. Choosing the right tools/framework will save time over the long term and it will help you to reach a goal. Don’t forget that the automation development will be the same as your product life cycle – it never ends.

Doing automation in the wrong way will be a complete waste of time and resources.