Revisiting end-to-end testing to improve reliability, speed, and development

When a layperson imagines quality control, they usually imagine end-to-end testing. Approach the software as a user would and execute the same journey they would. The idea makes sense; If a tester can get through your software end-to-end, it works.

Well, it’s not that simple. Users make errors, get confused, enter insufficient data, and try to use the software for things it was not meant to. End-to-end testing attempts to map all those different scenarios and execute them before every software update to ensure the software still behaves as expected under regular use.

But even if you make a complete roadmap of user journeys through your software, end-to-end testing still has the potential to cause more problems than help find solutions. This article explores the common pitfalls of end-to-end testing and how you can leverage it better. 

Overview of end-to-end testing

You can perform end-to-end testing using automated tools, manually, and sometimes via APIs. You may test a simple grocery list app by performing everyday tasks such as adding, removing, deleting, and checking list items. If the app is connected to a server, then the server can be tested using API calls performing those tasks, making the testing an automated process.

End-to-end testing aims to ensure that users can complete specific tasks in your application. It is done by running scenarios and then identifying failure points so the development team can fix them. Unfortunately, that is easier said than done.

The pitfalls of end-to-end testing 

End-to-end testing is often a necessity. You must see that the user processes are performing as expected and that no significant and obvious faults arise. But too high a focus on end-to-end testing is not ideal and can lead to slow development cycles and increased costs.

Slow execution

End-to-end testing naturally takes a long time. It must be thorough and slow, even if you review code using automation testing, scripting, or APIs. When you commit an update to the software, running a complete array of end-to-end tests is impractical due to the slow execution. Therefore end-to-end testing is reserved for the end of the development cycle.

But keeping testing until the end of the development cycle means that developers get slow feedback on their changes, and if you find an issue at this stage, you have to fix it late. To put an even more significant strain on the problem, any fix can cause other areas of the code to break, meaning you must restart end-to-end testing after the fixes are applied. This feedback loop could be faster and more efficient.

There are several ways to improve the efficiency of end-to-end testing:

• Prioritize essential workflows for end-to-end testing and only run them on limited user journeys.
• Use automation to increase testing speed, allowing faster feedback loops.
• Keep end-to-end as a failsafe rather than a first application testing, so when testing reaches this stage, most, if not all, tests pass.

To keep end-to-end testing as a failsafe, you should employ other testing methods, such as unit testing. Another option would be to use an intelligent auto-completion tool that can reduce bugs in your code. AI coding assistant tools like Tabnine or Copilot can, for instance, auto-complete your lines of code based on your best practices, style and patterns, to ensure consistency and accuracy. AI can also automatically generate fixes for bugs, making your testing process much easier. 

Flaky testing 

Google reports about 1.5% of tests failing flakily, meaning tests that fail and pass with the same code. When those tests are unit tests, you can identify them as flaky by running them multiple times and then removing the flaky tests. With end-to-end testing, this would be much more difficult to detect.

End-to-end testing covers a lot of ground because the user interacts with many modules during a user journey. It is easy to see that if one portion of a long series of actions is flaky, the entire test becomes flaky. And if every step has a 1.5% chance of being unreliable, that can add up, resulting in end-to-end testing having a very high chance of being flaky themselves, much higher than 1.5%.

Team friction

End-to-end testing run in a single environment is largely reproducible and consistent (see flaky testing), but most developers use distributed systems these days. Testing in distributed systems poses many challenges, especially for end-to-end testing, as the same user journey can produce vastly different results in different scenarios.

Teams testing on various platforms can have miscommunication issues and friction between them. To ensure end-to-end testing goes smoothly, you must synchronize technologies, data, and information processed between the teams. Using cloud and standard test runtime tools like WebLOAD and LoadNinja will help create a single source of truth for end-to-end testing and improve the quality of tests and the end experience for the user. 

Increasing costs 

Interconnected applications increase the complexity, time, effort, and cost of end-to-end testing. Third-party services connected to your software further increase the cost of running tests, especially when you have no control over those services and they also have a chance of failure.

Improving reliability, speed, and development in end-to-end testing 

After reviewing some of the most common pitfalls, we look at a few methodologies to enhance end-to-end testing.

Reviewing the order of tests

Due to the abovementioned issues, you must keep end-to-end testing at the correct balance with the unit and integration testing. End-to-end testing does things that other forms of testing don’t do, ensuring that common user workflows perform as expected. While it is key to testing common user scenarios and ensuring the application’s stability, other types of testing, such as user testing, are more suitable for isolated bugs. Therefore, relying solely on E2E is not enough. 

Google recommends 10% end-to-end tests, 20% integration testing, and 70% unit tests. This 10/20/70 split is a good baseline for you to work from, but naturally, it will differ based on your product and team. Keep in mind that an inverted pyramid suffers from many of the pitfalls we’ve outlined, and an hourglass shape probably means you’re using end-to-end testing when you should be using integration testing.

Improving visibility

Sometimes, the issue wasn’t a fault in the software but rather a system failure that can happen regardless of changes in your code. OS, browser, or hardware crashes may cause your tests to fail, even though it has nothing to do with your code. When performing automated UI testing, it can be helpful to record the testing sessions, either with video or a log of operations.

Recording tests in such a way helps identify the cause of the issue.

Introducing automation 

Automation tools can help reduce issues such as misconfigurations, which introduce much of the flakiness in end-to-end testing. Equally, automating key tests as part of your CI/CD pipeline will make it easier to ensure security across all stages of the development process, without the extra burden on developers. 

Disabling flaky tests 

Flaky tests are a severe hindrance in end-to-end testing, but you can mitigate the effect of flaky tests by disabling them. Identifying which tests are flaky requires that you perform a large volume of tests, which is an issue in a relatively slow process as end-to-end testing. But if you identify a flaky test, disabling it until you can fix the test is often the best course of action.

Don’t overdo end-to-end testing

End-to-end testing is not the be-all and end-all that it seems to be. There are many pitfalls to avoid, but also just as many solutions. Make sure you have a good balance between end-to-end testing and other more reliable, quicker forms of testing, and you can get the most out of it. If you’re interested in how Tabnine can help you make testing more efficient, book a demo to see how it works.