Flaky Cypress Tests - What They Are and How To Deal With Them?

Flaky Tests

We are using numerous blessings of technology every moment but, as normal users, we hardly pay attention to the huge operations going on behind those web products and services. With every passing moment, the requirement for more advanced & efficient features is getting higher, user databases are becoming huge, and eventually, the demand from technology is touching the sky. Developers are constantly brainstorming on efficient web products and developing new features along with ensuring the best performance from those products. But, as we all know, maintaining them is more important than keeping on developing more applications. QA is the most effective way to achieve the best quality and efficiency.

If we talk about QA, we can never undervalue Cypress tests, the most widely used, efficient way to ensure the best quality of your web products. While the cypress tests can be really helpful, the flaky ones can cause massive setbacks in performance and waste a lot of time & resources. That’s why we are going to know about the flaky cypress tests in detail and explore the ways to deal with them

What Are Flaky Cypress Tests?

If you observe some tests failing at random intervals while no one is making any changes to the code, then those tests can be called flaky. These tests make you confused whether the tests are flaky or you are observing bugs, and to detect that, you get to rerun the same tests multiple times. As a result, your CI/CD pipeline gets significantly slowed down and eventually the flaky tests lead you towards unpredictability & frustration due to repeated troubleshooting. In the long-term, these can make your customers lose trust in your product and your business will face immense downfall. Hence, we must work out some useful and efficient ways to deal with them.

How To Detect Flaky Cypress Tests?

There are mainly two ways to detect if any cypress test is flaky or not. One is rerunning a test or even a whole test suite multiple times without any change in the code to see if there is any change in the number of failed test suites at every run. If the number is different every time or most of the time, then there is something wrong with your test suite.

In the second method, you need to rerun the test in a different order each time. If your test fails due to a change in the order, you may have forgotten to take interdependencies into account.

Now, as we make you familiar with the hardships in detecting flaky Cypress tests or executing the Cypress tests in general, we must provide you with a way to get rid of those issues. So, PreFlight’s Cypress recorder lets you create your tests in minutes, and that too using a user-friendly interface. After you are done creating your test, you will get its auto-generated Cypress code instantly. Then you are free to run those codes with all the customizations that you want. Our tool lets you conduct numerous tests with the most efficient cypress code generated in minutes. Anyone with no coding knowledge can also easily generate Cypress test codes and execute massive QA projects within the shortest amount of time.

How To Deal With The Flaky Cypress Tests?

After detecting the Flaky cypress tests, it’s time to get rid of them so that your product doesn’t lose its place in the market. Let’s check out the best practices for that.

Visualize Test Runs

In order to get rid of flaky cypress tests, you must visualize them first. Visualizing multiple randomized test runs will let you analyze how well your tests are performing and let you observe the increment or decrement in the number of flaky tests. Developing a table with well-organized rows and columns helps you achieve the best results in visualizing test runs.

Reduce Inconsistent Interactions In The DOM

One crucial cause of flaky tests is repeated interaction with elements in the DOM that do not render consistently. Those elements may look fine when they are tested manually but they produce errors during Cypress tests. Cypress keeps on retrying the commands without maintaining the best-suited order on the malfunctioning elements. The best way to solve the issue is combining the commands like .get(), .find() in single commands and setting assertions for those single commands. It mainly retries the command that is prior to the assertion that failed in the chain. Another reason for this kind of flakiness is the timing of the events that mostly deal with various input fields. You can use custom Cypress commands to resolve this problem. So, from this point, we can conclude that one of the best ways to get rid of the flaky Cypress tests is by asserting every command that you want Cypress to retry and using custom Cypress commands wherever necessary.

Quarantining Flaky Tests

Flaky tests need to be rerun multiple times with implemented solutions to check whether they are getting resolved or not. But, operating on them when they are among the non-flaky tests causes the rerunning of the entire test suite and eventually, the efficiency reduces. Hence, the best way of resolving this is to create a separate test suite for them. A separate test suite prevents duplication in the part of work required to solve the flaky tests. It provides you the opportunity to fix the flaky tests without any interruption. This way you will get a better idea about the role of inter-test dependence in flakiness

Reduce Inconsistency In Requests

Another form of flakiness is caused by inconsistent responses to network requests. Such flakiness can occur when the test execution continues even without a request having received a response. But, the good news is that there is an efficient solution to this issue. You can use intercept commands to stub responses for the requests that you want to receive a successful response. This is a nice strategy for testing a page that makes calls to external APIs. Besides using intercept commands, using Cypress fixtures is another great way to ensure that the flakiness is reduced

If the test execution continues without a network request getting properly finished, it can cause flakiness. And, if you manually enforce waiting time, it can increase the overall time required for the test suite to run. Hence, an effective method to get rid of this issue is using parallelization in the CI pipeline

Clean Up State

Previously generated data that you have forgotten about can cause the present tests to get off the expected track. These states and data can stay in the caches, databases, or even variables. Stay aware of the cleaning-up state after the test is completed. Often clean-up errors are ignored. The worst-case scenario is rebuilding the whole system for every test run. As transactions can be rolled back after a test run, the database can be brought back to its old state

Look For Timeouts

Asynchronous tests are a form of test that needs significant access to network resources. We all know that the network turns out to be fast or slow depending on the services using it. As a result, the asynchronous tests often become flaky due to timeouts. And, the more crucial thing is that too short timeouts can cause significant flakiness in tests. The best practices to prevent this type of flakiness are -

  • Set the timeout variables in bulk so that you can change them quickly in the future.
  • In case of any complex test that has significant reliance on asynchronous services, make sure to check the services for availability. It will prevent the timeouts from becoming too long.

Use Test Doubles

Some services do not focus on minutely determining if the test is the real one or somewhat duplicate. So, you can easily create a simplified version of the actual test. Though there are debates on the fact that test doubles are not always equivalent to the real ones, they are truly useful to look for updates without experiencing flakiness with the real services.

Check The System Clock

The system clock is a type of data that cannot be known in advance. And, there are several tests that depend on this data. Using faulty data can cause immense flakiness in the Cypress tests. Hence, the best practice for getting rid of it is wrapping such data sources in your code and avoiding direct reliance on them. That way you can replace their outputs with hard-coded data before running a test.

Check For Memory Leaks

Available resources and other systems running on any hardware are extremely crucial for testing it. If your code that uses those resources has memory leaks, it becomes the cause of flakiness in the tests. You can decide if your code has memory leaks by observing if the memory usage of the test suite grows with every test run.

This flakiness can be controlled if the failure of the tests is pre-known. And, that can be achieved by using resource allocation pools as wrappers between your code and the actual memory allocation. Then the flakiness can be fixed by controlling the memory allocation.

Properly Utilize The PreFlight Dashboard

PreFlight always makes sure that our clients get the best service and the most convenient user experience from our products. Similarly, to deal with flaky Cypress tests, you can access the following awesome features on our dashboard.

  • Lets You Keep The Tests Small: Well, keeping the tests small may sound like ignoring some of the important features or aspects. But, it is an absolutely false assumption. Keeping the tests small means focusing on testing the products from the user's perspective only. In simple language, you should test a web application in a way a user interacts with it while ignoring all the unnecessary actions. And, ignoring various unnecessary parts of a lengthy Cypress test significantly improves the overall test efficiency. Our dashboard allows you to set suitable visual and text assertions to verify that the application looks and performs as you expect. You can also get immense help from our dashboard in creating small tests that can run parallelly to provide a better testing experience.
  • Helps You Get Rid Of Repetition: Same workflows can be useful for multiple tests and if you keep duplicating it, you will need to make the same updates in all the tests for any change. To solve this problem, our dashboard enables you to manage all the workflows in different Cypress tests from a single location.
  • Lets You Create Repeatable Tests: As we are mostly focusing on automating the tests to reduce or absolutely eliminate manual testing, we must ensure that our created tests can be run repeatedly. In various cases like a product being out-of-stock, a record getting deleted, testing a calendar with a static date, testing a feature that allows only one active user session at a time, a sign-up flow that requires one unique email address, etc., the tests become unrepeatable. But, our dashboard has features that allow you to create dynamic and repeatable Cypress tests in minutes.
  • Enables You To Create Parallel Tests: Despite creating multiple small tests as the alternative to a single test, they may take longer to get executed if they are not running parallelly. You can use the smart features of our dashboard to create small Cypress tests that can run parallelly and significantly reduce the overall test execution time. This is mainly done by creating a new record for a test and then modifying it immediately. This approach makes sure that any test does not break another one.
  • Use Stable Selectors/Locators: Automation tools use selectors/locators to identify an element to interact with. Selectors are of two types i.e.
  • CSS Selectors: These are used in the document.querySelector API, CSS Styling, and jQuery.
  • XPath Selectors: These identify elements using a query language used for XML documents and some more ways like by any element’s text value.

The better choice between these is the CSS selectors as they are easier to read and front-end developers can easily deal with them. While you are building your selectors manually, our smart features help you add markups in your application that are used exclusively by the selectors in your tests. Adding these attributes will give your automation an explicit, and hopefully static, element identifier.

  • Helps You Isolate The Data That Is Untouched By Automated Tests: The users, accounts, and any form of data that is used by automated Cypress tests should not be used in any kind of manual operation. Otherwise, the users can end up modifying the underlying state of workflows and as a result, the Cypress tests may fail or become flaky. And, the worst part is that these issues are very difficult to debug as it is terribly cumbersome to find out how the data is modified. You can get useful features on our dashboard to separate manual and automated testing at the highest level possible.

Conclusion

We went through some of the most useful and effective ways to deal with flaky Cypress tests. You can stay assured that Team PreFlight is continuously digging and brainstorming to find out the most efficient features to provide the best automated testing experience to our users. You can just check out how our simple yet smartest browser extension lets you create and perform numerous complex tests in minutes. We also have the most efficient Cypress Test Recorder that enables you to create complex tests in a glimpse and generates their Cypress codes within moments.

To know more about our amazing tools, feel free to reach out to us or check out our initial documentation. Also, do visit our blog page to learn more about more interesting topics.