Debugging Failed tests with Videos and Screenshots

Failing tests are inevitable. Otherwise, why would you write tests that are always green? So when tests fail, it's great to have a wealth of post-mortem debugging tools to understand what happened in the failed testation. Two simple tools provided by playwright are screenshots and video recordings. Here we have a simple test that is designed to fail for it to do style application. And in case you are wondering why it's going to fail, it's going to fail because we are adding cheese with a Z. Whereas we are expecting cheese with an S, which is the correct spelling. Let's say we want to run this test on the terminal and

after the test fails, we want to visually see what happened within our application. And that is where screenshots and videos come in. Screenshots and videos of our test executions can be controlled from within the use section of our playwright config. To enable screenshots, we use the screenshot option and one of the possible values is only on failure, and this will actually take a screenshot of the final state of our application if the test ever fails. There are a few other possible values for this as well. One of them is on which would mean that the screenshot is always taken at the end of the test and the default is off, which means

that no screenshots are taken at the end of the test. To control the video recording, we have the video option and one of the possible values is retained on failure. Unlike a screenshot which can be taken at the end of the test, the choice for a video recording needs to be made upfront. So we will either always be recording the video or we will not be recording the video. And the reason why you might not want to record the video is because it does slow the test down because playwright has to do more and with the retain on failure option, the video will only be kept if the test eventually fails. Other possible values are on and off and on. First retry with on first retry.

The video will only be recorded if the test is being retried because for previous failure it is worth pointing out that by the default retries are only configured on the CI server. So there will be two retries if the test is running on the CI and locally, there will be no retries. 'cause honestly, if the test fails locally, you should really start debugging the test at that point. Now with our current screenshot and video options, which will give us a screenshot and a video, if the test fails, let's go ahead and execute the test that is designed to fail. Once the test execution completes, playwright will open up the test pure for us and you can see that the test failed.

You can see the reason why the test failed as well within the logs. For example, we expect it to find by cheese, but we found by cheese with a Z. But below that we also have the screenshots available. So you can see within the screenshot that the US displaying the wrong text. That is it is showing cheese with a Z. And if you scroll down a bit more after the screenshot, we find the video as well. And we can see that the to-do was added with the wrong text. And now we are expecting for GS with an S to show up, which never happens. As a side note, these screenshots and these videos are also saved to the file system under the test results folder,

so we can easily share them with our colleagues if you wanted to. Of course, it's great when you see these generated screenshots and videos locally, but what makes these test artifacts really shine is when an error happens on our continuous integration or CI Server. You might recall from our previous lesson that we have a file called playwright yaml present within the dot GitHub folder, which is designed to run all of our tests. As soon as we push our code over to GitHub, one of the steps within this job is NPX playwright test, which will run all of our tests. And of course, we expect one of these to start failing, which is the failed test that we are about to push.

Another job that this particular action will run is called upload artifact, and it'll always run this, which means that even if the test fail, we will still get our artifacts and the artifacts will be the files that are contained within the path playwright dash report. And the artifact within the GitHub action is also going to be called playwright dash report. Now I've just pushed our code over to GitHub and you can see that it's already started running our test and we can see the running test and drill into its details. Of course, this particular execution will eventually fail because we have pushed a test that is designed to fail

and no surprise that it does fail. Eventually, however, you can see that the upload artifact succeeded, and if you go into the summary tab at the bottom, we can find the artifacts that GitHub is collected, and of course one of them is playwright Dash report. This is going to be a zip file of the contents of the playwright dash report folder. So right now I'm just downloading it to the desktop and extracting it. And within it you can find this file called index html, which we can open in the browser and you can see the details that we saw previously locally as well. You can also see the screenshot and we can open that in a new tab.

And if you scroll down a bit more, we can find the video as well and we can open up that in a new tab. And these tabs will actually point out how these files are located within the playwright dash report folder. And from here you can just save them as something else on your file system as well to shade them around with your colleagues.