Visual Regression Testing (VRT)

A core part of web application testing is VRT, which stands for visual regression testing. It is a testing pattern where we make sure that our app visually looks the way that we expect it to consider a simple test for our reduced style application, where we want to visually make sure that our app renders what we expect it to. We can do this with a simple test where we expect that a page should have a screenshot that matches the required visuals. It is worth pointing out that the screenshot name is actually optional, and if you don't provide one, then playwright will generate a name for us. Now, initially of course, there will be no screenshots,

so if you run this test right now, it will fail. The test will execute with a playwright runner, and once it completes, you can see that it is erroring out. If you look at the error, you can see that it has noticed that there is no screenshot, and in fact, it has created that screenshot for us. The screenshots are created in a folder next to our test, and the folder name is our test file name followed by dash snapshots. The screenshot name is going to be the file that we provided, which is initial, followed by the project name, which is chromium, followed by the operating system, which in our case is Davin for Mac os.

Once we have the screenshot created and we run the test again, this time playwright will compare the existing screenshot with a new one that it is going to take, and if it matches, the test is going to pass and no surprise in this particular case, the test completes successfully. The screenshots that are created should be committed as a part of your repo, for example, with Git, and that will ensure that anybody else that runs the same test is going to get a successful result. As you make modifications to our applications, our screenshots will continue to ensure that the visuals do not get broken, and if it does break due to an intentional change,

we can fix it quite easily with a simple command. To update the screenshots, notice the color of the to-dos heading within our application. Let's jump into the CSS and make a modification to change the color to something like the color black. Now, let's run our VT spec file again, but this time by using the terminal, of course we expect the test to fail because our application has been modified. But a great thing to note is the level of details that will be present within the test results. You can see that the test errored out because the screenshot comparison failed, and if you look at the steps,

you can see exactly which step caused the test failure. But additionally, we also get this portion called the image mismatch. This section highlights the pixels on the screen that are different between the previous screenshot and the screenshot that was generated as a part of this test. The previous screenshot is present within the expected section and the new screenshot is present within the actual section. We also get this nice slider so that we can scroll between the two to exactly see the visual difference between the actual and the expected. Now that we've seen these two images,

it is very clear why the diff is highlighting the new heading. Now, if the stiff is intentional, which in our case it is, we can actually update the existing screenshot quite easily by running the test again, but this time passing in another command line flag called update snapshots. This time, instead of doing a comparison between the old and the new, the test will simply replace the existing image with a new image that is generated and the test will pass. So now if you look at the image that is present within our snapshots folder, you can see that it contains the updated visuals

with the to-dos colored in black. Quite commonly in web applications, portions of the app are dynamic and change on every render. Common examples include banner ads and even information panels designed to intentionally vary as it increases user engagement. These can break our visual screenshot comparisons, but playwright has a simple fix for these scenarios. As an example, let's say that we don't want to compare the footer of our application. We can pass in an options object to the two half screenshot method, and one of the options is the mask property,

which takes an array of playwright locators. We pass in a locator for our footer, and now when the screenshot is taken, the footer is going to be masked out with the standard color of magenta. The first time we run the test, the screenshot is taken, and now if you run the test again, it makes sure that that screenshot does not change. Let's take a look at the screenshot that is generated with the footer masked. You can see that the footer portion of a to-DOS application has been masked out with a nice magenta block. What this means is that if any modifications are made to the footer block, for example, we modify the text

to be something completely different and then run the test again. The footer is not going to be a part of the visual comparison and the test will continue to run successfully. The same mask filtering can be applied to any of the portions of the application that you don't want to be a part of. The visual comparison, for example, banner ads and information panels, underlying the VRT support in playwright is the library called pixel match, and playwright provides various options for controlling how the visual comparison between the screenshots is performed. Pixel match is a very popular library that is used by a lot of other frameworks as well.

The options for the pixel match library are exposed as options to the two have screenshot method. For example, here we are modifying the threshold for the comparison to be 0.5. You can see that the default value is 0.2 and zero means extremely strict and one means completely relaxed. There are other properties that we could modify as well, and we can even provide default values using the playwright config. We can use these options to reduce any chance of flakiness in our tests because of the rendering behaviors of our application. In addition to screenshots of the entire page,

playwright also supports visual regression testing for individual locators. This can help you narrow in on the key portions of your user interface. Let's create a new test to make sure that the new to-do input of our application does not change visually. Instead of passing the page to our expect method, we pass in a page locator pointing to the new to-do input, and now if you run the test, of course the first time the test will Fail and the screenshot will be created. But from that point on, if you run the test again, it'll make sure that the screenshot matches

its previous expectation. Let's take a look at the screenshot that is created for our to do input, and of course, it is just the to do input portion of our page.