Nala Visual Compare nomenclature

[amauch-adobe]

What are everyone's thoughts about visual tests having the same name as a functionality test but instead of a different file extension?

For example,

functionality test => columns.test.js
visual compare test => columns.vc.test.js

Playwright looks for test files based on this pattern by default *.test.js so this naming will work.

[amauch-adobe]

Here is a follow-up question for the location structure, for the screenshots directory that needs to be committed per this documentation (https://playwright.dev/docs/test-snapshots) last paragraph. I was thinking we could do two options:

1. have all screenshots fall under a parent screenshots directory, but have different sub-directories for different types of test screenshots, i.e. /visual-compare/... and/or /test-failures/... etc. Then we would do .gitignore rules on the specific sub-directories that shouldn't be committed up.
2. Entirely just have visual comparison screenshots that need to be committed in a different directory and keep the .gitignore as it is currently that doesn't commit the default /screenshots directory.

I don't have a strong opinion on either option, just wanting to get community input on what makes more sense.

[amauch-adobe]

I haven't tried that yet, but it is certainly permitted since you can tell it the name of the screenshot you want it to compare it against.

[skumar09]

I think we should try that, so that we can always have a base snapshot reference against which visual comparison is done, WDYT?

[amauch-adobe]

@skumar09 where are you thinking the base images could come from, who/how would those base images be made?

expect(page).toHaveScreenshot() can take a path to where the images are placed as well as the name of the screenshot so it isn't auto-generated. I'm actually doing that right now by creating a config value for the screenshot path template and then providing a name for the screenshot. When the expect() comes up in the code it first checks to see if there is a base image already created to use against, if there isn't one it creates it, otherwise, it just compares against it.

[skumar09]

I see, that make sense, 👍
(I was just thinking of capturing bases images by any screenshot capture tool(s), there are many chrome plugins for them )

[amauch-adobe]

Just asking a question for responses, did we want to have the base screenshot process be automated or a manual action from the engineer using a different tool that they would then upload to cloud storage?

I'm leaning toward having it automated since it can be and also because that is how the current way is for most teams using the HelpX screenshot tool within Jenkins. Plus, most licensed products do baseline screenshots automated as well, Applitools/Percy.io, etc.

Open to people's comments though.

[amauch-adobe]

What does everything think about how this structure is set up? I have screenshots going into the screenshots directory, but with subdirectories specific to visual compare tests and different team directories as shown below for both the base screenshots and the visual tests themselves:

[skumar09]

@amauch-adobe : I am also thinking of having a separate visual-test.config.js , for visual testing.

[amauch-adobe]

I think that would be a great idea.

[amauch-adobe]

If we are going to save screenshots to the cloud are we all agreeing that it should be in AWS, i.e. S3?

[JackySun9]

The question here: do we need a separate tool or combine them with E2E test?
If do above structure, how do you define them for different envs? the content might be different for them

[chrischrischris]

Might be a bit more complicated, but you could also create another git repo specifically to store these.

[amauch-adobe]

@chrischrischris Actually, I like that idea since that would be following what Playwright/Microsoft says to do having screenshots in a repo. We then could just add a step in the GitHub Actions to checkout the repo, and direct it to look for it. This way we don't have to request S3 storage or have to maintain it. The only potential concern I see is if you are running a visual test on a hlx.page or staging branches. Although, running visual tests against hlx.page content that is going to be under development or authoring doesn't make much sense, and in general we test production since we are looking to ensure content won't be affected visually by a merge going forward or aggressively checking pages are following brand compliance as they are pushed live. I think it would still work just fine.

@JackySun9 what environments and use cases are you aware of for visual testing currently? Just looking to get an idea.

[JackySun9]

We are using it to monitor some critical pages on prod, so you can start there and improve later