[Defer Uploads v2] Added support for handling multiple root resource …

…in single asset discovery phase (#1728) * [Defer Uploads v2] Added support for handling multiple root resource in single asset discovery phase * update domSnapshot structure in config + reload page in case on multidom * Fix test * Added tests for widths endpoint + multiple root resources * add multi dom case * Added case for percyCSS with multiple root resource * Remove extra code and improve coverage * Fix coverage * Coverage resolved * Addressed comments * Minor improvement * Addressed comments * wait for page navigation while force reloading * Updated test to have different resource in different doms * Add logging for test browser userAgent * Fix tests * Change userAgent condition for responsiveSnapshotCapture * Added test endpoint for changing widths * Added waitForResize script * Add flag for changing config in test api * Added tests for waitForResize * Added defer uploads option in test config endpoint * Responsive DOM SDK Utils (#1746) * Fixed cookies parsing bug (#1727) * v1.29.4-beta.3 (#1729) * adding secure flag for cookies that starts with __Secure (#1730) * releasing v1.29.4-beta.4 (#1731) * updating path-to-regexp to 6.3.0 where the vulnerabilities are fixed (#1733) * releasing v1.29.4-beta.5 (#1735) * ⬆️ Bump micromatch from 4.0.7 to 4.0.8 (#1706) Bumps [micromatch](https://github.com/micromatch/micromatch) from 4.0.7 to 4.0.8. - [Release notes](https://github.com/micromatch/micromatch/releases) - [Changelog](https://github.com/micromatch/micromatch/blob/master/CHANGELOG.md) - [Commits](micromatch/micromatch@4.0.7...4.0.8) --- updated-dependencies: - dependency-name: micromatch dependency-type: direct:production ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * ⬆️ Bump @babel/preset-env from 7.19.4 to 7.25.4 (#1704) Bumps [@babel/preset-env](https://github.com/babel/babel/tree/HEAD/packages/babel-preset-env) from 7.19.4 to 7.25.4. - [Release notes](https://github.com/babel/babel/releases) - [Changelog](https://github.com/babel/babel/blob/main/CHANGELOG.md) - [Commits](https://github.com/babel/babel/commits/v7.25.4/packages/babel-preset-env) --- updated-dependencies: - dependency-name: "@babel/preset-env" dependency-type: direct:development update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * ⬆️ Bump axios from 1.6.7 to 1.7.7 (#1737) Bumps [axios](https://github.com/axios/axios) from 1.6.7 to 1.7.7. - [Release notes](https://github.com/axios/axios/releases) - [Changelog](https://github.com/axios/axios/blob/v1.x/CHANGELOG.md) - [Commits](axios/axios@v1.6.7...v1.7.7) --- updated-dependencies: - dependency-name: axios dependency-type: indirect ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * ⬆️ Bump body-parser from 1.19.0 to 1.20.3 (#1720) Bumps [body-parser](https://github.com/expressjs/body-parser) from 1.19.0 to 1.20.3. - [Release notes](https://github.com/expressjs/body-parser/releases) - [Changelog](https://github.com/expressjs/body-parser/blob/master/HISTORY.md) - [Commits](expressjs/body-parser@1.19.0...1.20.3) --- updated-dependencies: - dependency-name: body-parser dependency-type: indirect ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * ⬆️ Bump eslint-plugin-import from 2.29.1 to 2.30.0 (#1740) Bumps [eslint-plugin-import](https://github.com/import-js/eslint-plugin-import) from 2.29.1 to 2.30.0. - [Release notes](https://github.com/import-js/eslint-plugin-import/releases) - [Changelog](https://github.com/import-js/eslint-plugin-import/blob/main/CHANGELOG.md) - [Commits](import-js/eslint-plugin-import@v2.29.1...v2.30.0) --- updated-dependencies: - dependency-name: eslint-plugin-import dependency-type: direct:development update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * ⬆️👷 Bump github/codeql-action from 3.25.10 to 3.26.8 (#1741) Bumps [github/codeql-action](https://github.com/github/codeql-action) from 3.25.10 to 3.26.8. - [Release notes](https://github.com/github/codeql-action/releases) - [Changelog](https://github.com/github/codeql-action/blob/main/CHANGELOG.md) - [Commits](github/codeql-action@23acc5c...294a9d9) --- updated-dependencies: - dependency-name: github/codeql-action dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * fix: packages/core/package.json to reduce vulnerabilities (#1732) The following vulnerabilities are fixed with an upgrade: - https://snyk.io/vuln/SNYK-JS-MICROMATCH-6838728 Co-authored-by: snyk-bot <[email protected]> * ⬆️ Bump tsd from 0.28.1 to 0.31.2 (#1738) Bumps [tsd](https://github.com/tsdjs/tsd) from 0.28.1 to 0.31.2. - [Release notes](https://github.com/tsdjs/tsd/releases) - [Commits](tsdjs/tsd@v0.28.1...v0.31.2) --- updated-dependencies: - dependency-name: tsd dependency-type: direct:development update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Throttle Logs Endpoint from CLI (#1734) * v1.29.4-beta.6 (#1742) * Multi DOM Capture JS-utils --------- Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: ninadbstack <[email protected]> Co-authored-by: Pradum Kumar <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: AgnellusX1 <[email protected]> Co-authored-by: snyk-bot <[email protected]> Co-authored-by: Amandeep singh <[email protected]> * Handle case when cookie is passed as object from sdk * Remove unnecessary fields from cookies --------- Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: Amit Singh Sansoya <[email protected]> Co-authored-by: ninadbstack <[email protected]> Co-authored-by: Pradum Kumar <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: AgnellusX1 <[email protected]> Co-authored-by: snyk-bot <[email protected]> Co-authored-by: Amandeep singh <[email protected]>
percy · Sep 30, 2024 · 8a9e41b · 8a9e41b
1 parent 1746805
commit 8a9e41b
Show file tree

Hide file tree

Showing 14 changed files with 397 additions and 48 deletions.
diff --git a/packages/core/src/api.js b/packages/core/src/api.js
@@ -62,6 +62,12 @@ export function createPercyServer(percy, port) {
       build: percy.testing?.build ?? percy.build,
       loglevel: percy.loglevel(),
       config: percy.config,
+      widths: {
+        // This is always needed even if width is passed
+        mobile: percy.deviceDetails ? percy.deviceDetails.map((d) => d.width) : [],
+        // This will only be used if width is not passed in options
+        config: percy.config.snapshot.widths
+      },
       success: true,
       type: percy.client.tokenType()
     }))
@@ -187,6 +193,11 @@ export function createPercyServer(percy, port) {
       } else if (cmd === 'version') {
         // the version command will update the api version header for testing
         percy.testing.version = body;
+      } else if (cmd === 'config') {
+        percy.config.snapshot.widths = body.config;
+        percy.deviceDetails = body.mobile?.map((w) => { return { width: w }; });
+        percy.config.snapshot.responsiveSnapshotCapture = !!body.responsive;
+        percy.config.percy.deferUploads = !!body.deferUploads;
       } else if (cmd === 'error' || cmd === 'disconnect') {
         // the error or disconnect commands will cause specific endpoints to fail
         (percy.testing.api ||= {})[body] = cmd;

diff --git a/packages/core/src/config.js b/packages/core/src/config.js
@@ -78,6 +78,10 @@ export const configSchema = {
       sync: {
         type: 'boolean'
       },
+      responsiveSnapshotCapture: {
+        type: 'boolean',
+        default: false
+      },
       testCase: {
         type: 'string'
       },
@@ -291,6 +295,7 @@ export const snapshotSchema = {
         domTransformation: { $ref: '/config/snapshot#/properties/domTransformation' },
         enableLayout: { $ref: '/config/snapshot#/properties/enableLayout' },
         sync: { $ref: '/config/snapshot#/properties/sync' },
+        responsiveSnapshotCapture: { $ref: '/config/snapshot#/properties/responsiveSnapshotCapture' },
         testCase: { $ref: '/config/snapshot#/properties/testCase' },
         labels: { $ref: '/config/snapshot#/properties/labels' },
         thTestCaseExecutionId: { $ref: '/config/snapshot#/properties/thTestCaseExecutionId' },
@@ -454,7 +459,9 @@ export const snapshotSchema = {
                 type: 'array',
                 items: { type: 'string' }
               },
-              cookies: { type: 'string' },
+              cookies: { oneOf: [{ type: 'string' }, { type: 'array', items: { type: 'string' } }] },
+              userAgent: { type: 'string' },
+              width: { $ref: '/config/snapshot#/properties/widths/items' },
               resources: {
                 type: 'array',
                 items: {
@@ -473,7 +480,9 @@ export const snapshotSchema = {
                 items: { type: 'string' }
               }
             }
-          }]
+          },
+          { type: 'array', items: { $ref: '/snapshot#/$defs/dom/properties/domSnapshot/oneOf/1' } }
+          ]
         }
       },
       errors: {

diff --git a/packages/core/src/discovery.js b/packages/core/src/discovery.js
@@ -69,6 +69,11 @@ function debugSnapshotOptions(snapshot) {
   debugProp(snapshot, 'clientInfo');
   debugProp(snapshot, 'environmentInfo');
   debugProp(snapshot, 'domSnapshot', Boolean);
+  if (Array.isArray(snapshot.domSnapshot)) {
+    debugProp(snapshot, 'domSnapshot.0.userAgent');
+  } else {
+    debugProp(snapshot, 'domSnapshot.userAgent');
+  }
 
   for (let added of (snapshot.additionalSnapshots || [])) {
     log.debug(`Additional snapshot: ${added.name}`, snapshot.meta);
@@ -79,12 +84,18 @@ function debugSnapshotOptions(snapshot) {
 }
 
 // parse browser cookies in correct format if flag is enabled
-// it assumes that cookiesStr is string returned by document.cookie
-function parseCookies(cookiesStr) {
-  if (
-    process.env.PERCY_DO_NOT_USE_CAPTURED_COOKIES === 'true' ||
-    !(typeof cookiesStr === 'string' && cookiesStr !== '')
-  ) return null;
+function parseCookies(cookies) {
+  if (process.env.PERCY_DO_NOT_USE_CAPTURED_COOKIES === 'true') return null;
+
+  // If cookies is collected via SDK
+  if (Array.isArray(cookies) && cookies.every(item => typeof item === 'object' && 'name' in item && 'value' in item)) {
+    // omit other fields reason sometimes expiry comes as actual date where we expect it to be double
+    return cookies.map(c => ({ name: c.name, value: c.value, secure: c.secure }));
+  }
+
+  if (!(typeof cookies === 'string' && cookies !== '')) return null;
+  // it assumes that cookiesStr is string returned by document.cookie
+  const cookiesStr = cookies;
 
   return cookiesStr.split('; ').map(c => {
     const eqIdx = c.indexOf('=');
@@ -109,21 +120,51 @@ function waitForDiscoveryNetworkIdle(page, options) {
 
 // Creates an initial resource map for a snapshot containing serialized DOM
 function parseDomResources({ url, domSnapshot }) {
-  if (!domSnapshot) return new Map();
-  let isHTML = typeof domSnapshot === 'string';
-  let { html, resources = [] } = isHTML ? { html: domSnapshot } : domSnapshot;
-  let rootResource = createRootResource(url, html);
+  const map = new Map();
+  if (!domSnapshot) return map;
+  let allRootResources = new Set();
+  let allResources = new Set();
+
+  if (!Array.isArray(domSnapshot)) {
+    domSnapshot = [domSnapshot];
+  }
+
+  for (let dom of domSnapshot) {
+    let isHTML = typeof dom === 'string';
+    let { html, resources = [] } = isHTML ? { html: dom } : dom;
+    resources.forEach(r => allResources.add(r));
+    const attrs = dom.width ? { widths: [dom.width] } : {};
+    let rootResource = createRootResource(url, html, attrs);
+    allRootResources.add(rootResource);
+  }
+  allRootResources = Array.from(allRootResources);
+  map.set(allRootResources[0].url, allRootResources);
+  allResources = Array.from(allResources);
 
   // reduce the array of resources into a keyed map
-  return resources.reduce((map, { url, content, mimetype }) => {
+  return allResources.reduce((map, { url, content, mimetype }) => {
     // serialized resource contents are base64 encoded
     content = Buffer.from(content, mimetype.includes('text') ? 'utf8' : 'base64');
     // specify the resource as provided to prevent overwriting during asset discovery
     let resource = createResource(url, content, mimetype, { provided: true });
     // key the resource by its url and return the map
     return map.set(resource.url, resource);
     // the initial map is created with at least a root resource
-  }, new Map([[rootResource.url, rootResource]]));
+  }, map);
+}
+
+function createAndApplyPercyCSS({ percyCSS, roots }) {
+  let css = createPercyCSSResource(roots[0].url, percyCSS);
+
+  // replace root contents and associated properties
+  roots.forEach(root => {
+    Object.assign(root, createRootResource(root.url, (
+      root.content.replace(/(<\/body>)(?!.*\1)/is, (
+        `<link data-percy-specific-css rel="stylesheet" href="${css.pathname}"/>`
+      ) + '$&'))));
+  });
+
+  return css;
 }
 
 // Calls the provided callback with additional resources
@@ -132,14 +173,14 @@ function processSnapshotResources({ domSnapshot, resources, ...snapshot }) {
   resources = [...(resources?.values() ?? [])];
 
   // find any root resource matching the provided dom snapshot
-  let rootContent = domSnapshot?.html ?? domSnapshot;
-  let root = resources.find(r => r.content === rootContent);
+  // since root resources are stored as array
+  let roots = resources.find(r => Array.isArray(r));
 
   // initialize root resources if needed
-  if (!root) {
+  if (!roots) {
     let domResources = parseDomResources({ ...snapshot, domSnapshot });
     resources = [...domResources.values(), ...resources];
-    root = resources[0];
+    roots = resources.find(r => Array.isArray(r));
   }
 
   // inject Percy CSS
@@ -150,16 +191,13 @@ function processSnapshotResources({ domSnapshot, resources, ...snapshot }) {
       log.warn('DOM elements found outside </body>, percyCSS might not work');
     }
 
-    let css = createPercyCSSResource(root.url, snapshot.percyCSS);
-    resources.push(css);
-
-    // replace root contents and associated properties
-    Object.assign(root, createRootResource(root.url, (
-      root.content.replace(/(<\/body>)(?!.*\1)/is, (
-        `<link data-percy-specific-css rel="stylesheet" href="${css.pathname}"/>`
-      ) + '$&'))));
+    const percyCSSReource = createAndApplyPercyCSS({ percyCSS: snapshot.percyCSS, roots });
+    resources.push(percyCSSReource);
   }
 
+  // For multi dom root resources are stored as array
+  resources = resources.flat();
+
   // include associated snapshot logs matched by meta information
   resources.push(createLogResource(logger.query(log => (
     log.meta.snapshot?.testCase === snapshot.meta.snapshot.testCase && log.meta.snapshot?.name === snapshot.meta.snapshot.name
@@ -181,7 +219,8 @@ async function* captureSnapshotResources(page, snapshot, options) {
   const log = logger('core:discovery');
   let { discovery, additionalSnapshots = [], ...baseSnapshot } = snapshot;
   let { capture, captureWidths, deviceScaleFactor, mobile, captureForDevices } = options;
-  let cookies = parseCookies(snapshot?.domSnapshot?.cookies);
+  let cookies = snapshot?.domSnapshot?.cookies || snapshot?.domSnapshot?.[0]?.cookies;
+  cookies = parseCookies(cookies);
 
   // iterate over device to trigger reqeusts and capture other dpr width
   async function* captureResponsiveAssets() {
@@ -209,16 +248,19 @@ async function* captureSnapshotResources(page, snapshot, options) {
   };
 
   // used to resize the using capture options
-  let resizePage = width => page.resize({
-    height: snapshot.minHeight,
-    deviceScaleFactor,
-    mobile,
-    width
-  });
+  let resizePage = width => {
+    page.network.intercept.currentWidth = width;
+    return page.resize({
+      height: snapshot.minHeight,
+      deviceScaleFactor,
+      mobile,
+      width
+    });
+  };
 
   // navigate to the url
   yield resizePage(snapshot.widths[0]);
-  yield page.goto(snapshot.url, { cookies });
+  yield page.goto(snapshot.url, { cookies, forceReload: discovery.captureResponsiveAssetsEnabled });
 
   // wait for any specified timeout
   if (snapshot.discovery.waitForTimeout && page.enableJavaScript) {
@@ -242,7 +284,8 @@ async function* captureSnapshotResources(page, snapshot, options) {
   // Running before page idle since this will trigger many network calls
   // so need to run as early as possible. plus it is just reading urls from dom srcset
   // which will be already loaded after navigation complete
-  if (discovery.captureSrcset) {
+  // Don't run incase of responsiveSnapshotCapture since we are running discovery for all widths so images will get captured in all required widths
+  if (!snapshot.responsiveSnapshotCapture && discovery.captureSrcset) {
     await page.insertPercyDom();
     yield page.eval('window.PercyDOM.loadAllSrcsetLinks()');
   }
@@ -261,6 +304,7 @@ async function* captureSnapshotResources(page, snapshot, options) {
         yield page.evaluate(execute?.beforeResize);
         yield waitForDiscoveryNetworkIdle(page, discovery);
         yield resizePage(width = widths[i + 1]);
+        if (snapshot.responsiveSnapshotCapture) { yield page.goto(snapshot.url, { cookies, forceReload: true }); }
         yield page.evaluate(execute?.afterResize);
       }
     }
@@ -379,8 +423,15 @@ export function createDiscoveryQueue(percy) {
               disableCache: snapshot.discovery.disableCache,
               allowedHostnames: snapshot.discovery.allowedHostnames,
               disallowedHostnames: snapshot.discovery.disallowedHostnames,
-              getResource: u => snapshot.resources.get(u) || cache.get(u),
-              saveResource: r => { snapshot.resources.set(r.url, r); if (!r.root) { cache.set(r.url, r); } }
+              getResource: (u, width = null) => {
+                let resource = snapshot.resources.get(u) || cache.get(u);
+                if (resource && Array.isArray(resource) && resource[0].root) {
+                  const rootResource = resource.find(r => r.widths?.includes(width));
+                  resource = rootResource || resource[0];
+                }
+                return resource;
+              },
+              saveResource: r => { snapshot.resources.set(r.url, r); cache.set(r.url, r); }
             }
           });
 

diff --git a/packages/core/src/network.js b/packages/core/src/network.js
@@ -354,7 +354,7 @@ async function sendResponseResource(network, request, session) {
   let send = (method, params) => network.send(session, method, params);
 
   try {
-    let resource = network.intercept.getResource(url);
+    let resource = network.intercept.getResource(url, network.intercept.currentWidth);
     network.log.debug(`Handling request: ${url}`, meta);
 
     if (!resource?.root && hostnameMatches(disallowedHostnames, url)) {
@@ -499,7 +499,7 @@ async function saveResponseResource(network, request) {
     }
   }
 
-  if (resource) {
+  if (resource && !resource.root) {
     network.intercept.saveResource(resource);
   }
 }

diff --git a/packages/core/src/page.js b/packages/core/src/page.js
@@ -68,13 +68,18 @@ export class Page {
   }
 
   // Go to a URL and wait for navigation to occur
-  async goto(url, { waitUntil = 'load', cookies } = {}) {
+  async goto(url, { waitUntil = 'load', cookies, forceReload, skipCookies = false } = {}) {
     this.log.debug(`Navigate to: ${url}`, this.meta);
 
+    if (forceReload) {
+      this.log.debug('Navigating to blank page', this.meta);
+      await this.goto('about:blank', { skipCookies: true });
+    }
+
     let navigate = async () => {
       const userPassedCookie = this.session.browser.cookies;
       // set cookies before navigation so we can default the domain to this hostname
-      if (userPassedCookie.length || cookies) {
+      if (!skipCookies && (userPassedCookie.length || cookies)) {
         let defaultDomain = hostname(url);
         cookies = this.mergeCookies(userPassedCookie, cookies);
 

diff --git a/packages/core/src/utils.js b/packages/core/src/utils.js
@@ -113,8 +113,8 @@ export function createResource(url, content, mimetype, attrs) {
 
 // Creates a root resource object with an additional `root: true` property. The URL is normalized
 // here as a convenience since root resources are usually created outside of asset discovery.
-export function createRootResource(url, content) {
-  return createResource(normalizeURL(url), content, 'text/html', { root: true });
+export function createRootResource(url, content, attrs = {}) {
+  return createResource(normalizeURL(url), content, 'text/html', { ...attrs, root: true });
 }
 
 // Creates a Percy CSS resource object.

diff --git a/packages/core/test/api.test.js b/packages/core/test/api.test.js
@@ -51,6 +51,7 @@ describe('API Server', () => {
       success: true,
       loglevel: 'info',
       config: PercyConfig.getDefaults(),
+      widths: { mobile: [], config: PercyConfig.getDefaults().snapshot.widths },
       build: {
         id: '123',
         number: 1,
@@ -69,6 +70,19 @@ describe('API Server', () => {
     });
   });
 
+  it('should return widths present in config and fetch widths for devices', async () => {
+    await percy.start();
+    percy.deviceDetails = [{ width: 390, devicePixelRatio: 2 }];
+    percy.config = PercyConfig.getDefaults({ snapshot: { widths: [1000] } });
+
+    await expectAsync(request('/percy/healthcheck')).toBeResolvedTo(jasmine.objectContaining({
+      widths: {
+        mobile: [390],
+        config: [1000]
+      }
+    }));
+  });
+
   it('can set config options via the /config endpoint', async () => {
     let expected = PercyConfig.getDefaults({ snapshot: { widths: [1000] } });
     await percy.start();
@@ -717,6 +731,24 @@ describe('API Server', () => {
       expect(headers['x-percy-core-version']).toEqual('0.0.1');
     });
 
+    it('can manipulate the config widths via /test/api/config', async () => {
+      let { widths, config } = await get('/percy/healthcheck');
+      expect(widths.config).toEqual([375, 1280]);
+      expect(widths.mobile).toEqual([]);
+
+      await post('/test/api/config', { config: [390], deferUploads: true });
+      ({ widths, config } = await get('/percy/healthcheck'));
+      expect(widths.config).toEqual([390]);
+      expect(config.snapshot.responsiveSnapshotCapture).toEqual(false);
+      expect(config.percy.deferUploads).toEqual(true);
+
+      await post('/test/api/config', { config: [375, 1280], mobile: [456], responsive: true });
+      ({ widths, config } = await get('/percy/healthcheck'));
+      expect(widths.mobile).toEqual([456]);
+      expect(config.snapshot.responsiveSnapshotCapture).toEqual(true);
+      expect(config.percy.deferUploads).toEqual(false);
+    });
+
     it('can make endpoints return server errors via /test/api/error', async () => {
       let { statusCode } = await req('/percy/healthcheck');
       expect(statusCode).toEqual(200);