[WIP] Retry e2e tests on beefier runners 🐄 #31953

iethree · 2023-06-28T14:48:59Z

Description

We've been seeing more and more e2e failures caused by things like slow api requests. This detects if an e2e job is a re-run and runs it on a 4vcpu runner.

You can't put conditionals in runs-on, so you have to add a separate step and use the output

codecov · 2023-06-28T15:33:50Z

Codecov Report

Patch and project coverage have no change.

Comparison is base (7c94ccc) 74.18% compared to head (7932b6f) 74.19%.

Additional details and impacted files

@@           Coverage Diff           @@
##           master   #31953   +/-   ##
=======================================
  Coverage   74.18%   74.19%           
=======================================
  Files        2905     2905           
  Lines      103334   103328    -6     
  Branches    12830    12831    +1     
=======================================
- Hits        76663    76660    -3     
+ Misses      20862    20857    -5     
- Partials     5809     5811    +2

Flag	Coverage Δ
back-end	`86.70% <ø> (-0.01%)`	⬇️
front-end	`60.20% <ø> (+<0.01%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

see 12 files with indirect coverage changes

☔ View full report in Codecov by Sentry.
📢 Do you have feedback about the report comment? Let us know in this issue.

deploysentinel · 2023-06-28T16:40:55Z

Current Cypress Test Results Summary

✅ 69 Passing - ❌ 1 Failing - ⚠️ 1 Flaky

Run may still be in progress, this comment will be updated as current testing workflow or job completes...

(Last updated on 06/28/2023 06:41:31pm UTC)

Run Details

Running Workflow E2E Tests on Github Actions

Commit: 78b2fb91713ef88205e29868bd967d68a783f118

Started: 06/28/2023 06:18:14pm UTC

❌ Failures

📄 e2e/test/scenarios/models/models.cy.spec.js • 1 Failure

Test Case Results

Test Case Last 7 days Failures Last 7 days Flakes

scenarios > models data picker transforms the data picker
Retry 4 • Retry 3 • Retry 2 • Retry 1 • Initial Attempt

Error: Timed out retrying after 4000ms: Unable to find an element with the text: Great ...

Timed out retrying after 4000ms: Unable to find an element with the text: Great Balls of Fire. This could be because the text is broken up by multiple elements. In this case, you can provide a function for your text matcher to make your matcher more flexible.

Ignored nodes: comments, <script />, <style />
<span
  class="PopoverContainer PopoverContainer--open popover tether-element tether-abutted tether-abutted-left tether-element-attached-top tether-element-attached-left tether-target-attached-bottom tether-target-attached-left tether-enabled"
  data-state="visible"
  data-testid="popover"
  style="top: 0px; left: 0px; position: absolute; transform: translateX(48px) translateY(256px) translateZ(0px);"
>
  <span
    style="z-index: 3;"
  >
    <div
      class="PopoverBody PopoverBody--withBackground PopoverBody--autoWidth"
      id="DataPopover"
    >
      <div
        class="css-m1a2da eyiqke90"
      >
        <div
          class="css-17sifsc edcfyzd6"
        >
          <input
            class="css-6hz806 edcfyzd5"
            data-testid="list-search-field"
            placeholder="Search for a table…"
            value=""
          />
          <button
            class="css-j8wedt edcfyzd3"
            data-testid="input-left-icon-button"
            disabled=""
            type="button"
          >
            <svg
              aria-label="search icon"
              class="Icon Icon-search"
              clip-rule="evenodd"
              fill="currentcolor"
              fill-rule="evenodd"
              height="16"
              role="img"
              viewBox="0 0 16 16"
              width="16"
              xmlns="http://www.w3.org/2000/svg"
            >
              <path
                d="M7 1.25a5.75 5.75 0 1 0 3.501 10.312l2.969 2.968a.75.75 0 1 0 1.06-1.06l-2.968-2.97A5.75 5.75 0 0 0 7 1.25zM2.75 7a4.25 4.25 0 1 1 8.5 0 4.25 4.25 0 0 1-8.5 0z"
              />
            </svg>
          </button>
        </div>
      </div>
      <div
        aria-label="grid"
        aria-readonly="true"
        class="ReactVirtualized__Grid ReactVirtualized__List text-brand"
        id="DatabaseSchemaPicker"
        role="grid"
        style="box-sizing: border-box; direction: ltr; height: 110px; position: relative; width: 300px; will-change: transform; overflow: hidden auto; outline: none;"
        tabindex="0"
      >
        <div
          class="ReactVirtualized__Grid__innerScrollContainer"
          role="rowgroup"
          style="width: auto; height: 110px; max-width: 300px; max-height: 110px; overflow: hidden; pointer-events: auto; position: relative;"
        >
          <div
            class="List-section List-section--togglable"
            style="height: 55px; left: 0px; position: absolute; top: 0px; width: 100%;"
          >
            <div
              class="List-section-header px2 py2 flex align-center hover-parent hover--opacity cursor-pointer"
            >
              <h3
                class="List-section-title text-wrap"
              >
                <div
                  class="css-1uoamx5 eyiqke97"
                >
                  <svg
                    aria-label="chevronleft icon"
                    class="Icon Icon-chevronleft"
                    fill="currentcolor"
                    height="16"
                    role="img"
                    viewBox="0 0 16 16"
                    width="16"
                    xmlns="http://www.w3.org/2000/svg"
                  >
                    <path
                      d="M11.53 1.47a.75.75 0 0 1 0 1.06L6.06 8l5.47 5.47a.75.75 0 1 1-1.06 1.06l-6-6a.75.75 0 0 1 0-1.06l6-6a.75.75 0 0 1 1.06 0z"
                    />
                  </svg>
                  <span
                    class="css-1dw2ruu eyiqke96"
                  >
                    Raw Data
                  </span>
                </div>
              </h3>
            </div>
          </div>
          <div
            class="List-section List-section--togglable"
            style="height: 55px; left: 0px; position: absolute; top: 55px; width: 100%;"
          >
            <div
              class="List-section-header px2 py2 flex align-center hover-parent hover--opacity cursor-pointer"
            >
              <span
                class="List-section-icon mr1 flex align-center"
              >
                <svg
                  aria-label="database icon"
                  class="Icon Icon-database Icon text-default"
                  fill="currentcolor"
                  height="18"
                  role="img"
                  viewBox="0 0 16 16"
                  width="18"
                  xmlns="http://www.w3.org/2000/svg"
                >
                  <path
                    clip-rule="evenodd"
                    d="M2.665 13.67A.75.75 0 0 1 2.25 13V3a.75.75 0 0 1 .415-.67l.002-.002.002-.001.008-.004.024-.011a7.465 7.465 0 0 1 .38-.164c.255-.103.623-.236 1.088-.37A14.023 14.023 0 0 1 8 1.25c1.583 0 2.903.264 3.831.529.465.133.833.266 1.088.368a7.5 7.5 0 0 1 .38.165l.024.011.008.004.003.001L13 3l.335-.671A.751.751 0 0 1 13.75 3v10a.75.75 0 0 1-.415.67l-.002.002-.002.001-.008.004-.024.011a7.443 7.443 0 0 1-.38.164 11.33 11.33 0 0 1-1.088.37A14.02 14.02 0 0 1 8 14.75a14.02 14.02 0 0 1-3.831-.529 11.327 11.327 0 0 1-1.088-.368 7.409 7.409 0 0 1-.38-.165l-.024-.011-.008-.004-.002-.001-.002-.001zM3.75 3.497c.207-.079.487-.177.831-.275A12.524 12.524 0 0 1 8 2.75c1.417 0 2.597.236 3.419.471.344.098.624.196.831.275v1.24l-.096.038c-.327.133-.642.26-1.106.388-.674.185-1.604.338-3.048.338s-2.374-.153-3.049-.338a8.97 8.97 0 0 1-1.106-.388l-.095-.039V3.496zm7.695 3.113c.289-.08.559-.171.805-.262v2.388l-.096.04c-.327.132-.642.26-1.106.387-.674.185-1.604.338-3.048.338s-2.374-.153-3.049-.338a8.97 8.97 0 0 1-1.201-.427V6.347c.246.091.516.182.805.262C5.378 6.834 6.444 7 8 7c1.556 0 2.621-.166 3.445-.391zM3.75 10.347v2.157c.207.079.487.177.831.275A12.52 12.52 0 0 0 8 13.25a12.52 12.52 0 0 0 3.419-.471c.344-.098.624-.196.831-.275v-2.157c-.246.09-.516.182-.805.261-.824.226-1.89.393-3.445.393-1.556 0-2.622-.167-3.445-.393a9.777 9.777 0 0 1-.805-.26z"
                    fill-rule="evenodd"
                  />
                </svg>
              </span>
              <h3
                class="List-section-title text-wrap"
              >
                Sample Database
              </h3>
            </div>
          </div>
        </div>
      </div>
    </div>
  </span>
</span>

0.60% (3)

3 / 500 runs
failed over last 7 days

1.60% (8)

8 / 500 runs
flaked over last 7 days

⚠️ Flakes

📄 e2e/test/scenarios/models/reproductions/29951-model-editor-results-metadata.cy.spec.js • 1 Flake

Test Case Results

Test Case	Last 7 days Failures	Last 7 days Flakes
issue 29951 should allow to run the model query after changing custom columns (#29951) Retry 3 • Retry 2 • Retry 1 • Initial Attempt	16% (80) 80 / 500 runs failed over last 7 days	11.40% (57) 57 / 500 runs flaked over last 7 days

View Detailed Build Results

iethree self-assigned this Jun 28, 2023

metabase-bot bot added the .Team/PixelPolice 👮 label Jun 28, 2023

retry e2e tests on beefier runners

7932b6f

iethree force-pushed the beefy-e2e-retries branch from b47ddbb to 7932b6f Compare June 28, 2023 16:04

iethree closed this Jun 28, 2023

iethree deleted the beefy-e2e-retries branch October 4, 2023 13:38

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[WIP] Retry e2e tests on beefier runners 🐄 #31953

[WIP] Retry e2e tests on beefier runners 🐄 #31953

iethree commented Jun 28, 2023 •

edited

Loading

codecov bot commented Jun 28, 2023 •

edited

Loading

deploysentinel bot commented Jun 28, 2023 •

edited

Loading

Test Case Results

Test Case Results

[WIP] Retry e2e tests on beefier runners 🐄 #31953

[WIP] Retry e2e tests on beefier runners 🐄 #31953

Conversation

iethree commented Jun 28, 2023 • edited Loading

Description

codecov bot commented Jun 28, 2023 • edited Loading

Codecov Report

deploysentinel bot commented Jun 28, 2023 • edited Loading

Current Cypress Test Results Summary

❌ Failures

Test Case Results

⚠️ Flakes

Test Case Results

iethree commented Jun 28, 2023 •

edited

Loading

codecov bot commented Jun 28, 2023 •

edited

Loading

deploysentinel bot commented Jun 28, 2023 •

edited

Loading