Add latest changes from gitlab-org/gitlab@master

.gitlab/issue_templates/QA failure.md

@@ -31,6 +31,15 @@ Attach the screenshot and HTML snapshot of the page from the job's artifacts:
 1. Open the `gitlab-qa-run-2020-*/gitlab-{ce,ee}-qa-*/{,ee}/{api,browser_ui}/<path to failed test>` folder.
 1. Select the `.png` and `.html` files that appears in the job logs (look for `HTML screenshot: /path/to/html/page.html` / `Image screenshot: `/path/to/html/page.png`).
 1. Drag and drop them here.
+
+Note: You don't need to include a screenshot if the information it contains can be included as text. Include the text instead.
+E.g., error 500/404, "Retry later" errors, etc.
+
+If you include multiple screenshots it can be helpful to hide all but the first in a details/summary element, to avoid excessive scrolling:
+
+<details><summary>Expand for screenshot</summary>
+  drag and drop the screenshot here
+</details>
 -->
 
 ### Possible fixes

app/models/jira_import_state.rb

@@ -0,0 +1,79 @@
+# frozen_string_literal: true
+
+class JiraImportState < ApplicationRecord
+  include AfterCommitQueue
+
+  self.table_name = 'jira_imports'
+
+  STATUSES = { initial: 0, scheduled: 1, started: 2, failed: 3, finished: 4 }.freeze
+
+  belongs_to :project
+  belongs_to :user
+  belongs_to :label
+
+  validates :project, presence: true
+  validates :jira_project_key, presence: true
+  validates :jira_project_name, presence: true
+  validates :jira_project_xid, presence: true
+
+  validates :project, uniqueness: {
+    conditions: -> { where.not(status: STATUSES.values_at(:failed, :finished)) },
+    message: _('Cannot have multiple Jira imports running at the same time')
+  }
+
+  state_machine :status, initial: :initial do
+    event :schedule do
+      transition initial: :scheduled
+    end
+
+    event :start do
+      transition scheduled: :started
+    end
+
+    event :finish do
+      transition started: :finished
+    end
+
+    event :do_fail do
+      transition [:initial, :scheduled, :started] => :failed
+    end
+
+    after_transition initial: :scheduled do |state, _|
+      state.run_after_commit do
+        job_id = Gitlab::JiraImport::Stage::StartImportWorker.perform_async(project.id)
+        state.update(jid: job_id) if job_id
+      end
+    end
+
+    after_transition any => :finished do |state, _|
+      if state.jid.present?
+        Gitlab::SidekiqStatus.unset(state.jid)
+
+        state.update_column(:jid, nil)
+      end
+    end
+
+    # Supress warning:
+    # both JiraImportState and its :status machine have defined a different default for "status".
+    # although both have same value but represented in 2 ways: integer(0) and symbol(:initial)
+    def owner_class_attribute_default
+      'initial'
+    end
+  end
+
+  enum status: STATUSES
+
+  def in_progress?
+    scheduled? || started?
+  end
+
+  def refresh_jid_expiration
+    return unless jid
+
+    Gitlab::SidekiqStatus.set(jid, StuckImportJobsWorker::IMPORT_JOBS_EXPIRATION)
+  end
+
+  def self.jid_by(project_id:, status:)
+    select(:jid).with_status(status).find_by(project_id: project_id)
+  end
+end

app/models/project.rb

@@ -314,6 +314,7 @@ class Project < ApplicationRecord
   has_one :pages_metadatum, class_name: 'ProjectPagesMetadatum', inverse_of: :project
 
   has_many :import_failures, inverse_of: :project
+  has_many :jira_imports, -> { order 'jira_imports.created_at' }, class_name: 'JiraImportState', inverse_of: :project
 
   has_many :daily_report_results, class_name: 'Ci::DailyReportResult'
 
@@ -2424,6 +2425,10 @@ def environments_for_scope(scope)
     environments.where("name LIKE (#{::Gitlab::SQL::Glob.to_like(quoted_scope)})") # rubocop:disable GitlabSecurity/SqlInjection
   end
 
+  def latest_jira_import
+    jira_imports.last
+  end
+
   private
 
   def find_service(services, name)

doc/development/testing_guide/end_to_end/index.md

@@ -30,7 +30,7 @@ You can find these nightly pipelines at `https://gitlab.com/gitlab-org/quality/s
 #### Using the `package-and-qa` job
 
 It is possible to run end-to-end tests for a merge request, eventually being run in
-a pipeline in the [`gitlab-qa`](https://gitlab.com/gitlab-org/gitlab-qa/) project,
+a pipeline in the [`gitlab-qa-mirror`](https://gitlab.com/gitlab-org/gitlab-qa-mirror/) project,
 by triggering the `package-and-qa` manual action in the `test` stage (not
 available for forks).
 
@@ -49,8 +49,8 @@ pipelines.
 
 ```mermaid
 graph LR
-    A1 -.->|1. Triggers an omnibus-gitlab pipeline and wait for it to be done| A2
-    B2[`Trigger-qa` stage<br>`Trigger:qa-test` job] -.->|2. Triggers a gitlab-qa pipeline and wait for it to be done| A3
+    A1 -.->|1. Triggers an omnibus-gitlab-mirror pipeline and wait for it to be done| A2
+    B2[`Trigger-qa` stage<br>`Trigger:qa-test` job] -.->|2. Triggers a gitlab-qa-mirror pipeline and wait for it to be done| A3
 
 subgraph "gitlab-foss/gitlab pipeline"
     A1[`test` stage<br>`package-and-qa` job]
@@ -60,23 +60,23 @@ subgraph "omnibus-gitlab pipeline"
     A2[`Trigger-docker` stage<br>`Trigger:gitlab-docker` job] -->|once done| B2
     end
 
-subgraph "gitlab-qa pipeline"
+subgraph "gitlab-qa-mirror pipeline"
     A3>QA jobs run] -.->|3. Reports back the pipeline result to the `package-and-qa` job<br>and post the result  on the original commit tested| A1
     end
 ```
 
 1. Developer triggers a manual action, that can be found in CE / EE merge
    requests. This starts a chain of pipelines in multiple projects.
 
-1. The script being executed triggers a pipeline in [Omnibus GitLab][omnibus-gitlab]
+1. The script being executed triggers a pipeline in [Omnibus GitLab Mirror][omnibus-gitlab-mirror]
    and waits for the resulting status. We call this a _status attribution_.
 
 1. GitLab packages are being built in the [Omnibus GitLab][omnibus-gitlab]
    pipeline. Packages are then pushed to its Container Registry.
 
 1. When packages are ready, and available in the registry, a final step in the
    [Omnibus GitLab][omnibus-gitlab] pipeline, triggers a new
-   GitLab QA pipeline (those with access can view them at `https://gitlab.com/gitlab-org/gitlab-qa/pipelines`). It also waits for a resulting status.
+   GitLab QA pipeline (those with access can view them at `https://gitlab.com/gitlab-org/gitlab-qa-mirror/pipelines`). It also waits for a resulting status.
 
 1. GitLab QA pulls images from the registry, spins-up containers and runs tests
    against a test environment that has been just orchestrated by the `gitlab-qa`
@@ -86,12 +86,12 @@ subgraph "gitlab-qa pipeline"
    propagated upstream, through Omnibus, back to the CE / EE merge request.
 
 Please note, we plan to [add more specific information](https://gitlab.com/gitlab-org/quality/team-tasks/issues/156)
-about the tests included in each job/scenario that runs in `gitlab-qa`.
+about the tests included in each job/scenario that runs in `gitlab-qa-mirror`.
 
 #### With Pipeline for Merged Results
 
 In a Pipeline for Merged Results, the pipeline runs on a new ref that contains the merge result of the source and target branch.
-However, this ref is not available to the `gitlab-qa` pipeline.
+However, this ref is not available to the `gitlab-qa-mirror` pipeline.
 
 For this reason, the end-to-end tests on a Pipeline for Merged Results would use the head of the merge request source branch.
 
@@ -112,7 +112,7 @@ C --> D["Pipeline for merged results"]
 ##### Running custom tests
 
 The [existing scenarios](https://gitlab.com/gitlab-org/gitlab-qa/blob/master/docs/what_tests_can_be_run.md)
-that run in the downstream `gitlab-qa` pipeline include many tests, but there are times when you might want to run a
+that run in the downstream `gitlab-qa-mirror` pipeline include many tests, but there are times when you might want to run a
 test or a group of tests that are different than the groups in any of the existing scenarios.
 
 For example, when we [dequarantine](https://about.gitlab.com/handbook/engineering/quality/guidelines/debugging-qa-test-failures/#dequarantining-tests)
@@ -197,6 +197,7 @@ you can find an issue you would like to work on in
 [the `gitlab-qa` issue tracker][gitlab-qa-issues].
 
 [omnibus-gitlab]: https://gitlab.com/gitlab-org/omnibus-gitlab
+[omnibus-gitlab-mirror]: https://gitlab.com/gitlab-org/omnibus-gitlab-mirror
 [gitlab-qa]: https://gitlab.com/gitlab-org/gitlab-qa
 [gitlab-qa-readme]: https://gitlab.com/gitlab-org/gitlab-qa/tree/master/README.md
 [review-apps]: ../review_apps.md

doc/subscriptions/index.md

@@ -64,7 +64,7 @@ source projects, GitLab grants access to **Gold** features for all GitLab.com
 
 #### Self-managed
 
-A self-managed subscription uses a hybrid model. You pay for a subscription according to the maximum number of users enabled during the subscription period. For instances that aren't air-gapped or on a closed network, the maximum number of simultaneous users in the self-managed installation is checked each quarter, using [Seat Link](#seat-link).
+A self-managed subscription uses a hybrid model. You pay for a subscription according to the maximum number of users enabled during the subscription period. For instances that aren't offline or on a closed network, the maximum number of simultaneous users in the self-managed installation is checked each quarter, using [Seat Link](#seat-link).
 
 Every occupied seat, whether by person, job, or bot is counted in the subscription, with the following exceptions:
 
@@ -255,7 +255,7 @@ Seat Link provides **only** the following information to GitLab:
 - Historical maximum user count
 - Active users count
 
-For air-gapped or closed network customers, the existing [true-up model](#users-over-license) will be used. Prorated charges are not possible without user count data.
+For offline or closed network customers, the existing [true-up model](#users-over-license) will be used. Prorated charges are not possible without user count data.
 
 <details>
 <summary>Click here to view example content of a Seat Link POST request.</summary>

doc/topics/airgap/index.md

@@ -1,7 +1,7 @@
-# Air-gapped GitLab
+# Offline GitLab
 
-Computers in an air-gapped network are isolated from the public internet as a security measure.
-This page lists all the information available for running GitLab in an air-gapped environment.
+Computers in an offline environment are isolated from the public internet as a security measure. This
+page lists all the information available for running GitLab in an offline environment.
 
 ## Quick start
 
@@ -14,21 +14,21 @@ Follow these best practices to use GitLab's features in an offline environment:
 
 - [Operating the GitLab Secure scanners in an offline environment](../../user/application_security/offline_deployments/index.md).
 
-## Loading Docker images onto your air-gapped host
+## Loading Docker images onto your offline host
 
 To use many GitLab features, including
 [security scans](../../user/application_security/index.md#working-in-an-offline-environment)
 and [Auto Devops](../autodevops/), the GitLab Runner must be able to fetch the
 relevant Docker images.
 
 The process for making these images available without direct access to the public internet
-involves downloading the images then packaging and transferring them to the air-gapped host.
-Here's an example of such a transfer:
+involves downloading the images then packaging and transferring them to the offline host. Here's an
+example of such a transfer:
 
 1. Download Docker images from public internet.
 1. Package Docker images as tar archives.
-1. Transfer images to air-gapped environment.
-1. Load transferred images into air-gapped Docker registry.
+1. Transfer images to offline environment.
+1. Load transferred images into offline Docker registry.
 
 ### Example image packager script
 
@@ -51,7 +51,7 @@ done
 
 ### Example image loader script
 
-This example loads the images from a bastion host to an air-gapped host. In certain configurations,
+This example loads the images from a bastion host to an offline host. In certain configurations,
 physical media may be needed for such a transfer:
 
 ```sh

doc/topics/airgap/quick_start_guide.md

@@ -1,4 +1,4 @@
-# Getting started with an air-gapped GitLab Installation
+# Getting started with an offline GitLab Installation
 
 This is a step-by-step guide that helps you install, configure, and use a self-managed GitLab
 instance entirely offline.

doc/user/application_security/container_scanning/index.md

@@ -180,7 +180,7 @@ using environment variables.
 | `CLAIR_DB_CONNECTION_STRING`   | This variable represents the [connection string](https://www.postgresql.org/docs/9.3/libpq-connect.html#AEN39692) to the [PostgreSQL server hosting the vulnerabilities definitions](https://hub.docker.com/r/arminc/clair-db) database and **shouldn't be changed** unless you're running the image locally as described in the [Running the standalone Container Scanning Tool](#running-the-standalone-container-scanning-tool) section. The host value for the connection string must match the [alias](https://gitlab.com/gitlab-org/gitlab/-/blob/898c5da43504eba87b749625da50098d345b60d6/lib/gitlab/ci/templates/Security/Container-Scanning.gitlab-ci.yml#L23) value of the `Container-Scanning.gitlab-ci.yml` template file, which defaults to `clair-vulnerabilities-db`. | `postgresql://postgres:password@clair-vulnerabilities-db:5432/postgres?sslmode=disable&statement_timeout=60000` |
 | `CI_APPLICATION_REPOSITORY`    | Docker repository URL for the image to be scanned.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 | `$CI_REGISTRY_IMAGE/$CI_COMMIT_REF_SLUG`                                                                        |
 | `CI_APPLICATION_TAG`           | Docker respository tag for the image to be scanned.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                | `$CI_COMMIT_SHA`                                                                                                |
-| `CLAIR_DB_IMAGE`               | The Docker image name and tag for the [PostgreSQL server hosting the vulnerabilities definitions](https://hub.docker.com/r/arminc/clair-db). It can be useful to override this value with a specific version, for example, to provide a consistent set of vulnerabilities for integration testing purposes, or to refer to a locally hosted vulnerabilities database for an on-premise air-gapped installation.                                                                                                                                                                                                                                                                                                                                                                      | `arminc/clair-db:latest`                                                                                        |
+| `CLAIR_DB_IMAGE`               | The Docker image name and tag for the [PostgreSQL server hosting the vulnerabilities definitions](https://hub.docker.com/r/arminc/clair-db). It can be useful to override this value with a specific version, for example, to provide a consistent set of vulnerabilities for integration testing purposes, or to refer to a locally hosted vulnerabilities database for an on-premise offline installation.                                                                                                                                                                                                                                                                                                                                                                      | `arminc/clair-db:latest`                                                                                        |
 | `CLAIR_DB_IMAGE_TAG`           | (**DEPRECATED - use `CLAIR_DB_IMAGE` instead**) The Docker image tag for the [PostgreSQL server hosting the vulnerabilities definitions](https://hub.docker.com/r/arminc/clair-db). It can be useful to override this value with a specific version, for example, to provide a consistent set of vulnerabilities for integration testing purposes.                                                                                                                                                                                                                                                                                                                                                                                                                                   | `latest`                                                                                                        |
 | `DOCKERFILE_PATH`              | The path to the `Dockerfile` to be used for generating remediations. By default, the scanner will look for a file named `Dockerfile` in the root directory of the project, so this variable should only be configured if your `Dockerfile` is in a non-standard location, such as a subdirectory. See [Solutions for vulnerabilities](#solutions-for-vulnerabilities-auto-remediation) for more details.                                                                                                                                                                                                                                                                                                                                                                          | `Dockerfile`                                                                                                    |
 | `ADDITIONAL_CA_CERT_BUNDLE`   | Bundle of CA certs that you want to trust. | "" |
@@ -210,7 +210,7 @@ If you want to whitelist specific vulnerabilities, you'll need to:
    in the [whitelist example file](https://github.com/arminc/clair-scanner/blob/v12/example-whitelist.yaml).
 1. Add the `clair-whitelist.yml` file to the Git repository of your project.
 
-### Running Container Scanning in an offline environment deployment
+### Running Container Scanning in an offline environment
 
 Container Scanning can be executed on an offline GitLab Ultimate installation by using the following process:
 

doc/user/application_security/dast/index.md

@@ -461,7 +461,7 @@ dast:
 The DAST job does not require the project's repository to be present when running, so by default
 [`GIT_STRATEGY`](../../../ci/yaml/README.md#git-strategy) is set to `none`.
 
-## Running DAST in an offline environment deployment
+## Running DAST in an offline environment
 
 DAST can be executed on an offline GitLab Ultimate installation by using the following process: