Index dcp13 in prod2 (#3843)

[achave11-ucsc]

#3843

Author

• PR title references issue
• PR title matches issue title (preceded by Fix: for bugs) _{or there is a good reason why they're different}
• Title of main commit references issue
• PR is connected to Zenhub issue and description links to issue

Author (reindex)

• Added r tag to commit title _{or this PR does not require reindexing}
• Added reindex label to PR _{or this PR does not require reindexing}

Author (freebies & chains)

• Freebies are blocked on this PR _{or there are no freebies in this PR}
• Freebies are referenced in commit titles _{or there are no freebies in this PR}
• This PR is blocked by previous PR in the chain _{or this PR is not chained to another PR}
• Added chain label to the blocking PR _{or this PR is not chained to another PR}

Author (upgrading)

• Documented upgrading of deployments in UPGRADING.rst _{or this PR does not require upgrading}
• Added u tag to commit title _{or this PR does not require upgrading}
• Added upgrade label to PR _{or this PR does not require upgrading}
• Added announcement to PR description _{or this PR does not require announcement}
• Added checklist items for additional operator tasks _{or this PR does not require additional tasks}

Author (requirements, before every review)

• Ran make requirements_update _{or this PR does not touch requirements*.txt, common.mk, Makefile and Dockerfile}
• Added R tag to commit title _{or this PR does not touch requirements*.txt}
• Added reqs label to PR _{or this PR does not touch requirements*.txt}

Author (before every review)

• make integration_test passes in personal deployment _{or this PR does not touch functionality that could break the IT}
• Rebased branch on develop, squashed old fixups

Primary reviewer (after approval)

• Commented in issue about demo expectations _{or labelled issue as no demo}
• Decided if PR can be labeled no sandbox
• PR title is appropriate as title of merge commit
• Moved ticket to Approved column
• Assigned PR to an operator

Operator (before pushing merge the commit)

• Checked reindex label and r commit title tag
• Checked that demo expectations are clear _{or issue is labeled as no demo}
• Rebased and squashed branch
• Sanity-checked history
• Pushed PR branch to Github
• Branch pushed to Gitlab and added sandbox label _{or PR is labeled no sandbox}
• Build passed in sandbox _{or PR is labeled no sandbox}
• Started reindex in sandbox _{or this PR does not require reindexing sandbox}
• Checked for failures in sandbox _{or this PR does not require reindexing sandbox}
• Added PR reference to merge commit title
• Collected commit title tags in merge commit title
• Moved linked issue to Merged column
• Pushed merge commit to Github

Operator (after pushing the merge commit)

• Made announcement requested by author _{or PR description does not contain an announcement}
• Moved freebies to Merged column _{or there are no freebies in this PR}
• Shortened the PR chain _{or this PR is not the base of another PR}
• Verified that N reviews labelling is accurate _{or this PR is authored by lead}
• Pushed merge commit to Gitlab _{or merge commit can be pushed later, with another PR}
• Deleted PR branch from Github and Gitlab
• Build passes on Gitlab
• Moved issues to prod or Merged prod _{or this PR does not represent a promotion}

Operator (reindex)

• Started reindex in target deployment _{or this PR does not require reindexing}
• Index dcp13 catalog only
• Checked for and triaged indexing failures _{or this PR does not require reindexing}
• Emptied fail queues in target deployment _{or this PR does not require reindexing}
• Filed backport PR _{or this PR does not represent a hotfix}

Operator

• Run IT locally
• Unassigned PR

[codecov]

Codecov Report

Merging #3867 (4e2037a) into develop (3f48630) will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff            @@
##           develop    #3867   +/-   ##
========================================
  Coverage    81.70%   81.70%           
========================================
  Files          125      125           
  Lines        15050    15050           
========================================
  Hits         12297    12297           
  Misses        2753     2753           

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 3f48630...4e2037a. Read the comment docs.

[coveralls]

Coverage remained the same at 81.924% when pulling 4e2037a on issues/achave11/3843-prod2-dcp13 into 3f48630 on develop.

[hannes-ucsc]

As agreed during demo, we'll deal with the duplication using a dictionary, in a separate PR. They way this is currently structured, it is very difficult to see which projects where updated. Please create three commits:

The first commit should rename dcp2_sources to dcp12_sources.

The second commit duplicates the dcp12 catalog and the dcp12_sources variable to dcp13 and dcp13_sources respectively.

The third commit adds the _dcp13 snapshots from the spreadsheet.

Also please make sure the list literals defining the sources are sorted by project UUID.

[hannes-ucsc]

Thanks for the quick turn around.

dcp1_sources wasn't sorted

remember to differentiate split commits in the message of the commit body

I'll fix the sorting and add the dictionary approach I mentioned in PL.

[achave11-ucsc]

IT passed with the exception of two manifest request which failed because of TDR slowness HumanCellAtlas/dcp2#57