Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[wip] persistent worker integration into 2.x #3

Draft
wants to merge 69 commits into
base: master
Choose a base branch
from
Draft
Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
69 commits
Select commit Hold shift + click to select a range
edf2211
Implement local resources for workers (#1282)
luxe Oct 17, 2023
97e3b90
build: override grpc dependencies with our dependencies
jasonschroeder-sfdc Oct 16, 2023
96f239d
chore(deps): bump protobuf runtime to 3.19.1
jasonschroeder-sfdc Oct 16, 2023
af3f34e
chore(deps) add transitive dependencies
jasonschroeder-sfdc Oct 16, 2023
380f8a1
feat: add Proto reflection service to shard worker
jasonschroeder-sfdc Oct 16, 2023
7e7979d
fixup! build: override grpc dependencies with our dependencies
jasonschroeder-sfdc Oct 16, 2023
1f9d01f
fixup! chore(deps) add transitive dependencies
jasonschroeder-sfdc Oct 16, 2023
578589f
Bug: Fix Blocked thread in WriteStreamObserver Caused by CASFile Writ…
amishra-u Oct 24, 2023
dfa5937
Pin the Java toolchain to `remotejdk_17` (#1509)
stefanobaghino Oct 24, 2023
f6459d1
docs: add markdown language specifiers for code blocks
jasonschroeder-sfdc Oct 18, 2023
018e177
Support OutputPaths in OutputDirectory
werkt Oct 25, 2023
8b37013
Permit Absolute Symlink Targets with configuration
werkt Oct 26, 2023
df9ce1d
chore: update bazel to 6.4.0 (#1513)
jasonschroeder-sfdc Oct 29, 2023
bd740c9
Rename instance types (#1514)
luxe Oct 30, 2023
2a61f77
Create SymlinkNode outputs during upload (#1515)
werkt Oct 30, 2023
76c2657
feat: Implement CAS lease extension (#1455)
amishra-u Oct 31, 2023
cfa2e18
Bump org.json:json from 20230227 to 20231013 in /admin/main (#1516)
dependabot[bot] Oct 31, 2023
ff00c8f
Re-add missing graceful shutdown functionality (#1520)
80degreeswest Nov 1, 2023
afb0603
Technically correct to unwrap EE on lock failure
werkt Jul 14, 2023
9b5ec43
Bump rules_oss_audit and patch for py3.11
werkt Oct 17, 2023
f9ef75a
Prevent healthStatusManager NPE on start failure
werkt Oct 17, 2023
20512f6
Consistent check for publicName presence
werkt Nov 1, 2023
654032e
Read through external with query THROUGH=true
werkt Nov 1, 2023
b4359c5
Add --port option to worker
werkt Nov 1, 2023
5195809
Restore worker --root cmdline specification
werkt Nov 1, 2023
938c789
Make bf-executor small blob names consistent
werkt Nov 1, 2023
87face1
Configured output size operation failure
werkt Nov 2, 2023
58faec9
Restore abbrev port as -p
werkt Nov 2, 2023
cf6fc58
Update zstd-jni for latest version
jerrymarino Oct 31, 2023
6bc70e1
Attempt to resolve windows stamping
werkt Nov 3, 2023
b226725
Bug: Fix workerSet update logic for RemoteCasWriter
amishra-u Nov 2, 2023
751ac90
Detail storage requirements
werkt Nov 4, 2023
2bf3eae
Fix worker execution env title
werkt Nov 4, 2023
b6bddff
Add storage example descriptions
werkt Nov 4, 2023
c490925
Check for context cancelled before responding to error (#1526)
justinwon777 Nov 6, 2023
2a51c31
chore(deps): bump com.google.errorprone:error-prone
jasonschroeder-sfdc Oct 16, 2023
7ea1a9f
Worker name execution properties matching
werkt Nov 8, 2023
a4822c1
updates
luxe Apr 21, 2023
f1ea9b5
updates
luxe Apr 21, 2023
ccf763d
updates
luxe Apr 21, 2023
b7f5661
updates
luxe Apr 21, 2023
72b40ae
updates
luxe Apr 23, 2023
14c759b
Update ShardWorkerContext.java
luxe Nov 8, 2023
ca9bb92
Update ShardWorkerContext.java
luxe Nov 8, 2023
10e68c4
[execution] allow tmpfs and cgroups enforcement
werkt Nov 9, 2023
35883a4
Release resources when not keeping an operation (#1535)
werkt Nov 9, 2023
9d80f4e
Update queues.md
werkt Nov 9, 2023
aac33b6
Implement custom label header support for Grpc metrics interceptor (#…
rastenis Nov 10, 2023
f9882f7
Specify direct guava dependency usage (#1538)
werkt Nov 11, 2023
69e0248
Update lombok dependency for jdk21 (#1540)
werkt Nov 11, 2023
339aa13
Reorganize DequeueMatchEvaluator (#1537)
werkt Nov 11, 2023
025305a
Upgrade com_google_protobuf for jvm compatibility (#1539)
werkt Nov 11, 2023
f720909
Create buildfarm-worker-base-build-and-deploy.yml (#1534)
80degreeswest Nov 11, 2023
b7daba3
Add base image generation scripts (#1532)
80degreeswest Nov 11, 2023
dcff4f0
Fix buildfarm-worker-base-build-and-deploy.yml (#1541)
80degreeswest Nov 11, 2023
52318f8
Add public buildfarm image generation actions (#1542)
80degreeswest Nov 14, 2023
e343393
Update base image building action (#1544)
80degreeswest Nov 16, 2023
dae7f78
Add release image generation action (#1545)
80degreeswest Nov 16, 2023
b01889d
Limit workflow to canonical repository (#1547)
werkt Nov 16, 2023
dcee798
Check for "cores" exec property as min-cores match (#1548)
werkt Nov 16, 2023
221eae9
Consider output_* as relative to WD (#1550)
werkt Nov 19, 2023
021071b
Integrate persistent workers into shard Worker via PersistentExecutor
wiwa Oct 27, 2023
6bd2ab3
slight cleanup
wiwa Feb 1, 2023
4b748d4
Formatting
wiwa Feb 1, 2023
2ac7a3e
Fix static analysis
wiwa Feb 1, 2023
957c4b5
String#strip() -> String#trim()
wiwa Feb 1, 2023
9ff3134
Rename and add tests for util/InputIndexer and init PersistentExecuto…
wiwa Oct 17, 2023
9ca026e
Add test utils and tests for ProtoCoordinator
wiwa Oct 17, 2023
20cd93e
Address Feedback
wiwa Nov 19, 2023
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Add release image generation action (buildfarm#1545)
  • Loading branch information
80degreeswest authored Nov 16, 2023
commit dae7f78eda808b745d1e0bc59cfdfb4a3ba67681
4 changes: 4 additions & 0 deletions .bazelci/presubmit.yml
Original file line number Diff line number Diff line change
Expand Up @@ -41,6 +41,8 @@ tasks:
name: "Unit Tests"
build_targets:
- "..."
build_flags:
- "--build_tag_filters=-container"
test_flags:
- "--test_tag_filters=-integration,-redis"
test_targets:
Expand All @@ -49,6 +51,8 @@ tasks:
name: "Unit Tests"
build_targets:
- "..."
build_flags:
- "--build_tag_filters=-container"
test_flags:
- "--test_tag_filters=-integration,-redis"
test_targets:
Expand Down
4 changes: 2 additions & 2 deletions .github/workflows/buildfarm-images-build-and-deploy.yml
Original file line number Diff line number Diff line change
Expand Up @@ -23,8 +23,8 @@ jobs:

- name: Build Server Image
id: buildAndPushServerImage
run: bazel run public_push_buildfarm-server
run: bazel run public_push_buildfarm-server --define release_version=latest

- name: Build Worker Image
id: buildAndPushWorkerImage
run: bazel run public_push_buildfarm-worker
run: bazel run public_push_buildfarm-worker --define release_version=latest
29 changes: 29 additions & 0 deletions .github/workflows/buildfarm-release-build-and-deploy.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,29 @@
name: Build and Push Buildfarm Releases

on:
release:
types: [published]

jobs:
build:
name: Build Buildfarm Images
runs-on: ubuntu-latest
steps:
- uses: bazelbuild/setup-bazelisk@v2

- name: Checkout
uses: actions/checkout@v3

- name: Login to Bazelbuild Docker Hub
uses: docker/login-action@v3
with:
username: ${{ secrets.BAZELBUILD_DOCKERHUB_USERNAME }}
password: ${{ secrets.BAZELBUILD_DOCKERHUB_TOKEN }}

- name: Build Server Image
id: buildAndPushServerImage
run: bazel run public_push_buildfarm-server --define release_version=${{ github.event.release.tag_name }}

- name: Build Worker Image
id: buildAndPushWorkerImage
run: bazel run public_push_buildfarm-worker --define release_version=${{ github.event.release.tag_name }}
4 changes: 2 additions & 2 deletions BUILD
Original file line number Diff line number Diff line change
Expand Up @@ -204,7 +204,7 @@ container_push(
image = ":buildfarm-server",
registry = "index.docker.io",
repository = "bazelbuild/buildfarm-server",
tag = "latest",
tag = "$(release_version)",
tags = ["container"],
)

Expand All @@ -214,6 +214,6 @@ container_push(
image = ":buildfarm-shard-worker",
registry = "index.docker.io",
repository = "bazelbuild/buildfarm-worker",
tag = "latest",
tag = "$(release_version)",
tags = ["container"],
)