From b16b9337d12dbaf4392344378a8abd12dfc6da17 Mon Sep 17 00:00:00 2001
From: "release-please[bot]"
<55107282+release-please[bot]@users.noreply.github.com>
Date: Mon, 28 Oct 2024 16:00:11 -0400
Subject: [PATCH 01/19] chore(main): release 2.49.1-SNAPSHOT (#3317)
:robot: I have created a release *beep* *boop*
---
2.49.1-SNAPSHOT
### Updating meta-information for bleeding-edge SNAPSHOT release.
grpc/grpc-java (io.grpc:grpc-core)
###
[`v1.68.1`](https://redirect.github.com/grpc/grpc-java/releases/tag/v1.68.1)
##### gRPC Java 1.68.1 Release Notes
v1.68.0 was a mistake. This is the first release of version 1.68.x
##### Bug Fixes
- xds: Fix NullPointerException introduced in "Fix load reporting when
pick first is used for locality-routing"
([#11553](https://redirect.github.com/grpc/grpc-java/issues/11553)).
This was in 1.67.1 but not 1.68.0
##### Behavior Changes
- core: JSON parsing rejects duplicate keys in objects
([#11575](https://redirect.github.com/grpc/grpc-java/issues/11575))
([`4be69e3`](https://redirect.github.com/grpc/grpc-java/commit/4be69e3f8)).
This is the existing behavior in C core. Duplicate keys in objects are
dangerous as which value takes effect is undefined. Previously, the last
value was used
- okhttp: Detect transport executors with no remaining threads
([#11503](https://redirect.github.com/grpc/grpc-java/issues/11503))
([`3a6be9c`](https://redirect.github.com/grpc/grpc-java/commit/3a6be9ca1)).
The transport uses two threads, but one is on-demand. If the executor
provided to `builder.transportExecutor()` runs out of threads (e.g., it
is a fixed-size thread pool), *all* transports can be wedged, unable to
run on-demand tasks, until keepalive kills one of them. Two threads are
now used when handshaking a new transport, and the transport will time
out after 1 second with “Timed out waiting for second handshake thread”
if two threads are unavailable
- gcp-csm-o11y: Get `mesh_id` value from `CSM_MESH_ID` environment
variable, instead of getting it from bootstrap file
([`84d30af`](https://redirect.github.com/grpc/grpc-java/commit/84d30afad))
##### Improvements
- New grpc-context-override-opentelemetry artifact
([#11523](https://redirect.github.com/grpc/grpc-java/issues/11523))
([`782a44a`](https://redirect.github.com/grpc/grpc-java/commit/782a44ad6))
([#11599](https://redirect.github.com/grpc/grpc-java/issues/11599))
([`e59ae5f`](https://redirect.github.com/grpc/grpc-java/commit/e59ae5fad)).
This is a `io.grpc.Context` storage override to store its state in
`io.opentelemetry.context.Context`. Libraries should not add a
dependency on this artifact, as applications can only have one storage
override in their classpath
- New grpc-s2a artifact. It is a transport that offloads the handshake
similar to ALTS, but for TLS. It provides
`io.grpc.s2a.S2AChannelCredentials`
- api: Enhance name resolver \`ResolutionResult\` to hold addresses or
error so the single listener API *onResult2* is used to convey both
success and error cases for name resolution
([#11330](https://redirect.github.com/grpc/grpc-java/issues/11330))
([`1ded8af`](https://redirect.github.com/grpc/grpc-java/commit/1ded8aff8))
- core: Handle NameResolver/LoadBalancer exceptions when panicking
([`b692b9d`](https://redirect.github.com/grpc/grpc-java/commit/b692b9d26)).
This expands the class of bugs that will fail RPCs with the panic error,
versus some undefined behavior
- core: Use the default service config in case of initial name resolver
address resolution error
([#11577](https://redirect.github.com/grpc/grpc-java/issues/11577))
([`fa26a8b`](https://redirect.github.com/grpc/grpc-java/commit/fa26a8bc5))
- core: `StreamTracer.inboundMessageRead()` now reports uncompressed
message size when the message does not need compression
([#11598](https://redirect.github.com/grpc/grpc-java/issues/11598))
([`2aae68e`](https://redirect.github.com/grpc/grpc-java/commit/2aae68e11)).
Previously it always reported `-1` (unknown)
- netty: Avoid TCP_USER_TIMEOUT warning when explicitly specifying a
non-epoll channel type to use
([#11564](https://redirect.github.com/grpc/grpc-java/issues/11564))
([`62f4098`](https://redirect.github.com/grpc/grpc-java/commit/62f409810))
- okhttp: Don't warn about missing Conscrypt
([`6f35422`](https://redirect.github.com/grpc/grpc-java/commit/6f3542297)).
This is especially helpful when using TLS but not running on Android
- android: For `UdsChannelBuilder`, use fake IP instead of localhost
([`a908b5e`](https://redirect.github.com/grpc/grpc-java/commit/a908b5e40)).
This avoids an unnecessary DNS lookup
- xds: Add xDS node ID in select control plane errors to enable
cross-referencing with control plane logs when debugging
([`f3cf7c3`](https://redirect.github.com/grpc/grpc-java/commit/f3cf7c3c7))
- xds: Enhanced how ADS stream terminations are handled, specifically
addressing cases where a response has or hasn't been received
(#2e9c3e19f)
- binder: Update status code documentation for Android 11's package
visibility rules.
([#11551](https://redirect.github.com/grpc/grpc-java/issues/11551))
([`99be6e9`](https://redirect.github.com/grpc/grpc-java/commit/99be6e985))
- binder: Update binderDied() error description to spell out the
possibilities for those unfamiliar with Android internals.
([#11628](https://redirect.github.com/grpc/grpc-java/issues/11628))
([`46c1b38`](https://redirect.github.com/grpc/grpc-java/commit/46c1b387f))
- example-gauth: Use application default creds instead of file argument
([#11595](https://redirect.github.com/grpc/grpc-java/issues/11595))
([`94a0a0d`](https://redirect.github.com/grpc/grpc-java/commit/94a0a0d1c))
##### Dependencies
- Updated protobuf-java to 3.25.5. This helps avoid CVE-2024-7254
([`2ff837a`](https://redirect.github.com/grpc/grpc-java/commit/2ff837ab6))
Thanks to:\
[@Juneezee](https://redirect.github.com/Juneezee)\
[@lgalfaso](https://redirect.github.com/lgalfaso)\
[@bestbeforetoday](https://redirect.github.com/bestbeforetoday)\
[@hlx502](https://redirect.github.com/hlx502)\
[@JoeCqupt](https://redirect.github.com/JoeCqupt)
###
[`v1.68.0`](https://redirect.github.com/grpc/grpc-java/releases/tag/v1.68.0):
MISTAKE
This was supposed to be v1.67.0, but there was a mistake during the
release process. This has everything in v1.67.1, *except* for:
- xds: Fix NullPointerException introduced in "Fix load reporting when
pick first is used for locality-routing"
([https://github.com/grpc/grpc-java/pull/11553](https://redirect.github.com/grpc/grpc-java/pull/11553))
---
### Configuration
📅 **Schedule**: Branch creation - At any time (no schedule defined),
Automerge - At any time (no schedule defined).
🚦 **Automerge**: Disabled by config. Please merge this manually once you
are satisfied.
♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the
rebase/retry checkbox.
🔕 **Ignore**: Close this PR and you won't be reminded about these
updates again.
---
- [ ] If you want to rebase/retry this PR, check
this box
---
This PR was generated by [Mend Renovate](https://mend.io/renovate/).
View the [repository job
log](https://developer.mend.io/github/googleapis/sdk-platform-java).
---
.cloudbuild/library_generation/library_generation.Dockerfile | 2 +-
gapic-generator-java-pom-parent/pom.xml | 2 +-
gax-java/dependencies.properties | 2 +-
3 files changed, 3 insertions(+), 3 deletions(-)
diff --git a/.cloudbuild/library_generation/library_generation.Dockerfile b/.cloudbuild/library_generation/library_generation.Dockerfile
index e52b938c19..0c8442cb93 100644
--- a/.cloudbuild/library_generation/library_generation.Dockerfile
+++ b/.cloudbuild/library_generation/library_generation.Dockerfile
@@ -56,7 +56,7 @@ FROM docker.io/library/python@sha256:38e179a0f0436c97ecc76bcd378d7293ab3ee79e4b8
ARG OWLBOT_CLI_COMMITTISH=38fe6f89a2339ee75c77739b31b371f601b01bb3
ARG PROTOC_VERSION=25.5
-ARG GRPC_VERSION=1.67.1
+ARG GRPC_VERSION=1.68.1
ARG JAVA_FORMAT_VERSION=1.7
ENV HOME=/home
ENV OS_ARCHITECTURE="linux-x86_64"
diff --git a/gapic-generator-java-pom-parent/pom.xml b/gapic-generator-java-pom-parent/pom.xml
index 15eb60901d..1991f17d01 100644
--- a/gapic-generator-java-pom-parent/pom.xml
+++ b/gapic-generator-java-pom-parent/pom.xml
@@ -26,7 +26,7 @@
1.3.2
- 1.67.1
+ 1.68.1
1.29.0
1.45.0
2.11.0
diff --git a/gax-java/dependencies.properties b/gax-java/dependencies.properties
index 6c6331f518..689a2f9d36 100644
--- a/gax-java/dependencies.properties
+++ b/gax-java/dependencies.properties
@@ -28,7 +28,7 @@ version.gax_httpjson=2.57.1-SNAPSHOT
version.com_google_protobuf=3.25.5
version.google_java_format=1.15.0
-version.io_grpc=1.67.1
+version.io_grpc=1.68.1
# Maven artifacts.
# Note, the actual name of each property matters (bazel build scripts depend on it).
From 25023afad8f483a3eae8846c4bc1fbe2c7a260c5 Mon Sep 17 00:00:00 2001
From: Joe Wang <106995533+JoeWang1127@users.noreply.github.com>
Date: Tue, 5 Nov 2024 11:44:40 -0500
Subject: [PATCH 08/19] docs: modify hermetic build docs (#3331)
In this PR:
- Add instructions on release note generation.
- Move generating libraries using docker image to `README.md`
- Move generating libraries using python scripts to `DEVELOPMENT.md`.
- Move instructions on building image to `DEVELOPMENT.md`.
---
hermetic_build/DEVELOPMENT.md | 228 ++++++++++++++++++
.../{library_generation => }/README.md | 205 +++++++++++-----
.../library_generation/DEVELOPMENT.md | 207 ----------------
.../cli/generate_release_note.py | 5 +-
4 files changed, 382 insertions(+), 263 deletions(-)
create mode 100644 hermetic_build/DEVELOPMENT.md
rename hermetic_build/{library_generation => }/README.md (67%)
delete mode 100644 hermetic_build/library_generation/DEVELOPMENT.md
diff --git a/hermetic_build/DEVELOPMENT.md b/hermetic_build/DEVELOPMENT.md
new file mode 100644
index 0000000000..b6e78389dc
--- /dev/null
+++ b/hermetic_build/DEVELOPMENT.md
@@ -0,0 +1,228 @@
+> [!IMPORTANT]
+> All examples assume you are inside the repository root folder.
+
+
+# Linting
+
+When contributing, ensure your changes to python code have a valid format.
+
+```
+python -m pip install black
+black {source_file_or_directory}
+```
+
+# Install package dependencies
+
+```shell
+python -m pip install --require-hashes -r hermetic_build/common/requirements.txt
+python -m pip install hermetic_build/common
+python -m pip install --require-hashes -r hermetic_build/library_generation/requirements.txt
+python -m pip install hermetic_build/library_generation
+python -m pip install --require-hashes -r hermetic_build/release_note_generation/requirements.txt
+python -m pip install hermetic_build/release_note_generation
+```
+
+# Run the integration tests
+
+The integration tests build the docker image declared in
+`.cloudbuild/library_generation/library_generation.Dockerfile`, pull GAPIC
+repositories, generate the libraries and compare the results with the source
+code declared in a "golden branch" of the repo.
+
+It requires docker and python (>= 3.12.0) to be installed.
+
+```shell
+python -m unittest hermetic_build/library_generation/tests/integration_tests.py
+```
+
+# Run the unit tests
+
+There is one unit test file per component.
+Every unit test script ends with `unit_tests.py`.
+To avoid specifying them individually, we can use the following command:
+
+```shell
+python -m unittest discover -s hermetic_build -p "*unit_tests.py"
+```
+
+> [!NOTE]
+> The output of this command may look erratic during the first 30 seconds.
+> This is normal. After the tests are done, an "OK" message should be shown.
+
+# Run the library generation scripts in your local environment
+
+Although the scripts are designed to run in a Docker container, you can also
+run them directly.
+This section explains how to run the entrypoint script
+(`hermetic_build/library_generation/cli/entry_point.py`).
+
+## Assumptions made by the scripts
+
+### The Hermetic Build's well-known folder
+Located in `${HOME}/.library_generation`, this folder is assumed by the scripts
+to contain certain tools.
+
+Developers must make sure this folder is properly configured before running the
+scripts locally.
+Note that this relies on the `HOME` environment variable which is always defined
+as per [POSIX env var definition](https://pubs.opengroup.org/onlinepubs/9699919799/basedefs/V1_chap08.html).
+
+#### Put the gapic-generator-java jar in its well-known location
+
+1. Run the following command to install gapic-generator-java.
+
+ ```shell
+ mvn install -B -ntp -DskipTests -Dclirr.skip -Dcheckstyle.skip
+ ```
+ This will generate a jar located in `~/.m2/repository/com/google/api/gapic-generator-java/{version}/gapic-generator-java-{version}.jar`
+
+2. Move the jar into its well-known location.
+
+ ```shell
+ mv /path/to/jar "${HOME}/.library_generation/gapic-generator-java.jar"
+ ```
+
+#### Put the java formatter jar in its well-known location
+
+1. Download google-java-format-{version}-all-deps.jar from [Maven Central](https://central.sonatype.com/artifact/com.google.googlejavaformat/google-java-format)
+or [GitHub releases](https://github.com/google/google-java-format/releases).
+2. Move the jar into its well-known location.
+
+ ```shell
+ mv /path/to/jar "${HOME}/.library_generation/google-java-format.jar"
+ ```
+
+## Installing prerequisites
+
+In order to run the generation scripts directly, there are a few tools we
+need to install beforehand.
+
+### Install the owl-bot CLI
+
+This requires node.js to be installed.
+Check this [installation guide](https://github.com/nvm-sh/nvm?tab=readme-ov-file#install--update-script)
+for NVM, Node.js's version manager.
+
+After you install it, you can install the owl-bot CLI with the following
+commands:
+
+```shell
+git clone https://github.com/googleapis/repo-automation-bots
+cd repo-automation-bots/packages/owl-bot
+npm i && npm run compile && npm link
+owl-bot copy-code --version
+```
+
+The key step is `npm link`, which will make the command available in you current
+shell session.
+
+## Run the script
+The entrypoint script (`hermetic_build/library_generation/cli/entry_point.py`)
+allows you to generate a GAPIC repository with a given api definition (proto,
+service yaml).
+
+### Download the api definition
+For example, from googleapis
+
+```shell
+git clone https://github.com/googleapis/googleapis
+export api_definitions_path="$(pwd)/googleapis"
+```
+
+### Download the repo
+For example, google-cloud-java
+```shell
+git clone https://github.com/googleapis/google-cloud-java
+export path_to_repo="$(pwd)/google-cloud-java"
+```
+
+### Install the scripts
+
+You can skip this step if you've installed the packages in [Install package dependencies](#install-package-dependencies).
+
+```shell
+python -m pip install --require-hashes -r hermetic_build/common/requirements.txt
+python -m pip install hermetic_build/common
+python -m pip install --require-hashes -r hermetic_build/library_generation/requirements.txt
+python -m pip install hermetic_build/library_generation
+```
+
+### Run the script
+
+```shell
+python hermetic_build/library_generation/cli/entry_point.py generate \
+ --repository-path="${path_to_repo}" \
+ --api-definitions-path="${api_definitions_path}"
+```
+
+# Build the image from source
+
+1. Run the following command to build the image from source
+
+ ```shell
+ docker build \
+ -f .cloudbuild/library_generation/library_generation.Dockerfile \
+ -t local:image-tag \
+ .
+ ```
+
+2. Set the version of gapic-generator-java
+
+ ```shell
+ LOCAL_GENERATOR_VERSION=$(mvn \
+ org.apache.maven.plugins:maven-help-plugin:evaluate \
+ -Dexpression=project.version \
+ -pl gapic-generator-java \
+ -DforceStdout \
+ -q)
+ ```
+
+3. Run the image
+
+ ```shell
+ # Assume you want to generate the library in the current working directory
+ # and the generation configuration is in the same directory.
+ docker run \
+ --rm \
+ --quiet \
+ -u "$(id -u):$(id -g)" \
+ -v "$(pwd):/workspace" \
+ -v /path/to/api-definitions:/workspace/apis \
+ -e GENERATOR_VERSION="${LOCAL_GENERATOR_VERSION}" \
+ local:image-tag \
+ --generation-config-path=/workspace/generation_config_file \
+ --library-names=apigee-connect,asset \
+ --repository-path=/workspace \
+ --api-definitions-path=/workspace/apis
+ ```
+ Note that if you specify the generator version using environment variable,
+ `-e GENERATOR_VERSION="${LOCAL_GENERATOR_VERSION}"` in the above example,
+ you should not set `gapic_generator_version` and `protoc_version` in the
+ generation configuration because values in the generation configuration will
+ take precedence.
+
+# Debug the library generation container
+If you are working on changing the way the containers are created, you may want
+to inspect the containers to check the setup.
+It would be convenient in such case to have a text editor/viewer available.
+You can achieve this by modifying the Dockerfile as follows:
+
+```dockerfile
+# install OS tools
+RUN apk update && apk add \
+ unzip curl rsync openjdk11 jq bash nodejs npm git less vim
+```
+
+We add `less` and `vim` as text tools for further inspection.
+
+You can also run a shell in a new container by running:
+
+```shell
+docker run \
+ --rm \
+ -it \
+ -u $(id -u):$(id -g) \
+ -v /path/to/google-cloud-java:/workspace \
+ --entrypoint="bash" \
+ $(cat image-id)
+```
diff --git a/hermetic_build/library_generation/README.md b/hermetic_build/README.md
similarity index 67%
rename from hermetic_build/library_generation/README.md
rename to hermetic_build/README.md
index 0b4208ac3e..583e766bdf 100644
--- a/hermetic_build/library_generation/README.md
+++ b/hermetic_build/README.md
@@ -1,44 +1,43 @@
+> [!IMPORTANT]
+> All scripts/examples assume you are inside the repository root folder.
+
# Generate a repository containing GAPIC Client Libraries
-The script, `entry_point.py`, allows you to generate a repository containing
-GAPIC client libraries (a monorepo, for example, google-cloud-java) from a
-configuration file.
+Running the docker image built from `hermetic_build/library_generation`
+directory, you can generate a repository containing GAPIC client libraries (a
+monorepo, for example, google-cloud-java) from a configuration file.
+
+Instead of running the docker image, if you prefer running the underlying python
+scripts directly, please refer to the [development guide](DEVELOPMENT.md#run-the-script)
+for additional instructions.
## Environment
- OS: Linux
-- Java runtime environment (8 or above)
-- Python (3.12 or above)
-- Docker
-- Git
+- Docker
-## Prerequisite
+## Prerequisites
In order to generate a version for each library, a versions.txt has to exist
-in `repository_path`.
-Please refer to [Repository path](#repository-path--repositorypath---optional) for more information.
-
-## Parameters to generate a repository using `entry_point.py`
-
-### Baseline generation configuration yaml (`baseline_generation_config`)
+in `repository-path`.
+Please refer to [Repository path](#repository-path--repositorypath---optional)
+for more information.
-An absolute or relative path to a generation_config.yaml.
-This config file is used for computing changed libraries, not library
-generation.
+## Parameters to generate a repository using the docker image
-### Current generation configuration yaml (`current_generation_config`)
+### Generation configuration yaml (`generation-config-path`)
An absolute or relative path to a configuration file containing parameters to
generate the repository.
-Please refer [Configuration to generate a repository](#configuration-to-generate-a-repository)
+Please refer to [Configuration to generate a repository](#configuration-to-generate-a-repository)
for more information.
-### Repository path (`repository_path`), optional
+### Repository path (`repository-path`), optional
The path to where the generated repository goes.
The default value is the current working directory when running the script.
-For example, `cd google-cloud-java && python entry_point.py ...` without
+For example, `cd google-cloud-java && python /path/to/entry_point.py ...` without
specifying the `--repository_path` option will modify the `google-cloud-java`
repository the user `cd`'d into.
@@ -47,28 +46,37 @@ right version for each library.
Please refer [here](go/java-client-releasing#versionstxt-manifest) for more info
of versions.txt.
-### Api definitions path (`api_definitions_path`), optional
+### A list of library names (`library-names`), optional
+
+A list of library names that will be generated, separated by comma.
+The library name of a library is the value of `library_name` or `api_shortname`,
+if `library_name` is not specified, in the generation configuration.
+
+If not specified, all libraries in the generation
+configuration will be generated.
+
+### Api definitions path (`api-definitions-path`), optional
The path to where the api definition (proto, service yaml) resides.
The default value is the current working directory when running the script.
-Note that you need not only the protos defined the service, but also the transitive
-dependencies of those protos.
+**Note that you need not only the protos defined the service, but also the
+transitive dependencies of those protos.**
Any missing dependencies will cause `File not found` error.
-For example, if your service is defined in `example_service.proto` and it imports
-`google/api/annotations.proto`, you need the `annotations.proto` resides in a
-folder that has the exact structure of the import statement (`google/api` in this
-case), and set `api_definitions_path` to the path contains the root folder (`google`
-in this case).
+For example, if your service is defined in `example_service.proto` and it
+imports `google/api/annotations.proto`, you need the `annotations.proto` resides
+in a folder that has the exact structure of the import statement (`google/api`
+in this case), and set `api_definitions_path` to the path contains the root
+folder (`google` in this case).
-## Output of `entry_point.py`
+## Output
### GAPIC libraries
-For each module (e.g. `google-cloud-java/java-asset`), the following files/folders
-will be created/modified:
+For each module (e.g. `google-cloud-java/java-asset`), the following
+files/folders will be created/modified:
| Name | Notes |
|:------------------------------------|:-------------------------------------------------------------------------|
@@ -185,32 +193,58 @@ libraries:
- proto_path: google/cloud/asset/v1p7beta1
```
-# Local Environment Setup before running `entry_point.py`
-
-1. Assuming Python 3 is installed, follow official guide from [Python.org](https://packaging.python.org/en/latest/guides/installing-using-pip-and-virtual-environments/#create-and-use-virtual-environments) to create a virtual environment.
-The virtual environment can be installed to any folder, usually it is recommended to be installed under the root folder of the project(`sdk-platform-java` in this case).
-2. Assuming the virtual environment is installed under `sdk-platform-java`.
-Run the following command under the root folder of `sdk-platform-java` to install `library_generation` and its dependencies.
+# Run the library generation image
+
+1. Download the API definitions to a local directory, e.g., from [googleapis](https://github.com/googleapis/googleapis).
+
+2. Run the docker image.
+ ```shell
+ # Assume you want to generate the library in the current working directory
+ # and the generation configuration is in the same directory.
+ docker run \
+ --rm \
+ --quiet \
+ -u "$(id -u):$(id -g)" \
+ -v "$(pwd):/workspace" \
+ -v /path/to/api_definition:/workspace \
+ gcr.io/cloud-devrel-public-resources/java-library-generation:image-tag
+ ```
- ```bash
- python -m pip install --require-hashes -r hermetic_build/common/requirements.txt
- python -m pip install hermetic_build/common
- python -m pip install --require-hashes -r hermetic_build/library_generation/requirements.txt
- python -m pip install hermetic_build/library_generation
+ * `-u "$(id -u)":"$(id -g)"` makes docker run the container impersonating
+ yourself.
+ This avoids folder ownership changes since it runs as root by default.
+ * `-v "$(pwd):/workspace"` maps the host machine's current working directory
+ to the /workspace folder.
+ The image is configured to perform changes in this directory.
+ * `-v /path/to/api_definition:/workspace` maps the host machine's API
+ definitions folder to `/workspace/apis` folder.
+
+3. An advanced example:
+ ```shell
+ docker run \
+ --rm \
+ --quiet \
+ -u "$(id -u):$(id -g)" \
+ -v "$(pwd):/workspace" \
+ -v /path/to/api_definition:/workspace/apis \
+ gcr.io/cloud-devrel-public-resources/java-library-generation:image-tag \
+ --generation-config-path=/workspace/generation_config_file \
+ --library-names=apigee-connect,asset \
+ --repository-path=/workspace \
+ --api-definitions-path=/workspace/apis
```
+
+ * `--generation-config-path=/workspace/generation_config_file` set the
+ generation configuration to `/workspace/generation_config_file`.
+ * `--api-definitions-path=/workspace/apis` set the API definition path to
+ `/workspace/apis`.
-3. Download api definition to a local directory
+To debug the image, please refer to [development guide](DEVELOPMENT.md#debug-the-library-generation-container)
+for more info.
-## An example to generate a repository using `entry_point.py`
+## An example to generate a repository using the docker image
-```bash
-python hermetic_build/library_generation/cli/entry_point.py generate \
- --baseline-generation-config-path=/path/to/baseline_config_file \
- --current-generation-config-path=/path/to/current_config_file \
- --repository-path=path/to/repository \
- --api-definitions-path=path/to/api_definition
-```
-If you run `entry_point.py` with the example [configuration](#an-example-of-generation-configuration)
+If you run the docker image with the example [configuration](#an-example-of-generation-configuration)
shown above, the repository structure is:
```
$repository_path
@@ -284,7 +318,70 @@ $repository_path
|_versions.txt
```
-# Owlbot Java Postprocessor
+# Generate release notes from API definition changes
+
+The script, `hermetic_build/release_note_generation/cli/generate_release_note.py`
+allows you to generate a file containing release notes from API definition
+changes in [googleapis](https://github.com/googleapis/googleapis) GitHub
+repository.
+
+## Environment
+
+- OS: Linux
+- Python (3.12.0 or above)
+
+## Parameters to generate a release note
+
+### Baseline generation configuration path (`baseline-generation-config-path`)
+
+Absolute or relative path to a generation configuration.
+Please refer to [Configuration to generate a repository](#configuration-to-generate-a-repository)
+for more information.
+
+Note that the `googleapis_commitish` in this configuration is used to retrieve
+the first commit, exclusively, to generate the release notes.
+
+### Current generation configuration path (`current-generation-config-path`)
+
+Absolute or relative path to a generation configuration.
+The release notes will be generated from commits that are related to the
+libraries specified in this configuration.
+Please refer to [Configuration to generate a repository](#configuration-to-generate-a-repository)
+for more information.
+
+Note that the `googleapis_commitish` entry in this configuration is used to
+retrieve the last commit, inclusively, to generate the release notes.
+
+### Repository path (`repository-path`), optional
+
+The path to which the file, `pr_description.txt` containing the release notes
+will be sent.
+If not specified, the file will be generated to the current working directory.
+
+## Generate a release notes file in a local environment
+
+1. Install python (>= 3.12.0).
+It is recommended to create a python virtual environment through the
+[official guide](https://packaging.python.org/en/latest/guides/installing-using-pip-and-virtual-environments/#create-and-use-virtual-environments).
+
+2. Run the following commands to install python packages
+ ```shell
+ cd /path/to/sdk-platform-java
+ pip install --require-hashes -r hermetic_build/common/requirements.txt
+ pip install hermetic_build/common
+ pip install --require-hashes -r hermetic_build/release_note_generation/requirements.txt
+ pip install hermetic_build/release_note_generation
+ ```
+3. Run the following commands to generate a release note
+ ```shell
+ cd /path/to/sdk-platform-java
+ python hermetic_build/release_note_generation/cli/generate_release_note.py generate \
+ --baseline-generation-config-path=/path/to/baseline_generation_config \
+ --current-generation-config-path=/path/to/current_generation_config \
+ --repository-path=/path/to/send/release_note
+ ```
+
+# OwlBot Java Postprocessor
We have transferred the
[implementation](https://github.com/googleapis/synthtool/tree/59fe44fde9866a26e7ee4e4450fd79f67f8cf599/docker/owlbot/java)
diff --git a/hermetic_build/library_generation/DEVELOPMENT.md b/hermetic_build/library_generation/DEVELOPMENT.md
deleted file mode 100644
index df7843aeaa..0000000000
--- a/hermetic_build/library_generation/DEVELOPMENT.md
+++ /dev/null
@@ -1,207 +0,0 @@
-> [!IMPORTANT]
-> All examples assume you are inside the `hermetic_build` folder.
-
-
-# Linting
-
-When contributing, ensure your changes to python code have a valid format.
-
-```
-python -m pip install black
-black .
-```
-
-# Running the integration tests
-
-The integration tests build the docker image declared in
-`.cloudbuild/library_generation/library_generation.Dockerfile`, pull GAPIC
-repositories, generate the libraries and compares the results with the source
-code declared in a "golden branch" of the repo.
-
-It requires docker and python 3.x to be installed.
-
-```
-python -m pip install --require-hashes -r library_generation/requirements.txt
-python -m pip install library_generation
-python -m unittest library_generation/tests/integration_tests.py
-```
-
-# Running the unit tests
-
-The unit tests of the hermetic build scripts are contained in several scripts,
-corresponding to a specific component.
-Every unit test script ends with `unit_tests.py`.
-To avoid them specifying them individually, we can use the following command:
-
-```bash
-python -m unittest discover -s library_generation/tests/ -p "*unit_tests.py"
-```
-
-> [!NOTE]
-> The output of this command may look erratic during the first 30 seconds.
-> This is normal. After the tests are done, an "OK" message should be shown.
-
-# Running the scripts in your local environment
-
-Although the scripts are designed to be run in a Docker container, you can also
-run them directly.
-This section explains how to run the entrypoint script
-(`library_generation/cli/entry_point.py`).
-
-## Assumptions made by the scripts
-### The Hermetic Build's well-known folder
-Located in `${HOME}/.library_generation`, this folder is assumed by the scripts
-to contain certain tools.
-
-Developers must make sure this folder is properly configured before running the
-scripts locally.
-Note that this relies on the `HOME` en var which is always defined as per
-[POSIX env var definition](https://pubs.opengroup.org/onlinepubs/9699919799/basedefs/V1_chap08.html).
-
-#### Put the gapic-generator-java jar in its well-known location
-
-Run `cd sdk-platform-java && mvn install -DskipTests -Dclirr.skip
--Dcheckstyle.skip`.
-This will generate a jar located in
-`~/.m2/repository/com/google/api/gapic-generator-java/{version}/gapic-generator-java-{version}.jar`
-
-Then `mv` the jar into the well-known location of the jar.
-The generation scripts will assume the jar is there.
-
-```shell
-mv /path/to/jar "${HOME}/.library_generation/gapic-generator-java.jar"
-```
-
-#### Put the java formatter jar in its well-known location
-
-Download google-java-format-{version}-all-deps.jar from [Maven Central](https://central.sonatype.com/artifact/com.google.googlejavaformat/google-java-format)
-or [GitHub releases](https://github.com/google/google-java-format/releases).
-Then `mv` the jar into the well-known location of the jar.
-The generation scripts will assume the jar is there.
-
-```shell
-mv /path/to/jar "${HOME}/.library_generation/google-java-format.jar"
-```
-
-## Installing prerequisites
-
-In order to run the generation scripts directly, there are a few tools we
-need to install beforehand.
-
-### Install the owl-bot CLI
-
-Requires node.js to be installed.
-Check this [installation guide](https://github.com/nvm-sh/nvm?tab=readme-ov-file#install--update-script)
-for NVM, Node.js's version manager.
-
-After you install it, you can install the owl-bot CLI with the following
-commands:
-```bash
-git clone https://github.com/googleapis/repo-automation-bots
-cd repo-automation-bots/packages/owl-bot
-npm i && npm run compile && npm link
-owl-bot copy-code --version
-```
-
-The key step is `npm link`, which will make the command available in you current
-shell session.
-
-
-## Running the script
-The entrypoint script (`library_generation/cli/entry_point.py`) allows you to
-generate a GAPIC repository with a given api definition (proto, service yaml).
-
-### Download the api definition
-For example, googleapis
-```
-git clone https://github.com/googleapis/googleapis
-export api_definitions_path="$(pwd)/googleapis"
-```
-
-### Download the repo
-For example, google-cloud-java
-```
-git clone https://github.com/googleapis/google-cloud-java
-export path_to_repo="$(pwd)/google-cloud-java"
-```
-
-### Install the scripts
-```
-python -m pip install .
-```
-
-### Run the script
-```
-python library_generation/cli/entry_point.py generate \
- --repository-path="${path_to_repo}" \
- --api-definitions-path="${api_definitions_path}"
-```
-
-
-# Running the scripts using the docker container image
-This is convenient in order to avoid installing the dependencies manually.
-
-> [!IMPORTANT]
-> From now, the examples assume you are in the root of your sdk-platform-java
-> folder.
-
-## Build the docker image
-```bash
-docker build --file .cloudbuild/library_generation/library_generation.Dockerfile --iidfile image-id .
-```
-
-This will create an `image-id` file at the root of the repo with the hash ID of
-the image.
-
-## Run the docker image
-The docker image will perform changes on its internal `/workspace` folder,
-to which you need to map a folder on your host machine (i.e. map your downloaded
-repo to this folder).
-
-To run the docker container on the google-cloud-java repo, you must run:
-```bash
-docker run \
- -u "$(id -u)":"$(id -g)" \
- -v /path/to/google-cloud-java:/workspace \
- -v /path/to/api-definition:/workspace/apis \
- $(cat image-id) \
- --api-definitions-path=/workspace/apis
-```
-
- * `-u "$(id -u)":"$(id -g)"` makes docker run the container impersonating
- yourself. This avoids folder ownership changes since it runs as root by
- default.
- * `-v /path/to/google-cloud-java:/workspace` maps the host machine's
- google-cloud-java folder to the /workspace folder.
- The image is configured to perform changes in this directory.
- * `-v /path/to/api-definition:/workspace/apis` maps the host machine's
- api-definition folder to /workspace/apis folder.
- * `$(cat image-id)` obtains the image ID created in the build step.
- * `--api-definitions-path=/workspace/apis` set the API definition path to
- `/workspace/apis`.
-
-## Debug the created containers
-If you are working on changing the way the containers are created, you may want
-to inspect the containers to check the setup.
-It would be convenient in such case to have a text editor/viewer available.
-You can achieve this by modifying the Dockerfile as follows:
-
-```docker
-# install OS tools
-RUN apt-get update && apt-get install -y \
- unzip openjdk-17-jdk rsync maven jq less vim \
- && apt-get clean
-```
-
-We add `less` and `vim` as text tools for further inspection.
-
-You can also run a shell in a new container by running:
-
-```bash
-docker run \
- --rm -it \
- -u $(id -u):$(id -g) \
- -v /path/to/google-cloud-java:/workspace \
- --entrypoint="bash" \
- $(cat image-id)
-```
diff --git a/hermetic_build/release_note_generation/cli/generate_release_note.py b/hermetic_build/release_note_generation/cli/generate_release_note.py
index 294f4baf50..a54e228f62 100644
--- a/hermetic_build/release_note_generation/cli/generate_release_note.py
+++ b/hermetic_build/release_note_generation/cli/generate_release_note.py
@@ -54,8 +54,9 @@ def main(ctx):
default=".",
show_default=True,
help="""
- The repository path to which the generated files will be sent.
- If not specified, the repository will be generated to the current working
+ The path where the file `pr_description.txt`, which contains the release
+ notes, will be sent.
+ If not specified, the file will be generated to the current working
directory.
""",
)
From 0cddadb8ad3eddfffa356a479964d8a720937503 Mon Sep 17 00:00:00 2001
From: Min Zhu
Date: Tue, 5 Nov 2024 16:59:03 -0500
Subject: [PATCH 09/19] feat: enable selective generation based on service
config include list (#3323)
following changes in client.proto that propagated to `ClientProto.java`
([pr](https://github.com/googleapis/sdk-platform-java/pull/3309/files#diff-44c330ef5cfa380744be6c58a14aa543edceb6043d5b17da7b32bb728ef5d85f)),
apply changes from poc pr
(https://github.com/googleapis/sdk-platform-java/pull/3129).
For context:
[go/selective-api-gen-java-one-pager](http://goto.google.com/selective-api-gen-java-one-pager),
b/356380016
---
.../generator/gapic/protoparser/Parser.java | 99 +++++++++--
.../gapic/protoparser/ParserTest.java | 129 ++++++++++++++
.../test/proto/selective_api_generation.proto | 163 ++++++++++++++++++
.../selective_api_generation_v1beta1.yaml | 23 +++
4 files changed, 397 insertions(+), 17 deletions(-)
create mode 100644 gapic-generator-java/src/test/proto/selective_api_generation.proto
create mode 100644 gapic-generator-java/src/test/resources/selective_api_generation_v1beta1.yaml
diff --git a/gapic-generator-java/src/main/java/com/google/api/generator/gapic/protoparser/Parser.java b/gapic-generator-java/src/main/java/com/google/api/generator/gapic/protoparser/Parser.java
index 9a2da74747..2e17b9026b 100644
--- a/gapic-generator-java/src/main/java/com/google/api/generator/gapic/protoparser/Parser.java
+++ b/gapic-generator-java/src/main/java/com/google/api/generator/gapic/protoparser/Parser.java
@@ -14,6 +14,7 @@
package com.google.api.generator.gapic.protoparser;
+import com.google.api.ClientLibrarySettings;
import com.google.api.ClientProto;
import com.google.api.DocumentationRule;
import com.google.api.FieldBehavior;
@@ -84,6 +85,7 @@
import java.util.Optional;
import java.util.Set;
import java.util.function.Function;
+import java.util.logging.Level;
import java.util.logging.Logger;
import java.util.stream.Collectors;
import java.util.stream.IntStream;
@@ -160,11 +162,11 @@ public static GapicContext parse(CodeGeneratorRequest request) {
messages = updateResourceNamesInMessages(messages, resourceNames.values());
// Contains only resource names that are actually used. Usage refers to the presence of a
- // request message's field in an RPC's method_signature annotation. That is, resource name
- // definitions
- // or references that are simply defined, but not used in such a manner, will not have
- // corresponding Java helper
- // classes generated.
+ // request message's field in an RPC's method_signature annotation. That is, resource name
+ // definitions or references that are simply defined, but not used in such a manner,
+ // will not have corresponding Java helper classes generated.
+ // If selective api generation is configured via service yaml, Java helper classes are only
+ // generated if resource names are actually used by methods selected to generate.
Set outputArgResourceNames = new HashSet<>();
List mixinServices = new ArrayList<>();
Transport transport = Transport.parse(transportOpt.orElse(Transport.GRPC.toString()));
@@ -425,6 +427,71 @@ public static List parseService(
Transport.GRPC);
}
+ static boolean shouldIncludeMethodInGeneration(
+ MethodDescriptor method,
+ Optional serviceYamlProtoOpt,
+ String protoPackage) {
+ // default to include all when no service yaml or no library setting section.
+ if (!serviceYamlProtoOpt.isPresent()
+ || serviceYamlProtoOpt.get().getPublishing().getLibrarySettingsCount() == 0) {
+ return true;
+ }
+ List librarySettingsList =
+ serviceYamlProtoOpt.get().getPublishing().getLibrarySettingsList();
+ // Validate for logging purpose, this should be validated upstream.
+ // If library_settings.version does not match with proto package name
+ // Give warnings and disregard this config. default to include all.
+ if (!librarySettingsList.get(0).getVersion().isEmpty()
+ && !protoPackage.equals(librarySettingsList.get(0).getVersion())) {
+ if (LOGGER.isLoggable(Level.WARNING)) {
+ LOGGER.warning(
+ String.format(
+ "Service yaml config is misconfigured. Version in "
+ + "publishing.library_settings (%s) does not match proto package (%s)."
+ + "Disregarding selective generation settings.",
+ librarySettingsList.get(0).getVersion(), protoPackage));
+ }
+ return true;
+ }
+ // librarySettingsList is technically a list, but is processed upstream and
+ // only leave with 1 element. Otherwise, it is a misconfiguration and
+ // should be caught upstream.
+ List includeMethodsList =
+ librarySettingsList
+ .get(0)
+ .getJavaSettings()
+ .getCommon()
+ .getSelectiveGapicGeneration()
+ .getMethodsList();
+ // default to include all when nothing specified, this could be no java section
+ // specified in library setting, or the method list is empty
+ if (includeMethodsList.isEmpty()) {
+ return true;
+ }
+
+ return includeMethodsList.contains(method.getFullName());
+ }
+
+ private static boolean isEmptyService(
+ ServiceDescriptor serviceDescriptor,
+ Optional serviceYamlProtoOpt,
+ String protoPackage) {
+ List methodsList = serviceDescriptor.getMethods();
+ List methodListSelected =
+ methodsList.stream()
+ .filter(
+ method ->
+ shouldIncludeMethodInGeneration(method, serviceYamlProtoOpt, protoPackage))
+ .collect(Collectors.toList());
+ if (methodListSelected.isEmpty()) {
+ LOGGER.log(
+ Level.WARNING,
+ "Service {0} has no RPC methods and will not be generated",
+ serviceDescriptor.getName());
+ }
+ return methodListSelected.isEmpty();
+ }
+
public static List parseService(
FileDescriptor fileDescriptor,
Map messageTypes,
@@ -433,19 +500,11 @@ public static List parseService(
Optional serviceConfigOpt,
Set outputArgResourceNames,
Transport transport) {
-
+ String protoPackage = fileDescriptor.getPackage();
return fileDescriptor.getServices().stream()
.filter(
- serviceDescriptor -> {
- List methodsList = serviceDescriptor.getMethods();
- if (methodsList.isEmpty()) {
- LOGGER.warning(
- String.format(
- "Service %s has no RPC methods and will not be generated",
- serviceDescriptor.getName()));
- }
- return !methodsList.isEmpty();
- })
+ serviceDescriptor ->
+ !isEmptyService(serviceDescriptor, serviceYamlProtoOpt, protoPackage))
.map(
s -> {
// Workaround for a missing default_host and oauth_scopes annotation from a service
@@ -498,6 +557,8 @@ public static List parseService(
String pakkage = TypeParser.getPackage(fileDescriptor);
String originalJavaPackage = pakkage;
// Override Java package with that specified in gapic.yaml.
+ // this override is deprecated and legacy support only
+ // see go/client-user-guide#configure-long-running-operation-polling-timeouts-optional
if (serviceConfigOpt.isPresent()
&& serviceConfigOpt.get().getLanguageSettingsOpt().isPresent()) {
GapicLanguageSettings languageSettings =
@@ -518,6 +579,7 @@ public static List parseService(
.setMethods(
parseMethods(
s,
+ protoPackage,
pakkage,
messageTypes,
resourceNames,
@@ -709,6 +771,7 @@ public static Map parseResourceNames(
@VisibleForTesting
static List parseMethods(
ServiceDescriptor serviceDescriptor,
+ String protoPackage,
String servicePackage,
Map messageTypes,
Map resourceNames,
@@ -721,8 +784,10 @@ static List parseMethods(
// Parse the serviceYaml for autopopulated methods and fields once and put into a map
Map> autoPopulatedMethodsWithFields =
parseAutoPopulatedMethodsAndFields(serviceYamlProtoOpt);
-
for (MethodDescriptor protoMethod : serviceDescriptor.getMethods()) {
+ if (!shouldIncludeMethodInGeneration(protoMethod, serviceYamlProtoOpt, protoPackage)) {
+ continue;
+ }
// Parse the method.
TypeNode inputType = TypeParser.parseType(protoMethod.getInputType());
Method.Builder methodBuilder = Method.builder();
diff --git a/gapic-generator-java/src/test/java/com/google/api/generator/gapic/protoparser/ParserTest.java b/gapic-generator-java/src/test/java/com/google/api/generator/gapic/protoparser/ParserTest.java
index 2776fec687..6e8ffa7232 100644
--- a/gapic-generator-java/src/test/java/com/google/api/generator/gapic/protoparser/ParserTest.java
+++ b/gapic-generator-java/src/test/java/com/google/api/generator/gapic/protoparser/ParserTest.java
@@ -21,9 +21,11 @@
import static org.junit.jupiter.api.Assertions.assertThrows;
import static org.junit.jupiter.api.Assertions.assertTrue;
+import com.google.api.ClientLibrarySettings;
import com.google.api.FieldInfo.Format;
import com.google.api.MethodSettings;
import com.google.api.Publishing;
+import com.google.api.PythonSettings;
import com.google.api.Service;
import com.google.api.generator.engine.ast.ConcreteReference;
import com.google.api.generator.engine.ast.Reference;
@@ -46,6 +48,7 @@
import com.google.protobuf.Descriptors.MethodDescriptor;
import com.google.protobuf.Descriptors.ServiceDescriptor;
import com.google.protobuf.compiler.PluginProtos.CodeGeneratorRequest;
+import com.google.selective.generate.v1beta1.SelectiveApiGenerationOuterClass;
import com.google.showcase.v1beta1.EchoOuterClass;
import com.google.showcase.v1beta1.TestingOuterClass;
import com.google.testgapic.v1beta1.LockerProto;
@@ -58,6 +61,8 @@
import java.util.Map;
import java.util.Optional;
import java.util.Set;
+import java.util.stream.Collectors;
+import org.junit.Assert;
import org.junit.jupiter.api.BeforeEach;
import org.junit.jupiter.api.Test;
@@ -137,6 +142,7 @@ void parseMethods_basic() {
Parser.parseMethods(
echoService,
ECHO_PACKAGE,
+ ECHO_PACKAGE,
messageTypes,
resourceNames,
Optional.empty(),
@@ -200,6 +206,7 @@ void parseMethods_basicLro() {
Parser.parseMethods(
echoService,
ECHO_PACKAGE,
+ ECHO_PACKAGE,
messageTypes,
resourceNames,
Optional.empty(),
@@ -705,6 +712,128 @@ void parseServiceWithNoMethodsTest() {
assertEquals("EchoWithMethods", services.get(0).overriddenName());
}
+ @Test
+ void selectiveGenerationTest_shouldExcludeUnusedResourceNames() {
+ FileDescriptor fileDescriptor = SelectiveApiGenerationOuterClass.getDescriptor();
+ Map messageTypes = Parser.parseMessages(fileDescriptor);
+ Map resourceNames = Parser.parseResourceNames(fileDescriptor);
+
+ String serviceYamlFilename = "selective_api_generation_v1beta1.yaml";
+ String testFilesDirectory = "src/test/resources/";
+ Path serviceYamlPath = Paths.get(testFilesDirectory, serviceYamlFilename);
+ Optional serviceYamlOpt =
+ ServiceYamlParser.parse(serviceYamlPath.toString());
+ Assert.assertTrue(serviceYamlOpt.isPresent());
+
+ Set helperResourceNames = new HashSet<>();
+ Parser.parseService(
+ fileDescriptor, messageTypes, resourceNames, serviceYamlOpt, helperResourceNames);
+ // resource Name Foobarbaz is not present
+ assertEquals(2, helperResourceNames.size());
+ assertTrue(
+ helperResourceNames.stream()
+ .map(ResourceName::variableName)
+ .collect(Collectors.toSet())
+ .containsAll(ImmutableList.of("foobar", "anythingGoes")));
+ }
+
+ @Test
+ void selectiveGenerationTest_shouldGenerateOnlySelectiveMethods() {
+ FileDescriptor fileDescriptor = SelectiveApiGenerationOuterClass.getDescriptor();
+ Map messageTypes = Parser.parseMessages(fileDescriptor);
+ Map resourceNames = Parser.parseResourceNames(fileDescriptor);
+
+ // test with service yaml file to show usage of this feature, test itself
+ // can be done without this file and build a Service object from code.
+ String serviceYamlFilename = "selective_api_generation_v1beta1.yaml";
+ String testFilesDirectory = "src/test/resources/";
+ Path serviceYamlPath = Paths.get(testFilesDirectory, serviceYamlFilename);
+ Optional serviceYamlOpt =
+ ServiceYamlParser.parse(serviceYamlPath.toString());
+ Assert.assertTrue(serviceYamlOpt.isPresent());
+
+ List services =
+ Parser.parseService(
+ fileDescriptor, messageTypes, resourceNames, serviceYamlOpt, new HashSet<>());
+ assertEquals(1, services.size());
+ assertEquals("EchoServiceShouldGeneratePartial", services.get(0).overriddenName());
+ assertEquals(3, services.get(0).methods().size());
+ for (Method method : services.get(0).methods()) {
+ assertTrue(method.name().contains("ShouldInclude"));
+ }
+ }
+
+ @Test
+ void selectiveGenerationTest_shouldGenerateAllIfNoPublishingSectionInServiceYaml() {
+ Service service =
+ Service.newBuilder()
+ .setTitle("Selective generation testing with no publishing section")
+ .build();
+ Publishing publishing = service.getPublishing();
+ Assert.assertEquals(0, publishing.getLibrarySettingsCount());
+
+ FileDescriptor fileDescriptor = SelectiveApiGenerationOuterClass.getDescriptor();
+ List methods = fileDescriptor.getServices().get(0).getMethods();
+ String protoPackage = "google.selective.generate.v1beta1";
+
+ assertTrue(
+ Parser.shouldIncludeMethodInGeneration(methods.get(0), Optional.of(service), protoPackage));
+ }
+
+ @Test
+ void selectiveGenerationTest_shouldIncludeMethodInGenerationWhenProtoPackageMismatch() {
+ String protoPackage = "google.selective.generate.v1beta1";
+
+ // situation where service yaml has different version stated
+ ClientLibrarySettings clientLibrarySettings =
+ ClientLibrarySettings.newBuilder().setVersion("google.selective.generate.v1").build();
+ Publishing publishing =
+ Publishing.newBuilder().addLibrarySettings(clientLibrarySettings).build();
+ Service service =
+ Service.newBuilder()
+ .setTitle(
+ "Selective generation test when proto package "
+ + "does not match library_settings version from service yaml")
+ .setPublishing(publishing)
+ .build();
+
+ FileDescriptor fileDescriptor = SelectiveApiGenerationOuterClass.getDescriptor();
+ List methods = fileDescriptor.getServices().get(0).getMethods();
+
+ assertTrue(
+ Parser.shouldIncludeMethodInGeneration(methods.get(0), Optional.of(service), protoPackage));
+ }
+
+ @Test
+ void selectiveGenerationTest_shouldGenerateAllIfNoJavaSectionInServiceYaml() {
+ String protoPackage = "google.selective.generate.v1beta1";
+
+ // situation where service yaml has other language settings but no
+ // java settings in library_settings.
+ ClientLibrarySettings clientLibrarySettings =
+ ClientLibrarySettings.newBuilder()
+ .setVersion(protoPackage)
+ .setPythonSettings(PythonSettings.newBuilder().build())
+ .build();
+ Publishing publishing =
+ Publishing.newBuilder().addLibrarySettings(clientLibrarySettings).build();
+ Service service =
+ Service.newBuilder()
+ .setTitle(
+ "Selective generation test when no java section in "
+ + "library_settings from service yaml")
+ .setPublishing(publishing)
+ .build();
+
+ Assert.assertEquals(1, publishing.getLibrarySettingsCount());
+
+ FileDescriptor fileDescriptor = SelectiveApiGenerationOuterClass.getDescriptor();
+ List methods = fileDescriptor.getServices().get(0).getMethods();
+
+ assertTrue(
+ Parser.shouldIncludeMethodInGeneration(methods.get(0), Optional.of(service), protoPackage));
+ }
+
private void assertMethodArgumentEquals(
String name, TypeNode type, List nestedFields, MethodArgument argument) {
assertEquals(name, argument.name());
diff --git a/gapic-generator-java/src/test/proto/selective_api_generation.proto b/gapic-generator-java/src/test/proto/selective_api_generation.proto
new file mode 100644
index 0000000000..06da2c2e41
--- /dev/null
+++ b/gapic-generator-java/src/test/proto/selective_api_generation.proto
@@ -0,0 +1,163 @@
+// Copyright 2024 Google LLC
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// https://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+syntax = "proto3";
+
+import "google/api/annotations.proto";
+import "google/api/client.proto";
+import "google/api/field_behavior.proto";
+import "google/api/field_info.proto";
+import "google/api/resource.proto";
+import "google/longrunning/operations.proto";
+import "google/protobuf/duration.proto";
+import "google/protobuf/timestamp.proto";
+import "google/rpc/status.proto";
+
+package google.selective.generate.v1beta1;
+
+option java_package = "com.google.selective.generate.v1beta1";
+option java_multiple_files = true;
+option java_outer_classname = "SelectiveApiGenerationOuterClass";
+
+// resource not tied to message
+option (google.api.resource_definition) = {
+ type: "showcase.googleapis.com/AnythingGoes"
+ pattern: "*"
+};
+
+// This proto is used to test selective api generation
+// covered scenarios:
+// - A service with several rpcs, part of them should be generated
+// - A service with several rpcs, none of them should be generated
+// This proto should be tested side-by-side with yaml file:
+// - selective_api_generation_v1beta1.yaml
+
+service EchoServiceShouldGeneratePartial {
+ option (google.api.default_host) = "localhost:7469";
+
+ rpc EchoShouldInclude(EchoRequest) returns (EchoResponse) {
+ option (google.api.http) = {
+ post: "/v1beta1/echo:echo"
+ body: "*"
+ };
+ option (google.api.method_signature) = "name";
+ option (google.api.method_signature) = "";
+ }
+
+ rpc ChatShouldInclude(stream EchoRequest) returns (stream EchoResponse);
+
+ rpc ChatAgainShouldInclude(stream EchoRequest) returns (stream EchoResponse) {
+ option (google.api.method_signature) = "content";
+ }
+
+ rpc AnExcludedMethod(stream EchoRequestWithFoobarbaz) returns (stream EchoResponse);
+
+ rpc AnotherExcludedMethod(stream EchoRequest) returns (stream EchoResponse);
+
+}
+
+service EchoServiceShouldGenerateNone {
+ option (google.api.default_host) = "localhost:7469";
+ option (google.api.oauth_scopes) =
+ "https://www.googleapis.com/auth/cloud-platform";
+
+ rpc Echo(EchoRequest) returns (EchoResponse) {
+ option (google.api.method_signature) = "content";
+ }
+
+ rpc ChatAgain(stream EchoRequest) returns (stream EchoResponse) {
+ option (google.api.method_signature) = "content";
+ }
+}
+
+// resource name used for message EchoRequest.
+message Foobar {
+ option (google.api.resource) = {
+ type: "showcase.googleapis.com/Foobar"
+ pattern: "projects/{project}/foobars/{foobar}"
+ };
+
+ string name = 1;
+ string info = 2;
+}
+
+// resource name used only for message EchoRequestWithFoobarbaz.
+// should not be generated with selective generation when
+// AnExcludedMethod is not config as included.
+message Foobarbaz {
+ option (google.api.resource) = {
+ type: "showcase.googleapis.com/Foobarbaz"
+ pattern: "projects/{project}/foobarsbaz/{foobarbaz}"
+ pattern: "projects/{project}/chocolate/variants/{variant}/foobars/{foobar}"
+ };
+
+ string name = 1;
+ string info = 2;
+}
+
+// RPCs in inclusion list and not in the list both relies on this request message.
+message EchoRequest {
+ string name = 5 [
+ (google.api.resource_reference).type = "showcase.googleapis.com/Foobar",
+ (google.api.field_behavior) = REQUIRED
+ ];
+
+ string parent = 6 [
+ (google.api.resource_reference).child_type =
+ "showcase.googleapis.com/AnythingGoes",
+ (google.api.field_behavior) = REQUIRED
+ ];
+
+ oneof response {
+ // The content to be echoed by the server.
+ string content = 1;
+
+ // The error to be thrown by the server.
+ google.rpc.Status error = 2;
+ }
+
+ Foobar foobar = 4;
+}
+
+// This request message is used by AnExcludedMethod rpc.
+// To demonstrate that if AnExcludedMethod is not included in generation,
+// then the resource name Foobarbaz, which is only used by this method,
+// should not be generated.
+message EchoRequestWithFoobarbaz {
+ string name = 5 [
+ (google.api.resource_reference).type = "showcase.googleapis.com/Foobarbaz",
+ (google.api.field_behavior) = REQUIRED
+ ];
+
+ string parent = 6 [
+ (google.api.resource_reference).child_type =
+ "showcase.googleapis.com/AnythingGoes",
+ (google.api.field_behavior) = REQUIRED
+ ];
+
+ oneof response {
+ // The content to be echoed by the server.
+ string content = 1;
+
+ // The error to be thrown by the server.
+ google.rpc.Status error = 2;
+ }
+
+ Foobarbaz foobar = 4;
+}
+
+message EchoResponse {
+ // The content specified in the request.
+ string content = 1;
+}
diff --git a/gapic-generator-java/src/test/resources/selective_api_generation_v1beta1.yaml b/gapic-generator-java/src/test/resources/selective_api_generation_v1beta1.yaml
new file mode 100644
index 0000000000..021e257c50
--- /dev/null
+++ b/gapic-generator-java/src/test/resources/selective_api_generation_v1beta1.yaml
@@ -0,0 +1,23 @@
+type: google.api.Service
+config_version: 3
+name: selective_api_generation_testing.googleapis.com
+title: Selective Generation Testing API
+
+publishing:
+ # ...
+ library_settings:
+ - version: google.selective.generate.v1beta1
+ java_settings:
+ common:
+ selective_gapic_generation:
+ methods:
+ - google.selective.generate.v1beta1.EchoServiceShouldGeneratePartial.EchoShouldInclude
+ - google.selective.generate.v1beta1.EchoServiceShouldGeneratePartial.ChatShouldInclude
+ - google.selective.generate.v1beta1.EchoServiceShouldGeneratePartial.ChatAgainShouldInclude
+ reference_docs_uri: www.abc.net
+ destinations:
+ - PACKAGE_MANAGER
+ python_settings:
+ common:
+ destinations:
+ - PACKAGE_MANAGER
From e73740dbdb21d7c28908554fe3725504dc8ce84b Mon Sep 17 00:00:00 2001
From: Blake Li
Date: Wed, 6 Nov 2024 11:53:23 -0500
Subject: [PATCH 10/19] fix: Fix flaky test
ScheduledRetryingExecutorTest.testCancelOuterFutureAfterStart (#3335)
The test was flaky because the
[Thread.sleep(150)](https://github.com/googleapis/sdk-platform-java/blob/b031b18a9bb35b77ca21d3665217aa4c219ced57/gax-java/gax/src/test/java/com/google/api/gax/retrying/ScheduledRetryingExecutorTest.java#L288)
does not guarantee the main thread would wake up after exactly 150ms, it
might take much longer if the resource is limited on the machine or we
are running tests in parallel.
Hence if the thread sleeps more than
[1525ms](https://github.com/googleapis/sdk-platform-java/blob/0cddadb8ad3eddfffa356a479964d8a720937503/gax-java/gax/src/test/java/com/google/api/gax/retrying/ScheduledRetryingExecutorTest.java#L262),
the future would be completed already, and the cancellation of the
future would result in false.
Update the total request duration to 32500ms now, it is still not bullet
proof but we would have much less chance to run into flaky tests now.
fixes: https://github.com/googleapis/sdk-platform-java/issues/2669
---
.../gax/retrying/ScheduledRetryingExecutorTest.java | 11 ++++++-----
1 file changed, 6 insertions(+), 5 deletions(-)
diff --git a/gax-java/gax/src/test/java/com/google/api/gax/retrying/ScheduledRetryingExecutorTest.java b/gax-java/gax/src/test/java/com/google/api/gax/retrying/ScheduledRetryingExecutorTest.java
index 53c1707290..5a65895ef0 100644
--- a/gax-java/gax/src/test/java/com/google/api/gax/retrying/ScheduledRetryingExecutorTest.java
+++ b/gax-java/gax/src/test/java/com/google/api/gax/retrying/ScheduledRetryingExecutorTest.java
@@ -195,13 +195,13 @@ void testCancelGetAttempt(boolean withCustomRetrySettings) throws Exception {
setUp(withCustomRetrySettings);
for (int executionsCount = 0; executionsCount < EXECUTIONS_COUNT; executionsCount++) {
ScheduledExecutorService localExecutor = Executors.newSingleThreadScheduledExecutor();
- final int maxRetries = 100;
+ final int maxRetries = 20;
FailingCallable callable = new FailingCallable(maxRetries - 1, "request", "SUCCESS", tracer);
RetrySettings retrySettings =
FAST_RETRY_SETTINGS
.toBuilder()
- .setTotalTimeoutDuration(java.time.Duration.ofMillis(1000L))
+ .setTotalTimeoutDuration(java.time.Duration.ofMillis(5000L))
.setMaxAttempts(maxRetries)
.build();
@@ -259,10 +259,11 @@ void testCancelOuterFutureAfterStart() throws Exception {
.toBuilder()
// These params were selected to ensure that future tries to run and fail (at least
// once) but does not complete before it is cancelled. Assuming no computation time,
- // it would take 25 + 100 + 400 + 1000 = 1525ms for the future to complete, which should
+ // it would take 2500 + 10000 + 10000 + 10000 = 32500ms for the future to complete,
+ // which should
// be more than enough time to cancel the future.
- .setInitialRetryDelayDuration(java.time.Duration.ofMillis(25L))
- .setMaxRetryDelayDuration(java.time.Duration.ofMillis(1000L))
+ .setInitialRetryDelayDuration(java.time.Duration.ofMillis(2500L))
+ .setMaxRetryDelayDuration(java.time.Duration.ofMillis(10000L))
.setRetryDelayMultiplier(4.0)
.setTotalTimeoutDuration(java.time.Duration.ofMillis(60000L))
// Set this test to not use jitter as the randomized retry delay (RRD) may introduce
From 6372eee2a73dcddda6b52045741ca8329f2ab435 Mon Sep 17 00:00:00 2001
From: Joe Wang <106995533+JoeWang1127@users.noreply.github.com>
Date: Wed, 6 Nov 2024 16:36:19 -0500
Subject: [PATCH 11/19] chore: update current version in trigger (#3336)
In this PR:
- Setup release please to update version in
`.cloudbuild/library_generation/cloudbuild-library-generation-push-prod.yaml`
---
.../cloudbuild-library-generation-push-prod.yaml | 2 +-
release-please-config.json | 1 +
2 files changed, 2 insertions(+), 1 deletion(-)
diff --git a/.cloudbuild/library_generation/cloudbuild-library-generation-push-prod.yaml b/.cloudbuild/library_generation/cloudbuild-library-generation-push-prod.yaml
index 81691fa572..b3d6783087 100644
--- a/.cloudbuild/library_generation/cloudbuild-library-generation-push-prod.yaml
+++ b/.cloudbuild/library_generation/cloudbuild-library-generation-push-prod.yaml
@@ -15,7 +15,7 @@
timeout: 7200s # 2 hours
substitutions:
_IMAGE_NAME: "us-docker.pkg.dev/java-hermetic-build-prod/private-resources/java-library-generation"
- _GAPIC_GENERATOR_JAVA_VERSION: '2.46.2-SNAPSHOT' # {x-version-update:gapic-generator-java:current}
+ _GAPIC_GENERATOR_JAVA_VERSION: '2.49.1-SNAPSHOT' # {x-version-update:gapic-generator-java:current}
_SHA_IMAGE_ID: "${_IMAGE_NAME}:${COMMIT_SHA}"
_LATEST_IMAGE_ID: "${_IMAGE_NAME}:latest"
_VERSIONED_IMAGE_ID: "${_IMAGE_NAME}:${_GAPIC_GENERATOR_JAVA_VERSION}"
diff --git a/release-please-config.json b/release-please-config.json
index df5cc02468..c264303ad1 100644
--- a/release-please-config.json
+++ b/release-please-config.json
@@ -11,6 +11,7 @@
".cloudbuild/graalvm/cloudbuild-test-a.yaml",
".cloudbuild/graalvm/cloudbuild-test-b.yaml",
".cloudbuild/library_generation/cloudbuild-library-generation-push.yaml",
+ ".cloudbuild/library_generation/cloudbuild-library-generation-push-prod.yaml",
".cloudbuild/library_generation/library_generation.Dockerfile"
]
}
From 01e742de49e151efbd903808a859f595f99bc8de Mon Sep 17 00:00:00 2001
From: Diego Marquez
Date: Wed, 6 Nov 2024 17:06:07 -0500
Subject: [PATCH 12/19] docs(hermetic-build): indicate usage of Docker Buildkit
in development guide (#3337)
---
hermetic_build/DEVELOPMENT.md | 11 ++++++++---
1 file changed, 8 insertions(+), 3 deletions(-)
diff --git a/hermetic_build/DEVELOPMENT.md b/hermetic_build/DEVELOPMENT.md
index b6e78389dc..e54c57e9ec 100644
--- a/hermetic_build/DEVELOPMENT.md
+++ b/hermetic_build/DEVELOPMENT.md
@@ -160,13 +160,18 @@ python hermetic_build/library_generation/cli/entry_point.py generate \
1. Run the following command to build the image from source
```shell
- docker build \
+ DOCKER_BUILDKIT=1 docker build \
-f .cloudbuild/library_generation/library_generation.Dockerfile \
-t local:image-tag \
.
```
+ Please note that the build only works when using the new
+ [Docker BuildKit](https://docs.docker.com/build/buildkit/) (enabled through the `DOCKER_BUILDKIT` variable).
+ This is meant for local development only (in CloudTops) - GH Actions' Ubuntu-22.04 \
+ [comes with the latest Docker version](https://github.com/actions/runner-images/blob/e74605cd6d5407469cf224802f25057bafc23d70/images/ubuntu/Ubuntu2204-Readme.md?plain=1#L81-L83)
+ and is able to handle the build properly using the (updated) legacy builder.
-2. Set the version of gapic-generator-java
+3. Set the version of gapic-generator-java
```shell
LOCAL_GENERATOR_VERSION=$(mvn \
@@ -177,7 +182,7 @@ python hermetic_build/library_generation/cli/entry_point.py generate \
-q)
```
-3. Run the image
+4. Run the image
```shell
# Assume you want to generate the library in the current working directory
From a1db8b7b4ea58a86ebafc410912ace49a1eab11e Mon Sep 17 00:00:00 2001
From: Mridula <66699525+mpeddada1@users.noreply.github.com>
Date: Thu, 7 Nov 2024 12:02:13 -0500
Subject: [PATCH 13/19] chore: add options.logging in cloud build trigger
configs (#3342)
---
.cloudbuild/graalvm/cloudbuild-test-a-downstream-kms.yaml | 1 +
.../graalvm/cloudbuild-test-a-downstream-kmsinventory.yaml | 1 +
.cloudbuild/graalvm/cloudbuild-test-a.yaml | 4 +++-
.cloudbuild/graalvm/cloudbuild-test-b-downstream-kms.yaml | 1 +
.../graalvm/cloudbuild-test-b-downstream-kmsinventory.yaml | 1 +
.cloudbuild/graalvm/cloudbuild-test-b.yaml | 1 +
.cloudbuild/graalvm/cloudbuild.yaml | 2 ++
7 files changed, 10 insertions(+), 1 deletion(-)
diff --git a/.cloudbuild/graalvm/cloudbuild-test-a-downstream-kms.yaml b/.cloudbuild/graalvm/cloudbuild-test-a-downstream-kms.yaml
index 8b1efeb01e..ed15f58201 100644
--- a/.cloudbuild/graalvm/cloudbuild-test-a-downstream-kms.yaml
+++ b/.cloudbuild/graalvm/cloudbuild-test-a-downstream-kms.yaml
@@ -18,6 +18,7 @@ substitutions:
_JAVA_SHARED_CONFIG_VERSION: '1.11.3'
options:
machineType: 'E2_HIGHCPU_8'
+ logging: CLOUD_LOGGING_ONLY
steps:
- name: gcr.io/cloud-builders/docker
args: [
diff --git a/.cloudbuild/graalvm/cloudbuild-test-a-downstream-kmsinventory.yaml b/.cloudbuild/graalvm/cloudbuild-test-a-downstream-kmsinventory.yaml
index 9aab5964fd..a2455f3544 100644
--- a/.cloudbuild/graalvm/cloudbuild-test-a-downstream-kmsinventory.yaml
+++ b/.cloudbuild/graalvm/cloudbuild-test-a-downstream-kmsinventory.yaml
@@ -18,6 +18,7 @@ substitutions:
_JAVA_SHARED_CONFIG_VERSION: '1.11.3'
options:
machineType: 'E2_HIGHCPU_8'
+ logging: CLOUD_LOGGING_ONLY
steps:
- name: gcr.io/cloud-builders/docker
args: [
diff --git a/.cloudbuild/graalvm/cloudbuild-test-a.yaml b/.cloudbuild/graalvm/cloudbuild-test-a.yaml
index 6b0eec70d8..89df4c6f53 100644
--- a/.cloudbuild/graalvm/cloudbuild-test-a.yaml
+++ b/.cloudbuild/graalvm/cloudbuild-test-a.yaml
@@ -18,6 +18,7 @@ substitutions:
_JAVA_SHARED_CONFIG_VERSION: '1.11.3'
options:
machineType: 'E2_HIGHCPU_8'
+ logging: CLOUD_LOGGING_ONLY
steps:
- name: gcr.io/cloud-builders/docker
args: [
@@ -41,4 +42,5 @@ steps:
entrypoint: bash
args: [ './.kokoro/presubmit/showcase-native.sh' ]
waitFor: [ "graalvm-a-build" ]
- id: native-showcase
\ No newline at end of file
+ id: native-showcase
+
diff --git a/.cloudbuild/graalvm/cloudbuild-test-b-downstream-kms.yaml b/.cloudbuild/graalvm/cloudbuild-test-b-downstream-kms.yaml
index 7d5bdc407f..e0ab82af8d 100644
--- a/.cloudbuild/graalvm/cloudbuild-test-b-downstream-kms.yaml
+++ b/.cloudbuild/graalvm/cloudbuild-test-b-downstream-kms.yaml
@@ -18,6 +18,7 @@ substitutions:
_JAVA_SHARED_CONFIG_VERSION: '1.11.3'
options:
machineType: 'E2_HIGHCPU_8'
+ logging: CLOUD_LOGGING_ONLY
steps:
- name: gcr.io/cloud-builders/docker
args: [
diff --git a/.cloudbuild/graalvm/cloudbuild-test-b-downstream-kmsinventory.yaml b/.cloudbuild/graalvm/cloudbuild-test-b-downstream-kmsinventory.yaml
index 4f25eeef12..a0d0fecdc6 100644
--- a/.cloudbuild/graalvm/cloudbuild-test-b-downstream-kmsinventory.yaml
+++ b/.cloudbuild/graalvm/cloudbuild-test-b-downstream-kmsinventory.yaml
@@ -18,6 +18,7 @@ substitutions:
_JAVA_SHARED_CONFIG_VERSION: '1.11.3'
options:
machineType: 'E2_HIGHCPU_8'
+ logging: CLOUD_LOGGING_ONLY
steps:
- name: gcr.io/cloud-builders/docker
args: [
diff --git a/.cloudbuild/graalvm/cloudbuild-test-b.yaml b/.cloudbuild/graalvm/cloudbuild-test-b.yaml
index 030514d602..211b91a9df 100644
--- a/.cloudbuild/graalvm/cloudbuild-test-b.yaml
+++ b/.cloudbuild/graalvm/cloudbuild-test-b.yaml
@@ -18,6 +18,7 @@ substitutions:
_JAVA_SHARED_CONFIG_VERSION: '1.11.3'
options:
machineType: 'E2_HIGHCPU_8'
+ logging: CLOUD_LOGGING_ONLY
steps:
- name: gcr.io/cloud-builders/docker
args: [
diff --git a/.cloudbuild/graalvm/cloudbuild.yaml b/.cloudbuild/graalvm/cloudbuild.yaml
index 85f2ba73c2..e9c573815b 100644
--- a/.cloudbuild/graalvm/cloudbuild.yaml
+++ b/.cloudbuild/graalvm/cloudbuild.yaml
@@ -45,6 +45,8 @@ steps:
id: graalvm-b-build
waitFor: [ "-" ]
+options:
+ logging: CLOUD_LOGGING_ONLY
images:
- gcr.io/cloud-devrel-public-resources/graalvm_sdk_platform_a:${_SHARED_DEPENDENCIES_VERSION}
From 7f6e470fea1673a5cf50fe3b49263615a172afde Mon Sep 17 00:00:00 2001
From: ldetmer <1771267+ldetmer@users.noreply.github.com>
Date: Thu, 7 Nov 2024 17:24:35 -0500
Subject: [PATCH 14/19] fix: protobuf version not always getting set in headers
(#3322)
In order to maximize amount of times protobuf version is logged we have
updated logic as follows:
1) First try to use protbuf RuntimeVersion which is available in
protobuf 4.x
2) if not available try to read using manifest file
3) if manifest file doesn't not exist then default to "3" as we know
runtime < 4
Also updated showcase test to fail in the case protobuf version is
missing
Tested:
1) maven build using client library directly and overriding protobuf
2) spring boot started project + google cloud storage client client lib,
validated that protobuf header was sent
---
.../google/api/gax/core/GaxProperties.java | 27 ++++++++++-
.../api/gax/rpc/ApiClientHeaderProvider.java | 2 +-
.../com.google.api/gax/reflect-config.json | 10 ++++
.../api/gax/core/GaxPropertiesTest.java | 46 ++++++++++++++++---
.../showcase/v1beta1/it/ITVersionHeaders.java | 4 +-
5 files changed, 79 insertions(+), 10 deletions(-)
create mode 100644 gax-java/gax/src/main/resources/META-INF/native-image/com.google.api/gax/reflect-config.json
diff --git a/gax-java/gax/src/main/java/com/google/api/gax/core/GaxProperties.java b/gax-java/gax/src/main/java/com/google/api/gax/core/GaxProperties.java
index f15046afcb..994ba2eb82 100644
--- a/gax-java/gax/src/main/java/com/google/api/gax/core/GaxProperties.java
+++ b/gax-java/gax/src/main/java/com/google/api/gax/core/GaxProperties.java
@@ -49,7 +49,7 @@ public class GaxProperties {
private static final String GAX_VERSION = getLibraryVersion(GaxProperties.class, "version.gax");
private static final String JAVA_VERSION = getRuntimeVersion();
private static final String PROTOBUF_VERSION =
- getBundleVersion(Any.class).orElse(DEFAULT_VERSION);
+ getProtobufVersion(Any.class, "com.google.protobuf.RuntimeVersion");;
private GaxProperties() {}
@@ -148,4 +148,29 @@ static Optional getBundleVersion(Class clazz) {
return Optional.empty();
}
}
+
+ /**
+ * Returns the Protobuf runtime version as reported by com.google.protobuf.RuntimeVersion, if
+ * class is available, otherwise by reading from MANIFEST file. If niether option is available
+ * defaults to protobuf version 3 as RuntimeVersion class is available in protobuf version 4+
+ */
+ @VisibleForTesting
+ static String getProtobufVersion(Class clazz, String protobufRuntimeVersionClassName) {
+ try {
+ Class protobufRuntimeVersionClass = Class.forName(protobufRuntimeVersionClassName);
+ return protobufRuntimeVersionClass.getField("MAJOR").get(null)
+ + "."
+ + protobufRuntimeVersionClass.getField("MINOR").get(null)
+ + "."
+ + protobufRuntimeVersionClass.getField("PATCH").get(null);
+ } catch (ClassNotFoundException
+ | NoSuchFieldException
+ | IllegalAccessException
+ | SecurityException
+ | NullPointerException e) {
+ // If manifest file is not available default to protobuf generic version 3 as we know
+ // RuntimeVersion class is available in protobuf jar 4+.
+ return getBundleVersion(clazz).orElse("3");
+ }
+ }
}
diff --git a/gax-java/gax/src/main/java/com/google/api/gax/rpc/ApiClientHeaderProvider.java b/gax-java/gax/src/main/java/com/google/api/gax/rpc/ApiClientHeaderProvider.java
index 35307764d2..52f923e111 100644
--- a/gax-java/gax/src/main/java/com/google/api/gax/rpc/ApiClientHeaderProvider.java
+++ b/gax-java/gax/src/main/java/com/google/api/gax/rpc/ApiClientHeaderProvider.java
@@ -88,7 +88,7 @@ private static String checkAndAppendProtobufVersionIfNecessary(
// TODO(b/366417603): appending protobuf version to existing client library token until resolved
Pattern pattern = Pattern.compile("(gccl|gapic)\\S*");
Matcher matcher = pattern.matcher(apiClientHeaderValue);
- if (matcher.find()) {
+ if (matcher.find() && GaxProperties.getProtobufVersion() != null) {
return apiClientHeaderValue.substring(0, matcher.end())
+ "--"
+ PROTOBUF_HEADER_VERSION_KEY
diff --git a/gax-java/gax/src/main/resources/META-INF/native-image/com.google.api/gax/reflect-config.json b/gax-java/gax/src/main/resources/META-INF/native-image/com.google.api/gax/reflect-config.json
new file mode 100644
index 0000000000..3a38d53361
--- /dev/null
+++ b/gax-java/gax/src/main/resources/META-INF/native-image/com.google.api/gax/reflect-config.json
@@ -0,0 +1,10 @@
+[
+ {
+ "name": "com.google.protobuf.RuntimeVersion",
+ "fields" : [
+ { "name" : "MAJOR" },
+ { "name" : "MINOR" },
+ { "name" : "PATCH" }
+ ]
+ }
+]
\ No newline at end of file
diff --git a/gax-java/gax/src/test/java/com/google/api/gax/core/GaxPropertiesTest.java b/gax-java/gax/src/test/java/com/google/api/gax/core/GaxPropertiesTest.java
index 1369ec35ae..6560df4bc1 100644
--- a/gax-java/gax/src/test/java/com/google/api/gax/core/GaxPropertiesTest.java
+++ b/gax-java/gax/src/test/java/com/google/api/gax/core/GaxPropertiesTest.java
@@ -35,6 +35,7 @@
import static org.junit.jupiter.api.Assertions.assertTrue;
import com.google.common.base.Strings;
+import com.google.protobuf.Any;
import java.io.IOException;
import java.util.Optional;
import java.util.regex.Pattern;
@@ -160,12 +161,8 @@ void testGetJavaRuntimeInfo_nullJavaVersion() {
@Test
public void testGetProtobufVersion() throws IOException {
- Version version = readVersion(GaxProperties.getProtobufVersion());
-
- assertTrue(version.major >= 3);
- if (version.major == 3) {
- assertTrue(version.minor >= 25);
- }
+ assertTrue(
+ Pattern.compile("^\\d+\\.\\d+\\.\\d+").matcher(GaxProperties.getProtobufVersion()).find());
}
@Test
@@ -175,6 +172,36 @@ public void testGetBundleVersion_noManifestFile() throws IOException {
assertFalse(version.isPresent());
}
+ @Test
+ void testGetProtobufVersion_success() {
+ String version =
+ GaxProperties.getProtobufVersion(
+ Any.class, "com.google.api.gax.core.GaxPropertiesTest$RuntimeVersion");
+
+ assertEquals("3.13.6", version);
+ }
+
+ @Test
+ void testGetProtobufVersion_classNotFoundException() throws Exception {
+ String version = GaxProperties.getProtobufVersion(Any.class, "foo.NonExistantClass");
+
+ assertTrue(Pattern.compile("^\\d+\\.\\d+\\.\\d+").matcher(version).find());
+ }
+
+ @Test
+ void testgetProtobufVersion_noSuchFieldException() throws Exception {
+ String version = GaxProperties.getProtobufVersion(Any.class, "java.lang.Class");
+
+ assertTrue(Pattern.compile("^\\d+\\.\\d+\\.\\d+").matcher(version).find());
+ }
+
+ @Test
+ void testGetProtobufVersion_noManifest() throws Exception {
+ String version = GaxProperties.getProtobufVersion(GaxProperties.class, "foo.NonExistantClass");
+
+ assertEquals("3", version);
+ }
+
private Version readVersion(String version) {
assertTrue(Pattern.compile("^\\d+\\.\\d+\\.\\d+").matcher(version).find());
String[] versionComponents = version.split("\\.");
@@ -194,4 +221,11 @@ public Version(int major, int minor) {
this.minor = minor;
}
}
+
+ // Test class that emulates com.google.protobuf.RuntimeVersion for reflection lookup of fields
+ class RuntimeVersion {
+ public static final int MAJOR = 3;
+ public static final int MINOR = 13;
+ public static final int PATCH = 6;
+ }
}
diff --git a/showcase/gapic-showcase/src/test/java/com/google/showcase/v1beta1/it/ITVersionHeaders.java b/showcase/gapic-showcase/src/test/java/com/google/showcase/v1beta1/it/ITVersionHeaders.java
index 09255fe278..303cab98ec 100644
--- a/showcase/gapic-showcase/src/test/java/com/google/showcase/v1beta1/it/ITVersionHeaders.java
+++ b/showcase/gapic-showcase/src/test/java/com/google/showcase/v1beta1/it/ITVersionHeaders.java
@@ -241,7 +241,7 @@ void testHttpJsonCompliance_userApiVersionSetSuccess() throws IOException {
@Test
void testGrpcCall_sendsCorrectApiClientHeader() {
Pattern defautlGrpcHeaderPattern =
- Pattern.compile("gl-java/.* gapic/.*?--protobuf-.* gax/.* grpc/.* protobuf/.*");
+ Pattern.compile("gl-java/.* gapic/.*?--protobuf-\\d.* gax/.* grpc/.* protobuf/\\d.*");
grpcClient.echo(EchoRequest.newBuilder().build());
String headerValue = grpcInterceptor.metadata.get(API_CLIENT_HEADER_KEY);
assertTrue(defautlGrpcHeaderPattern.matcher(headerValue).matches());
@@ -250,7 +250,7 @@ void testGrpcCall_sendsCorrectApiClientHeader() {
@Test
void testHttpJson_sendsCorrectApiClientHeader() {
Pattern defautlHttpHeaderPattern =
- Pattern.compile("gl-java/.* gapic/.*?--protobuf-.* gax/.* rest/ protobuf/.*");
+ Pattern.compile("gl-java/.* gapic/.*?--protobuf-\\d.* gax/.* rest/ protobuf/\\d.*");
httpJsonClient.echo(EchoRequest.newBuilder().build());
ArrayList headerValues =
(ArrayList)
From b0adcf91887a626fc4b28629ebd9bbdba2772819 Mon Sep 17 00:00:00 2001
From: Joe Wang <106995533+JoeWang1127@users.noreply.github.com>
Date: Thu, 7 Nov 2024 19:22:54 -0500
Subject: [PATCH 15/19] chore: randomize temp output directory (#3341)
In this PR:
- Use `$RANDOM` (a shell variable that generates a random integer
between 0 and 32767) to randomize temp output directory to support
parallel generation.
---
hermetic_build/library_generation/generate_library.sh | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/hermetic_build/library_generation/generate_library.sh b/hermetic_build/library_generation/generate_library.sh
index 313b8243f8..f5cae1ba5a 100755
--- a/hermetic_build/library_generation/generate_library.sh
+++ b/hermetic_build/library_generation/generate_library.sh
@@ -115,7 +115,7 @@ if [ -z "${os_architecture}" ]; then
os_architecture=$(detect_os_architecture)
fi
-temp_destination_path="${output_folder}/temp_preprocessed"
+temp_destination_path="${output_folder}/temp_preprocessed-$RANDOM"
mkdir -p "${output_folder}/${destination_path}"
if [ -d "${temp_destination_path}" ]; then
# we don't want the preprocessed sources of a previous run
From 222fb452e00bd195ad51389ea308993a7e1bc956 Mon Sep 17 00:00:00 2001
From: Diego Marquez
Date: Thu, 7 Nov 2024 21:35:09 -0500
Subject: [PATCH 16/19] fix: use BuildKit instead of legacy builder to build
the Hermetic Build images (#3338)
Fixes the following error:
[Docker Buildkit](https://docs.docker.com/build/buildkit/) is a more
advanced version and more actively maintained. We confirmed locally that
the error occurs with an old version of the docker builder and that it
is fixed when using `DOCKER_BUILDKIT=1`
---
.../cloudbuild-library-generation-push-prod.yaml | 2 ++
.../library_generation/cloudbuild-library-generation-push.yaml | 2 ++
hermetic_build/DEVELOPMENT.md | 3 ---
hermetic_build/library_generation/tests/integration_tests.py | 1 +
4 files changed, 5 insertions(+), 3 deletions(-)
diff --git a/.cloudbuild/library_generation/cloudbuild-library-generation-push-prod.yaml b/.cloudbuild/library_generation/cloudbuild-library-generation-push-prod.yaml
index b3d6783087..1499c9053b 100644
--- a/.cloudbuild/library_generation/cloudbuild-library-generation-push-prod.yaml
+++ b/.cloudbuild/library_generation/cloudbuild-library-generation-push-prod.yaml
@@ -30,6 +30,8 @@ steps:
"--file", ".cloudbuild/library_generation/library_generation.Dockerfile", "."]
id: library-generation-build
waitFor: ["-"]
+ env:
+ - 'DOCKER_BUILDKIT=1'
options:
logging: CLOUD_LOGGING_ONLY
diff --git a/.cloudbuild/library_generation/cloudbuild-library-generation-push.yaml b/.cloudbuild/library_generation/cloudbuild-library-generation-push.yaml
index 012cd26817..359c021f69 100644
--- a/.cloudbuild/library_generation/cloudbuild-library-generation-push.yaml
+++ b/.cloudbuild/library_generation/cloudbuild-library-generation-push.yaml
@@ -30,6 +30,8 @@ steps:
"--file", ".cloudbuild/library_generation/library_generation.Dockerfile", "."]
id: library-generation-build
waitFor: ["-"]
+ env:
+ - 'DOCKER_BUILDKIT=1'
options:
logging: CLOUD_LOGGING_ONLY
diff --git a/hermetic_build/DEVELOPMENT.md b/hermetic_build/DEVELOPMENT.md
index e54c57e9ec..4ba3d4b9af 100644
--- a/hermetic_build/DEVELOPMENT.md
+++ b/hermetic_build/DEVELOPMENT.md
@@ -167,9 +167,6 @@ python hermetic_build/library_generation/cli/entry_point.py generate \
```
Please note that the build only works when using the new
[Docker BuildKit](https://docs.docker.com/build/buildkit/) (enabled through the `DOCKER_BUILDKIT` variable).
- This is meant for local development only (in CloudTops) - GH Actions' Ubuntu-22.04 \
- [comes with the latest Docker version](https://github.com/actions/runner-images/blob/e74605cd6d5407469cf224802f25057bafc23d70/images/ubuntu/Ubuntu2204-Readme.md?plain=1#L81-L83)
- and is able to handle the build properly using the (updated) legacy builder.
3. Set the version of gapic-generator-java
diff --git a/hermetic_build/library_generation/tests/integration_tests.py b/hermetic_build/library_generation/tests/integration_tests.py
index 5a1d5f7e53..a940069baa 100644
--- a/hermetic_build/library_generation/tests/integration_tests.py
+++ b/hermetic_build/library_generation/tests/integration_tests.py
@@ -202,6 +202,7 @@ def __build_image(cls, docker_file: str, cwd: str):
subprocess.check_call(
["docker", "build", "-f", docker_file, "-t", image_tag, "."],
cwd=cwd,
+ env=dict(os.environ, DOCKER_BUILDKIT="1"),
)
@classmethod
From b7d62968cd837a7addc06da8b9bc2131c36c7fbc Mon Sep 17 00:00:00 2001
From: Joe Wang <106995533+JoeWang1127@users.noreply.github.com>
Date: Fri, 8 Nov 2024 09:47:26 -0500
Subject: [PATCH 17/19] docs: fix list num (#3356)
In this PR:
- Fix list num in `DEVELOPMENT.md`.
---
hermetic_build/DEVELOPMENT.md | 10 +++++-----
1 file changed, 5 insertions(+), 5 deletions(-)
diff --git a/hermetic_build/DEVELOPMENT.md b/hermetic_build/DEVELOPMENT.md
index 4ba3d4b9af..1969bbcdfb 100644
--- a/hermetic_build/DEVELOPMENT.md
+++ b/hermetic_build/DEVELOPMENT.md
@@ -165,10 +165,10 @@ python hermetic_build/library_generation/cli/entry_point.py generate \
-t local:image-tag \
.
```
- Please note that the build only works when using the new
- [Docker BuildKit](https://docs.docker.com/build/buildkit/) (enabled through the `DOCKER_BUILDKIT` variable).
+ Please note that the build only works when using the new [Docker BuildKit](https://docs.docker.com/build/buildkit/)
+ (enabled through the `DOCKER_BUILDKIT` variable).
-3. Set the version of gapic-generator-java
+2. Set the version of gapic-generator-java
```shell
LOCAL_GENERATOR_VERSION=$(mvn \
@@ -179,7 +179,7 @@ python hermetic_build/library_generation/cli/entry_point.py generate \
-q)
```
-4. Run the image
+3. Run the image
```shell
# Assume you want to generate the library in the current working directory
@@ -196,7 +196,7 @@ python hermetic_build/library_generation/cli/entry_point.py generate \
--library-names=apigee-connect,asset \
--repository-path=/workspace \
--api-definitions-path=/workspace/apis
- ```
+ ```
Note that if you specify the generator version using environment variable,
`-e GENERATOR_VERSION="${LOCAL_GENERATOR_VERSION}"` in the above example,
you should not set `gapic_generator_version` and `protoc_version` in the
From 816b1608e52543b4b5ade40056c89740fe35a459 Mon Sep 17 00:00:00 2001
From: Joe Wang <106995533+JoeWang1127@users.noreply.github.com>
Date: Fri, 8 Nov 2024 09:47:37 -0500
Subject: [PATCH 18/19] chore: remove unused variable (#3357)
In this PR:
- Remove unused variable in `hermetic_library_generation.sh`.
---
.github/scripts/hermetic_library_generation.sh | 3 ---
1 file changed, 3 deletions(-)
diff --git a/.github/scripts/hermetic_library_generation.sh b/.github/scripts/hermetic_library_generation.sh
index 43b764fe34..3c2a7d0e47 100755
--- a/.github/scripts/hermetic_library_generation.sh
+++ b/.github/scripts/hermetic_library_generation.sh
@@ -81,9 +81,6 @@ git checkout "${current_branch}"
# copy generation configuration from target branch to current branch.
git show "${target_branch}":"${generation_config}" > "${baseline_generation_config}"
-# get .m2 folder so it's mapped into the docker container
-m2_folder=$(dirname "$(mvn help:evaluate -Dexpression=settings.localRepository -q -DforceStdout)")
-
# download api definitions from googleapis repository
googleapis_commitish=$(grep googleapis_commitish "${generation_config}" | cut -d ":" -f 2 | xargs)
api_def_dir=$(mktemp -d)
From a31c68232584bf90bc00ace8310adeab8fa26add Mon Sep 17 00:00:00 2001
From: Mend Renovate
Date: Wed, 13 Nov 2024 19:12:22 +0100
Subject: [PATCH 19/19] deps: update google auth library dependencies to
v1.30.0 (#3367)
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit
This PR contains the following updates:
| Package | Change | Age | Adoption | Passing | Confidence |
|---|---|---|---|---|---|
|
[com.google.auth:google-auth-library-credentials](https://redirect.github.com/googleapis/google-auth-library-java)
| `1.29.0` -> `1.30.0` |
[](https://docs.renovatebot.com/merge-confidence/)
|
[](https://docs.renovatebot.com/merge-confidence/)
|
[](https://docs.renovatebot.com/merge-confidence/)
|
[](https://docs.renovatebot.com/merge-confidence/)
|
|
[com.google.auth:google-auth-library-bom](https://redirect.github.com/googleapis/google-auth-library-java)
| `1.29.0` -> `1.30.0` |
[](https://docs.renovatebot.com/merge-confidence/)
|
[](https://docs.renovatebot.com/merge-confidence/)
|
[](https://docs.renovatebot.com/merge-confidence/)
|
[](https://docs.renovatebot.com/merge-confidence/)
|
|
[com.google.auth:google-auth-library-oauth2-http](https://redirect.github.com/googleapis/google-auth-library-java)
| `1.29.0` -> `1.30.0` |
[](https://docs.renovatebot.com/merge-confidence/)
|
[](https://docs.renovatebot.com/merge-confidence/)
|
[](https://docs.renovatebot.com/merge-confidence/)
|
[](https://docs.renovatebot.com/merge-confidence/)
|
---
> [!WARNING]
> Some dependencies could not be looked up. Check the Dependency
Dashboard for more information.
---
### Release Notes
googleapis/google-auth-library-java
(com.google.auth:google-auth-library-credentials)
###
[`v1.30.0`](https://redirect.github.com/googleapis/google-auth-library-java/blob/HEAD/CHANGELOG.md#1300-2024-11-08)
##### Features
- Support querying S2A Addresses from MDS
([#1400](https://redirect.github.com/googleapis/google-auth-library-java/issues/1400))
([df06bd1](https://redirect.github.com/googleapis/google-auth-library-java/commit/df06bd1f94d03c4f8807c2adf42d25d29b731531))
##### Bug Fixes
- Make it explicit that there is a network call to MDS to get
SecureSessionAgentConfig
([#1573](https://redirect.github.com/googleapis/google-auth-library-java/issues/1573))
([18020fe](https://redirect.github.com/googleapis/google-auth-library-java/commit/18020fedb855742ee27b6558f5de58d3818c6b48))
---
### Configuration
📅 **Schedule**: Branch creation - At any time (no schedule defined),
Automerge - At any time (no schedule defined).
🚦 **Automerge**: Disabled by config. Please merge this manually once you
are satisfied.
♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the
rebase/retry checkbox.
🔕 **Ignore**: Close this PR and you won't be reminded about these
updates again.
---
- [ ] If you want to rebase/retry this PR, check
this box
---
This PR was generated by [Mend Renovate](https://mend.io/renovate/).
View the [repository job
log](https://developer.mend.io/github/googleapis/sdk-platform-java).
---
gapic-generator-java-pom-parent/pom.xml | 2 +-
gax-java/dependencies.properties | 2 +-
2 files changed, 2 insertions(+), 2 deletions(-)
diff --git a/gapic-generator-java-pom-parent/pom.xml b/gapic-generator-java-pom-parent/pom.xml
index 1991f17d01..cbf713fe68 100644
--- a/gapic-generator-java-pom-parent/pom.xml
+++ b/gapic-generator-java-pom-parent/pom.xml
@@ -27,7 +27,7 @@
consistent across modules in this repository -->
1.3.2
1.68.1
- 1.29.0
+ 1.30.0
1.45.0
2.11.0
33.3.1-jre
diff --git a/gax-java/dependencies.properties b/gax-java/dependencies.properties
index 689a2f9d36..27509afd5c 100644
--- a/gax-java/dependencies.properties
+++ b/gax-java/dependencies.properties
@@ -38,7 +38,7 @@ version.io_grpc=1.68.1
maven.com_google_api_grpc_proto_google_common_protos=com.google.api.grpc:proto-google-common-protos:2.46.0
maven.com_google_api_grpc_grpc_google_common_protos=com.google.api.grpc:grpc-google-common-protos:2.46.0
maven.com_google_auth_google_auth_library_oauth2_http=com.google.auth:google-auth-library-oauth2-http:1.29.0
-maven.com_google_auth_google_auth_library_credentials=com.google.auth:google-auth-library-credentials:1.29.0
+maven.com_google_auth_google_auth_library_credentials=com.google.auth:google-auth-library-credentials:1.30.0
maven.io_opentelemetry_opentelemetry_api=io.opentelemetry:opentelemetry-api:1.42.1
maven.io_opencensus_opencensus_api=io.opencensus:opencensus-api:0.31.1
maven.io_opencensus_opencensus_contrib_grpc_metrics=io.opencensus:opencensus-contrib-grpc-metrics:0.31.1