From 25023afad8f483a3eae8846c4bc1fbe2c7a260c5 Mon Sep 17 00:00:00 2001 From: Joe Wang <106995533+JoeWang1127@users.noreply.github.com> Date: Tue, 5 Nov 2024 11:44:40 -0500 Subject: [PATCH] docs: modify hermetic build docs (#3331) In this PR: - Add instructions on release note generation. - Move generating libraries using docker image to `README.md` - Move generating libraries using python scripts to `DEVELOPMENT.md`. - Move instructions on building image to `DEVELOPMENT.md`. --- hermetic_build/DEVELOPMENT.md | 228 ++++++++++++++++++ .../{library_generation => }/README.md | 205 +++++++++++----- .../library_generation/DEVELOPMENT.md | 207 ---------------- .../cli/generate_release_note.py | 5 +- 4 files changed, 382 insertions(+), 263 deletions(-) create mode 100644 hermetic_build/DEVELOPMENT.md rename hermetic_build/{library_generation => }/README.md (67%) delete mode 100644 hermetic_build/library_generation/DEVELOPMENT.md diff --git a/hermetic_build/DEVELOPMENT.md b/hermetic_build/DEVELOPMENT.md new file mode 100644 index 0000000000..b6e78389dc --- /dev/null +++ b/hermetic_build/DEVELOPMENT.md @@ -0,0 +1,228 @@ +> [!IMPORTANT] +> All examples assume you are inside the repository root folder. + + +# Linting + +When contributing, ensure your changes to python code have a valid format. + +``` +python -m pip install black +black {source_file_or_directory} +``` + +# Install package dependencies + +```shell +python -m pip install --require-hashes -r hermetic_build/common/requirements.txt +python -m pip install hermetic_build/common +python -m pip install --require-hashes -r hermetic_build/library_generation/requirements.txt +python -m pip install hermetic_build/library_generation +python -m pip install --require-hashes -r hermetic_build/release_note_generation/requirements.txt +python -m pip install hermetic_build/release_note_generation +``` + +# Run the integration tests + +The integration tests build the docker image declared in +`.cloudbuild/library_generation/library_generation.Dockerfile`, pull GAPIC +repositories, generate the libraries and compare the results with the source +code declared in a "golden branch" of the repo. + +It requires docker and python (>= 3.12.0) to be installed. + +```shell +python -m unittest hermetic_build/library_generation/tests/integration_tests.py +``` + +# Run the unit tests + +There is one unit test file per component. +Every unit test script ends with `unit_tests.py`. +To avoid specifying them individually, we can use the following command: + +```shell +python -m unittest discover -s hermetic_build -p "*unit_tests.py" +``` + +> [!NOTE] +> The output of this command may look erratic during the first 30 seconds. +> This is normal. After the tests are done, an "OK" message should be shown. + +# Run the library generation scripts in your local environment + +Although the scripts are designed to run in a Docker container, you can also +run them directly. +This section explains how to run the entrypoint script +(`hermetic_build/library_generation/cli/entry_point.py`). + +## Assumptions made by the scripts + +### The Hermetic Build's well-known folder +Located in `${HOME}/.library_generation`, this folder is assumed by the scripts +to contain certain tools. + +Developers must make sure this folder is properly configured before running the +scripts locally. +Note that this relies on the `HOME` environment variable which is always defined +as per [POSIX env var definition](https://pubs.opengroup.org/onlinepubs/9699919799/basedefs/V1_chap08.html). + +#### Put the gapic-generator-java jar in its well-known location + +1. Run the following command to install gapic-generator-java. + + ```shell + mvn install -B -ntp -DskipTests -Dclirr.skip -Dcheckstyle.skip + ``` + This will generate a jar located in `~/.m2/repository/com/google/api/gapic-generator-java/{version}/gapic-generator-java-{version}.jar` + +2. Move the jar into its well-known location. + + ```shell + mv /path/to/jar "${HOME}/.library_generation/gapic-generator-java.jar" + ``` + +#### Put the java formatter jar in its well-known location + +1. Download google-java-format-{version}-all-deps.jar from [Maven Central](https://central.sonatype.com/artifact/com.google.googlejavaformat/google-java-format) +or [GitHub releases](https://github.com/google/google-java-format/releases). +2. Move the jar into its well-known location. + + ```shell + mv /path/to/jar "${HOME}/.library_generation/google-java-format.jar" + ``` + +## Installing prerequisites + +In order to run the generation scripts directly, there are a few tools we +need to install beforehand. + +### Install the owl-bot CLI + +This requires node.js to be installed. +Check this [installation guide](https://github.com/nvm-sh/nvm?tab=readme-ov-file#install--update-script) +for NVM, Node.js's version manager. + +After you install it, you can install the owl-bot CLI with the following +commands: + +```shell +git clone https://github.com/googleapis/repo-automation-bots +cd repo-automation-bots/packages/owl-bot +npm i && npm run compile && npm link +owl-bot copy-code --version +``` + +The key step is `npm link`, which will make the command available in you current +shell session. + +## Run the script +The entrypoint script (`hermetic_build/library_generation/cli/entry_point.py`) +allows you to generate a GAPIC repository with a given api definition (proto, +service yaml). + +### Download the api definition +For example, from googleapis + +```shell +git clone https://github.com/googleapis/googleapis +export api_definitions_path="$(pwd)/googleapis" +``` + +### Download the repo +For example, google-cloud-java +```shell +git clone https://github.com/googleapis/google-cloud-java +export path_to_repo="$(pwd)/google-cloud-java" +``` + +### Install the scripts + +You can skip this step if you've installed the packages in [Install package dependencies](#install-package-dependencies). + +```shell +python -m pip install --require-hashes -r hermetic_build/common/requirements.txt +python -m pip install hermetic_build/common +python -m pip install --require-hashes -r hermetic_build/library_generation/requirements.txt +python -m pip install hermetic_build/library_generation +``` + +### Run the script + +```shell +python hermetic_build/library_generation/cli/entry_point.py generate \ + --repository-path="${path_to_repo}" \ + --api-definitions-path="${api_definitions_path}" +``` + +# Build the image from source + +1. Run the following command to build the image from source + + ```shell + docker build \ + -f .cloudbuild/library_generation/library_generation.Dockerfile \ + -t local:image-tag \ + . + ``` + +2. Set the version of gapic-generator-java + + ```shell + LOCAL_GENERATOR_VERSION=$(mvn \ + org.apache.maven.plugins:maven-help-plugin:evaluate \ + -Dexpression=project.version \ + -pl gapic-generator-java \ + -DforceStdout \ + -q) + ``` + +3. Run the image + + ```shell + # Assume you want to generate the library in the current working directory + # and the generation configuration is in the same directory. + docker run \ + --rm \ + --quiet \ + -u "$(id -u):$(id -g)" \ + -v "$(pwd):/workspace" \ + -v /path/to/api-definitions:/workspace/apis \ + -e GENERATOR_VERSION="${LOCAL_GENERATOR_VERSION}" \ + local:image-tag \ + --generation-config-path=/workspace/generation_config_file \ + --library-names=apigee-connect,asset \ + --repository-path=/workspace \ + --api-definitions-path=/workspace/apis + ``` + Note that if you specify the generator version using environment variable, + `-e GENERATOR_VERSION="${LOCAL_GENERATOR_VERSION}"` in the above example, + you should not set `gapic_generator_version` and `protoc_version` in the + generation configuration because values in the generation configuration will + take precedence. + +# Debug the library generation container +If you are working on changing the way the containers are created, you may want +to inspect the containers to check the setup. +It would be convenient in such case to have a text editor/viewer available. +You can achieve this by modifying the Dockerfile as follows: + +```dockerfile +# install OS tools +RUN apk update && apk add \ + unzip curl rsync openjdk11 jq bash nodejs npm git less vim +``` + +We add `less` and `vim` as text tools for further inspection. + +You can also run a shell in a new container by running: + +```shell +docker run \ + --rm \ + -it \ + -u $(id -u):$(id -g) \ + -v /path/to/google-cloud-java:/workspace \ + --entrypoint="bash" \ + $(cat image-id) +``` diff --git a/hermetic_build/library_generation/README.md b/hermetic_build/README.md similarity index 67% rename from hermetic_build/library_generation/README.md rename to hermetic_build/README.md index 0b4208ac3e..583e766bdf 100644 --- a/hermetic_build/library_generation/README.md +++ b/hermetic_build/README.md @@ -1,44 +1,43 @@ +> [!IMPORTANT] +> All scripts/examples assume you are inside the repository root folder. + # Generate a repository containing GAPIC Client Libraries -The script, `entry_point.py`, allows you to generate a repository containing -GAPIC client libraries (a monorepo, for example, google-cloud-java) from a -configuration file. +Running the docker image built from `hermetic_build/library_generation` +directory, you can generate a repository containing GAPIC client libraries (a +monorepo, for example, google-cloud-java) from a configuration file. + +Instead of running the docker image, if you prefer running the underlying python +scripts directly, please refer to the [development guide](DEVELOPMENT.md#run-the-script) +for additional instructions. ## Environment - OS: Linux -- Java runtime environment (8 or above) -- Python (3.12 or above) -- Docker -- Git +- Docker -## Prerequisite +## Prerequisites In order to generate a version for each library, a versions.txt has to exist -in `repository_path`. -Please refer to [Repository path](#repository-path--repositorypath---optional) for more information. - -## Parameters to generate a repository using `entry_point.py` - -### Baseline generation configuration yaml (`baseline_generation_config`) +in `repository-path`. +Please refer to [Repository path](#repository-path--repositorypath---optional) +for more information. -An absolute or relative path to a generation_config.yaml. -This config file is used for computing changed libraries, not library -generation. +## Parameters to generate a repository using the docker image -### Current generation configuration yaml (`current_generation_config`) +### Generation configuration yaml (`generation-config-path`) An absolute or relative path to a configuration file containing parameters to generate the repository. -Please refer [Configuration to generate a repository](#configuration-to-generate-a-repository) +Please refer to [Configuration to generate a repository](#configuration-to-generate-a-repository) for more information. -### Repository path (`repository_path`), optional +### Repository path (`repository-path`), optional The path to where the generated repository goes. The default value is the current working directory when running the script. -For example, `cd google-cloud-java && python entry_point.py ...` without +For example, `cd google-cloud-java && python /path/to/entry_point.py ...` without specifying the `--repository_path` option will modify the `google-cloud-java` repository the user `cd`'d into. @@ -47,28 +46,37 @@ right version for each library. Please refer [here](go/java-client-releasing#versionstxt-manifest) for more info of versions.txt. -### Api definitions path (`api_definitions_path`), optional +### A list of library names (`library-names`), optional + +A list of library names that will be generated, separated by comma. +The library name of a library is the value of `library_name` or `api_shortname`, +if `library_name` is not specified, in the generation configuration. + +If not specified, all libraries in the generation +configuration will be generated. + +### Api definitions path (`api-definitions-path`), optional The path to where the api definition (proto, service yaml) resides. The default value is the current working directory when running the script. -Note that you need not only the protos defined the service, but also the transitive -dependencies of those protos. +**Note that you need not only the protos defined the service, but also the +transitive dependencies of those protos.** Any missing dependencies will cause `File not found` error. -For example, if your service is defined in `example_service.proto` and it imports -`google/api/annotations.proto`, you need the `annotations.proto` resides in a -folder that has the exact structure of the import statement (`google/api` in this -case), and set `api_definitions_path` to the path contains the root folder (`google` -in this case). +For example, if your service is defined in `example_service.proto` and it +imports `google/api/annotations.proto`, you need the `annotations.proto` resides +in a folder that has the exact structure of the import statement (`google/api` +in this case), and set `api_definitions_path` to the path contains the root +folder (`google` in this case). -## Output of `entry_point.py` +## Output ### GAPIC libraries -For each module (e.g. `google-cloud-java/java-asset`), the following files/folders -will be created/modified: +For each module (e.g. `google-cloud-java/java-asset`), the following +files/folders will be created/modified: | Name | Notes | |:------------------------------------|:-------------------------------------------------------------------------| @@ -185,32 +193,58 @@ libraries: - proto_path: google/cloud/asset/v1p7beta1 ``` -# Local Environment Setup before running `entry_point.py` - -1. Assuming Python 3 is installed, follow official guide from [Python.org](https://packaging.python.org/en/latest/guides/installing-using-pip-and-virtual-environments/#create-and-use-virtual-environments) to create a virtual environment. -The virtual environment can be installed to any folder, usually it is recommended to be installed under the root folder of the project(`sdk-platform-java` in this case). -2. Assuming the virtual environment is installed under `sdk-platform-java`. -Run the following command under the root folder of `sdk-platform-java` to install `library_generation` and its dependencies. +# Run the library generation image + +1. Download the API definitions to a local directory, e.g., from [googleapis](https://github.com/googleapis/googleapis). + +2. Run the docker image. + ```shell + # Assume you want to generate the library in the current working directory + # and the generation configuration is in the same directory. + docker run \ + --rm \ + --quiet \ + -u "$(id -u):$(id -g)" \ + -v "$(pwd):/workspace" \ + -v /path/to/api_definition:/workspace \ + gcr.io/cloud-devrel-public-resources/java-library-generation:image-tag + ``` - ```bash - python -m pip install --require-hashes -r hermetic_build/common/requirements.txt - python -m pip install hermetic_build/common - python -m pip install --require-hashes -r hermetic_build/library_generation/requirements.txt - python -m pip install hermetic_build/library_generation + * `-u "$(id -u)":"$(id -g)"` makes docker run the container impersonating + yourself. + This avoids folder ownership changes since it runs as root by default. + * `-v "$(pwd):/workspace"` maps the host machine's current working directory + to the /workspace folder. + The image is configured to perform changes in this directory. + * `-v /path/to/api_definition:/workspace` maps the host machine's API + definitions folder to `/workspace/apis` folder. + +3. An advanced example: + ```shell + docker run \ + --rm \ + --quiet \ + -u "$(id -u):$(id -g)" \ + -v "$(pwd):/workspace" \ + -v /path/to/api_definition:/workspace/apis \ + gcr.io/cloud-devrel-public-resources/java-library-generation:image-tag \ + --generation-config-path=/workspace/generation_config_file \ + --library-names=apigee-connect,asset \ + --repository-path=/workspace \ + --api-definitions-path=/workspace/apis ``` + + * `--generation-config-path=/workspace/generation_config_file` set the + generation configuration to `/workspace/generation_config_file`. + * `--api-definitions-path=/workspace/apis` set the API definition path to + `/workspace/apis`. -3. Download api definition to a local directory +To debug the image, please refer to [development guide](DEVELOPMENT.md#debug-the-library-generation-container) +for more info. -## An example to generate a repository using `entry_point.py` +## An example to generate a repository using the docker image -```bash -python hermetic_build/library_generation/cli/entry_point.py generate \ - --baseline-generation-config-path=/path/to/baseline_config_file \ - --current-generation-config-path=/path/to/current_config_file \ - --repository-path=path/to/repository \ - --api-definitions-path=path/to/api_definition -``` -If you run `entry_point.py` with the example [configuration](#an-example-of-generation-configuration) +If you run the docker image with the example [configuration](#an-example-of-generation-configuration) shown above, the repository structure is: ``` $repository_path @@ -284,7 +318,70 @@ $repository_path |_versions.txt ``` -# Owlbot Java Postprocessor +# Generate release notes from API definition changes + +The script, `hermetic_build/release_note_generation/cli/generate_release_note.py` +allows you to generate a file containing release notes from API definition +changes in [googleapis](https://github.com/googleapis/googleapis) GitHub +repository. + +## Environment + +- OS: Linux +- Python (3.12.0 or above) + +## Parameters to generate a release note + +### Baseline generation configuration path (`baseline-generation-config-path`) + +Absolute or relative path to a generation configuration. +Please refer to [Configuration to generate a repository](#configuration-to-generate-a-repository) +for more information. + +Note that the `googleapis_commitish` in this configuration is used to retrieve +the first commit, exclusively, to generate the release notes. + +### Current generation configuration path (`current-generation-config-path`) + +Absolute or relative path to a generation configuration. +The release notes will be generated from commits that are related to the +libraries specified in this configuration. +Please refer to [Configuration to generate a repository](#configuration-to-generate-a-repository) +for more information. + +Note that the `googleapis_commitish` entry in this configuration is used to +retrieve the last commit, inclusively, to generate the release notes. + +### Repository path (`repository-path`), optional + +The path to which the file, `pr_description.txt` containing the release notes +will be sent. +If not specified, the file will be generated to the current working directory. + +## Generate a release notes file in a local environment + +1. Install python (>= 3.12.0). +It is recommended to create a python virtual environment through the +[official guide](https://packaging.python.org/en/latest/guides/installing-using-pip-and-virtual-environments/#create-and-use-virtual-environments). + +2. Run the following commands to install python packages + ```shell + cd /path/to/sdk-platform-java + pip install --require-hashes -r hermetic_build/common/requirements.txt + pip install hermetic_build/common + pip install --require-hashes -r hermetic_build/release_note_generation/requirements.txt + pip install hermetic_build/release_note_generation + ``` +3. Run the following commands to generate a release note + ```shell + cd /path/to/sdk-platform-java + python hermetic_build/release_note_generation/cli/generate_release_note.py generate \ + --baseline-generation-config-path=/path/to/baseline_generation_config \ + --current-generation-config-path=/path/to/current_generation_config \ + --repository-path=/path/to/send/release_note + ``` + +# OwlBot Java Postprocessor We have transferred the [implementation](https://github.com/googleapis/synthtool/tree/59fe44fde9866a26e7ee4e4450fd79f67f8cf599/docker/owlbot/java) diff --git a/hermetic_build/library_generation/DEVELOPMENT.md b/hermetic_build/library_generation/DEVELOPMENT.md deleted file mode 100644 index df7843aeaa..0000000000 --- a/hermetic_build/library_generation/DEVELOPMENT.md +++ /dev/null @@ -1,207 +0,0 @@ -> [!IMPORTANT] -> All examples assume you are inside the `hermetic_build` folder. - - -# Linting - -When contributing, ensure your changes to python code have a valid format. - -``` -python -m pip install black -black . -``` - -# Running the integration tests - -The integration tests build the docker image declared in -`.cloudbuild/library_generation/library_generation.Dockerfile`, pull GAPIC -repositories, generate the libraries and compares the results with the source -code declared in a "golden branch" of the repo. - -It requires docker and python 3.x to be installed. - -``` -python -m pip install --require-hashes -r library_generation/requirements.txt -python -m pip install library_generation -python -m unittest library_generation/tests/integration_tests.py -``` - -# Running the unit tests - -The unit tests of the hermetic build scripts are contained in several scripts, -corresponding to a specific component. -Every unit test script ends with `unit_tests.py`. -To avoid them specifying them individually, we can use the following command: - -```bash -python -m unittest discover -s library_generation/tests/ -p "*unit_tests.py" -``` - -> [!NOTE] -> The output of this command may look erratic during the first 30 seconds. -> This is normal. After the tests are done, an "OK" message should be shown. - -# Running the scripts in your local environment - -Although the scripts are designed to be run in a Docker container, you can also -run them directly. -This section explains how to run the entrypoint script -(`library_generation/cli/entry_point.py`). - -## Assumptions made by the scripts -### The Hermetic Build's well-known folder -Located in `${HOME}/.library_generation`, this folder is assumed by the scripts -to contain certain tools. - -Developers must make sure this folder is properly configured before running the -scripts locally. -Note that this relies on the `HOME` en var which is always defined as per -[POSIX env var definition](https://pubs.opengroup.org/onlinepubs/9699919799/basedefs/V1_chap08.html). - -#### Put the gapic-generator-java jar in its well-known location - -Run `cd sdk-platform-java && mvn install -DskipTests -Dclirr.skip --Dcheckstyle.skip`. -This will generate a jar located in -`~/.m2/repository/com/google/api/gapic-generator-java/{version}/gapic-generator-java-{version}.jar` - -Then `mv` the jar into the well-known location of the jar. -The generation scripts will assume the jar is there. - -```shell -mv /path/to/jar "${HOME}/.library_generation/gapic-generator-java.jar" -``` - -#### Put the java formatter jar in its well-known location - -Download google-java-format-{version}-all-deps.jar from [Maven Central](https://central.sonatype.com/artifact/com.google.googlejavaformat/google-java-format) -or [GitHub releases](https://github.com/google/google-java-format/releases). -Then `mv` the jar into the well-known location of the jar. -The generation scripts will assume the jar is there. - -```shell -mv /path/to/jar "${HOME}/.library_generation/google-java-format.jar" -``` - -## Installing prerequisites - -In order to run the generation scripts directly, there are a few tools we -need to install beforehand. - -### Install the owl-bot CLI - -Requires node.js to be installed. -Check this [installation guide](https://github.com/nvm-sh/nvm?tab=readme-ov-file#install--update-script) -for NVM, Node.js's version manager. - -After you install it, you can install the owl-bot CLI with the following -commands: -```bash -git clone https://github.com/googleapis/repo-automation-bots -cd repo-automation-bots/packages/owl-bot -npm i && npm run compile && npm link -owl-bot copy-code --version -``` - -The key step is `npm link`, which will make the command available in you current -shell session. - - -## Running the script -The entrypoint script (`library_generation/cli/entry_point.py`) allows you to -generate a GAPIC repository with a given api definition (proto, service yaml). - -### Download the api definition -For example, googleapis -``` -git clone https://github.com/googleapis/googleapis -export api_definitions_path="$(pwd)/googleapis" -``` - -### Download the repo -For example, google-cloud-java -``` -git clone https://github.com/googleapis/google-cloud-java -export path_to_repo="$(pwd)/google-cloud-java" -``` - -### Install the scripts -``` -python -m pip install . -``` - -### Run the script -``` -python library_generation/cli/entry_point.py generate \ - --repository-path="${path_to_repo}" \ - --api-definitions-path="${api_definitions_path}" -``` - - -# Running the scripts using the docker container image -This is convenient in order to avoid installing the dependencies manually. - -> [!IMPORTANT] -> From now, the examples assume you are in the root of your sdk-platform-java -> folder. - -## Build the docker image -```bash -docker build --file .cloudbuild/library_generation/library_generation.Dockerfile --iidfile image-id . -``` - -This will create an `image-id` file at the root of the repo with the hash ID of -the image. - -## Run the docker image -The docker image will perform changes on its internal `/workspace` folder, -to which you need to map a folder on your host machine (i.e. map your downloaded -repo to this folder). - -To run the docker container on the google-cloud-java repo, you must run: -```bash -docker run \ - -u "$(id -u)":"$(id -g)" \ - -v /path/to/google-cloud-java:/workspace \ - -v /path/to/api-definition:/workspace/apis \ - $(cat image-id) \ - --api-definitions-path=/workspace/apis -``` - - * `-u "$(id -u)":"$(id -g)"` makes docker run the container impersonating - yourself. This avoids folder ownership changes since it runs as root by - default. - * `-v /path/to/google-cloud-java:/workspace` maps the host machine's - google-cloud-java folder to the /workspace folder. - The image is configured to perform changes in this directory. - * `-v /path/to/api-definition:/workspace/apis` maps the host machine's - api-definition folder to /workspace/apis folder. - * `$(cat image-id)` obtains the image ID created in the build step. - * `--api-definitions-path=/workspace/apis` set the API definition path to - `/workspace/apis`. - -## Debug the created containers -If you are working on changing the way the containers are created, you may want -to inspect the containers to check the setup. -It would be convenient in such case to have a text editor/viewer available. -You can achieve this by modifying the Dockerfile as follows: - -```docker -# install OS tools -RUN apt-get update && apt-get install -y \ - unzip openjdk-17-jdk rsync maven jq less vim \ - && apt-get clean -``` - -We add `less` and `vim` as text tools for further inspection. - -You can also run a shell in a new container by running: - -```bash -docker run \ - --rm -it \ - -u $(id -u):$(id -g) \ - -v /path/to/google-cloud-java:/workspace \ - --entrypoint="bash" \ - $(cat image-id) -``` diff --git a/hermetic_build/release_note_generation/cli/generate_release_note.py b/hermetic_build/release_note_generation/cli/generate_release_note.py index 294f4baf50..a54e228f62 100644 --- a/hermetic_build/release_note_generation/cli/generate_release_note.py +++ b/hermetic_build/release_note_generation/cli/generate_release_note.py @@ -54,8 +54,9 @@ def main(ctx): default=".", show_default=True, help=""" - The repository path to which the generated files will be sent. - If not specified, the repository will be generated to the current working + The path where the file `pr_description.txt`, which contains the release + notes, will be sent. + If not specified, the file will be generated to the current working directory. """, )