Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

test(cargo): make it possible to test cargo packages #17535

Merged
merged 4 commits into from
Nov 23, 2023
Merged

test(cargo): make it possible to test cargo packages #17535

merged 4 commits into from
Nov 23, 2023

Conversation

hituzi-no-sippo
Copy link
Contributor

No description provided.

@suzuki-shunsuke
Copy link
Member

Thank you for your contribution!
Nice catch!

@suzuki-shunsuke
Copy link
Member

suzuki-shunsuke commented Nov 20, 2023

📝 Rust and Cargo are required.

https://doc.rust-lang.org/cargo/getting-started/installation.html

curl https://sh.rustup.rs -sSf | sh

https://rustup.rs/

curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh

@suzuki-shunsuke
Copy link
Member

Please change the status to Ready for review when it gets ready.

@hituzi-no-sippo
Copy link
Contributor Author

There is a problem.

Testing the same package (cmdx test <pkg>) more than once results in failure.
The first test was successful.

The reason for failure after the second time is because the executable file aren't found.

ERRO[0000] check file_src is correct                     aqua_version=2.18.0 env=linux/amd64 error="check file_src is correct: exe_path isn't found: stat /root/aquaproj-aqua/pkgs/cargo/crates.io/shellharden/4.3.0/bin/shellharden: no such file or directory" exe_path=/root/a
quaproj-aqua/pkgs/cargo/crates.io/shellharden/4.3.0/bin/shellharden file_name=shellharden package_name=crates.io/shellharden package_version=4.3.0 program=aqua registry=standard
ERRO[0000] executable files aren't found
Files in the unarchived package:
.crates.toml
.crates2.json
bin/shellharden.exe
   aqua_version=2.18.0 env=linux/amd64 package_name=crates.io/shellharden package_version=4.3.0 program=aqua registry=standard
ERRO[0000] install the package                           aqua_version=2.18.0 env=linux/amd64 error="check file_src is correct" package_name=crates.io/shellharden package_version=4.3.0 program=aqua registry=standard
FATA[0000] aqua failed                                   aqua_version=2.18.0 env=linux/amd64 error="it failed to install some packages" program=aqua
[ERROR] Build failed linux/amd64
root@c0094dfb8255:/workspace# ls /root/aquaproj-aqua/pkgs/cargo/crates.io/shellharden/4.3.0/bin
shellharden.exe

The cause of the executable file aren't found is because the executable file was renamed when OS was Windows.

INFO[0000] rename a file                                 aqua_version=2.18.0 env=windows/amd64 file_name=shellharden new=/root/aquaproj-aqua/pkgs/cargo/crates.io/shellharden/4.3.0/bin/shellharden.exe old=/root/aquaproj-aqua/pkgs/cargo/crates.io/shellharden/4.3.0/bin/shellharden package_name=crates.io/shellharden package_version=4.3.0 program=aqua registry=standard

After having run aqua remove <pkg>, testing will be successful.
If you know of a solution other than running aqua remove, could you tell me the solution?

Reproduce the problem

In the case of using cmdx t <pkg> command

Run commands

  1. cmdx t "crates.io/shellharden"
    Test was successful.
  2. cmdx t "crates.io/shellharden"
    Test was failed.
  3. docker exec aqua-registry env aqua remove crates.io/shellharden
    Remove the package.
  4. cmdx t "crates.io/shellharden"
    Test was successful
  5. cmdx t "crates.io/shellharden"
    Test was failed
console log
$ cmdx t "crates.io/shellharden"
+ set -euxo pipefail
if [ "false" = true ]; then
  cmdx rm
fi
bash scripts/start.sh
bash scripts/test.sh "crates.io/shellharden"

+ '[' false = true ']'
+ bash scripts/start.sh
[INFO] Checking if the container aqua-registry exists
[INFO] Creaing a container aqua-registry
c0094dfb825533f7d696a15f52058a7633ee5be24ad0c99d1857666fa70e09f3
+ bash scripts/test.sh crates.io/shellharden
INFO[0000] download and unarchive the package            aqua_version=2.18.0 env=linux/amd64 package_name=aqua-proxy package_version=v1.2.4 program=aqua registry=
INFO[0000] create a symbolic link                        aqua_version=2.18.0 command=aqua-proxy env=linux/amd64 package_name=aqua-proxy package_version=v1.2.4 program=aqua registry=
INFO[0000] create a symbolic link                        aqua_version=2.18.0 command=shellharden env=linux/amd64 program=aqua
INFO[0000] Installing a crate                            aqua_version=2.18.0 env=linux/amd64 package_name=crates.io/shellharden package_version=4.3.0 program=aqua registry=standard
    Updating crates.io index
 Downloading crates ...
  Downloaded shellharden v4.3.0
  Installing shellharden v4.3.0
    Updating crates.io index
   Compiling shellharden v4.3.0
    Finished release [optimized] target(s) in 2.53s
  Installing /root/aquaproj-aqua/pkgs/cargo/crates.io/shellharden/4.3.0/bin/shellharden
   Installed package `shellharden v4.3.0` (executable `shellharden`)
warning: be sure to add `/root/aquaproj-aqua/pkgs/cargo/crates.io/shellharden/4.3.0/bin` to your PATH to be able to run the installed binaries
INFO[0000] download and unarchive the package            aqua_version=2.18.0 env=linux/arm64 package_name=aqua-proxy package_version=v1.2.4 program=aqua registry=
INFO[0000] download and unarchive the package            aqua_version=2.18.0 env=darwin/amd64 package_name=aqua-proxy package_version=v1.2.4 program=aqua registry=
INFO[0000] download and unarchive the package            aqua_version=2.18.0 env=darwin/arm64 package_name=aqua-proxy package_version=v1.2.4 program=aqua registry=
INFO[0000] create a proxy file                           aqua_version=2.18.0 env=windows/amd64 program=aqua proxy_path=/root/aquaproj-aqua/bin/shellharden
INFO[0000] create a proxy file                           aqua_version=2.18.0 env=windows/amd64 program=aqua proxy_path=/root/aquaproj-aqua/bat/shellharden.bat
INFO[0000] rename a file                                 aqua_version=2.18.0 env=windows/amd64 file_name=shellharden new=/root/aquaproj-aqua/pkgs/cargo/crates.io/shellharden/4.3.0/bin/shellharden.exe old=/root/aquaproj-aqua/pkgs/cargo/crates.io/shellharden/4.3.0/bin/shellh
arden package_name=crates.io/shellharden package_version=4.3.0 program=aqua registry=standard
$ cmdx t "crates.io/shellharden"
+ set -euxo pipefail
if [ "false" = true ]; then
  cmdx rm
fi
bash scripts/start.sh
bash scripts/test.sh "crates.io/shellharden"

+ '[' false = true ']'
+ bash scripts/start.sh
[INFO] Checking if the container aqua-registry exists
[INFO] Checking if the container aqua-registry is running
[INFO] Dockerfile isn't updated
+ bash scripts/test.sh crates.io/shellharden
ERRO[0000] check file_src is correct                     aqua_version=2.18.0 env=linux/amd64 error="check file_src is correct: exe_path isn't found: stat /root/aquaproj-aqua/pkgs/cargo/crates.io/shellharden/4.3.0/bin/shellharden: no such file or directory" exe_path=/root/a
quaproj-aqua/pkgs/cargo/crates.io/shellharden/4.3.0/bin/shellharden file_name=shellharden package_name=crates.io/shellharden package_version=4.3.0 program=aqua registry=standard
ERRO[0000] executable files aren't found
Files in the unarchived package:
.crates.toml
.crates2.json
bin/shellharden.exe
   aqua_version=2.18.0 env=linux/amd64 package_name=crates.io/shellharden package_version=4.3.0 program=aqua registry=standard
ERRO[0000] install the package                           aqua_version=2.18.0 env=linux/amd64 error="check file_src is correct" package_name=crates.io/shellharden package_version=4.3.0 program=aqua registry=standard
FATA[0000] aqua failed                                   aqua_version=2.18.0 env=linux/amd64 error="it failed to install some packages" program=aqua
[ERROR] Build failed linux/amd64
root@c0094dfb8255:/workspace# ls /root/aquaproj-aqua/pkgs/cargo/crates.io/shellharden/4.3.0/bin
shellharden.exe
root@c0094dfb8255:/workspace#
exit
exit status 1
$ docker exec aqua-registry env aqua remove crates.io/shellharden
INFO[0000] removing a package                            aqua_version=2.18.0 env=linux/amd64 package_name=crates.io/shellharden program=aqua
$ cmdx t "crates.io/shellharden"
+ set -euxo pipefail
if [ "false" = true ]; then
  cmdx rm
fi
bash scripts/start.sh
bash scripts/test.sh "crates.io/shellharden"

+ '[' false = true ']'
+ bash scripts/start.sh
[INFO] Checking if the container aqua-registry exists
[INFO] Checking if the container aqua-registry is running
[INFO] Dockerfile isn't updated
+ bash scripts/test.sh crates.io/shellharden
INFO[0000] Installing a crate                            aqua_version=2.18.0 env=linux/amd64 package_name=crates.io/shellharden package_version=4.3.0 program=aqua registry=standard
    Updating crates.io index
  Installing shellharden v4.3.0
    Updating crates.io index
   Compiling shellharden v4.3.0
    Finished release [optimized] target(s) in 0.84s
  Installing /root/aquaproj-aqua/pkgs/cargo/crates.io/shellharden/4.3.0/bin/shellharden
   Installed package `shellharden v4.3.0` (executable `shellharden`)
warning: be sure to add `/root/aquaproj-aqua/pkgs/cargo/crates.io/shellharden/4.3.0/bin` to your PATH to be able to run the installed binaries
INFO[0000] create a proxy file                           aqua_version=2.18.0 env=windows/amd64 program=aqua proxy_path=/root/aquaproj-aqua/bin/shellharden
INFO[0000] rename a file                                 aqua_version=2.18.0 env=windows/amd64 file_name=shellharden new=/root/aquaproj-aqua/pkgs/cargo/crates.io/shellharden/4.3.0/bin/shellharden.exe old=/root/aquaproj-aqua/pkgs/cargo/crates.io/shellharden/4.3.0/bin/shellh
arden package_name=crates.io/shellharden package_version=4.3.0 program=aqua registry=standard
$ cmdx t "crates.io/shellharden"
+ set -euxo pipefail
if [ "false" = true ]; then
  cmdx rm
fi
bash scripts/start.sh
bash scripts/test.sh "crates.io/shellharden"

+ '[' false = true ']'
+ bash scripts/start.sh
[INFO] Checking if the container aqua-registry exists
[INFO] Checking if the container aqua-registry is running
[INFO] Dockerfile isn't updated
+ bash scripts/test.sh crates.io/shellharden
ERRO[0000] check file_src is correct                     aqua_version=2.18.0 env=linux/amd64 error="check file_src is correct: exe_path isn't found: stat /root/aquaproj-aqua/pkgs/cargo/crates.io/shellharden/4.3.0/bin/shellharden: no such file or directory" exe_path=/root/a
quaproj-aqua/pkgs/cargo/crates.io/shellharden/4.3.0/bin/shellharden file_name=shellharden package_name=crates.io/shellharden package_version=4.3.0 program=aqua registry=standard
ERRO[0000] executable files aren't found
Files in the unarchived package:
.crates.toml
.crates2.json
bin/shellharden.exe
   aqua_version=2.18.0 env=linux/amd64 package_name=crates.io/shellharden package_version=4.3.0 program=aqua registry=standard
ERRO[0000] install the package                           aqua_version=2.18.0 env=linux/amd64 error="check file_src is correct" package_name=crates.io/shellharden package_version=4.3.0 program=aqua registry=standard
FATA[0000] aqua failed                                   aqua_version=2.18.0 env=linux/amd64 error="it failed to install some packages" program=aqua
[ERROR] Build failed linux/amd64
root@c0094dfb8255:/workspace#
exit
exit status 1

In the case of not using cmdx t <pkg> command

  1. Setup
    1. Run the Docker container
    2. copy pkg and registry files to container
  2. docker exec aqua-registry env AQUA_GOOS="linux" AQUA_GOARCH="amd64" aqua i
    aqua was successful.
    Install the package.
  3. docker exec aqua-registry env AQUA_GOOS="windows" AQUA_GOARCH="amd64" aqua i
    aqua was successful.
    Rename the executable file.
  4. docker exec aqua-registry env AQUA_GOOS="linux" AQUA_GOARCH="amd64" aqua i
    aqua was failed.
  5. docker exec aqua-registry env aqua remove crates.io/shellharden
    Remove the package.
  6. docker exec aqua-registry env AQUA_GOOS="linux" AQUA_GOARCH="amd64" aqua i
    aqua was successful.
    Install the package.
  7. docker exec aqua-registry env AQUA_GOOS="linux" AQUA_GOARCH="amd64" aqua i
    aqua was successful.
console log
$ ###############################################
$ # Setup start
$ ###############################################
$ cmdx remove
+ set -eux
docker stop -t 1 aqua-registry
docker rm aqua-registry

+ docker stop -t 1 aqua-registry
aqua-registry
+ docker rm aqua-registry
aqua-registry
$ bash scripts/start.sh
[INFO] Checking if the container aqua-registry exists
[INFO] Creaing a container aqua-registry
67c6953bd988f068adf9f35f3c3ee79bd77d9390c29ca4682937e32ab0f9c8a6
$ pkg='crates.io/shellharden'
$ docker cp "pkgs/$pkg/pkg.yaml" aqua-registry:/workspace/pkg.yaml
$ docker cp "pkgs/$pkg/registry.yaml" aqua-registry:/workspace/registry.yaml
$ ###############################################
$ # Setup End
$ ###############################################
$ docker exec aqua-registry env AQUA_GOOS="linux" AQUA_GOARCH="amd64" aqua i
INFO[0000] download and unarchive the package            aqua_version=2.18.0 env=linux/amd64 package_name=aqua-proxy package_version=v1.2.4 program=aqua registry=
INFO[0000] create a symbolic link                        aqua_version=2.18.0 command=aqua-proxy env=linux/amd64 package_name=aqua-proxy package_version=v1.2.4 program=aqua registry=
INFO[0000] create a symbolic link                        aqua_version=2.18.0 command=shellharden env=linux/amd64 program=aqua
INFO[0000] Installing a crate                            aqua_version=2.18.0 env=linux/amd64 package_name=crates.io/shellharden package_version=4.3.0 program=aqua registry=standard
    Updating crates.io index
 Downloading crates ...
  Downloaded shellharden v4.3.0
  Installing shellharden v4.3.0
    Updating crates.io index
   Compiling shellharden v4.3.0
    Finished release [optimized] target(s) in 3.05s
  Installing /root/aquaproj-aqua/pkgs/cargo/crates.io/shellharden/4.3.0/bin/shellharden
   Installed package `shellharden v4.3.0` (executable `shellharden`)
warning: be sure to add `/root/aquaproj-aqua/pkgs/cargo/crates.io/shellharden/4.3.0/bin` to your PATH to be able to run the installed binaries
$ docker exec aqua-registry env AQUA_GOOS="windows" AQUA_GOARCH="amd64" aqua i
INFO[0000] create a proxy file                           aqua_version=2.18.0 env=windows/amd64 program=aqua proxy_path=/root/aquaproj-aqua/bin/shellharden
INFO[0000] create a proxy file                           aqua_version=2.18.0 env=windows/amd64 program=aqua proxy_path=/root/aquaproj-aqua/bat/shellharden.bat
INFO[0000] rename a file                                 aqua_version=2.18.0 env=windows/amd64 file_name=shellharden new=/root/aquaproj-aqua/pkgs/cargo/crates.io/shellharden/4.3.0/bin/shellharden.exe old=/root/aquaproj-aqua/pkgs/cargo/crates.io/shellharden/4.3.0/bin/shellh
arden package_name=crates.io/shellharden package_version=4.3.0 program=aqua registry=standard
$ docker exec aqua-registry env AQUA_GOOS="linux" AQUA_GOARCH="amd64" aqua i
ERRO[0000] check file_src is correct                     aqua_version=2.18.0 env=linux/amd64 error="check file_src is correct: exe_path isn't found: stat /root/aquaproj-aqua/pkgs/cargo/crates.io/shellharden/4.3.0/bin/shellharden: no such file or directory" exe_path=/root/a
quaproj-aqua/pkgs/cargo/crates.io/shellharden/4.3.0/bin/shellharden file_name=shellharden package_name=crates.io/shellharden package_version=4.3.0 program=aqua registry=standard
ERRO[0000] executable files aren't found
Files in the unarchived package:
.crates.toml
.crates2.json
bin/shellharden.exe
   aqua_version=2.18.0 env=linux/amd64 package_name=crates.io/shellharden package_version=4.3.0 program=aqua registry=standard
ERRO[0000] install the package                           aqua_version=2.18.0 env=linux/amd64 error="check file_src is correct" package_name=crates.io/shellharden package_version=4.3.0 program=aqua registry=standard
FATA[0000] aqua failed                                   aqua_version=2.18.0 env=linux/amd64 error="it failed to install some packages" program=aqua
$ docker exec aqua-registry env aqua remove crates.io/shellharden
INFO[0000] removing a package                            aqua_version=2.18.0 env=linux/amd64 package_name=crates.io/shellharden program=aqua
$ docker exec aqua-registry env AQUA_GOOS="linux" AQUA_GOARCH="amd64" aqua i
INFO[0000] Installing a crate                            aqua_version=2.18.0 env=linux/amd64 package_name=crates.io/shellharden package_version=4.3.0 program=aqua registry=standard
    Updating crates.io index
  Installing shellharden v4.3.0
    Updating crates.io index
   Compiling shellharden v4.3.0
    Finished release [optimized] target(s) in 0.77s
  Installing /root/aquaproj-aqua/pkgs/cargo/crates.io/shellharden/4.3.0/bin/shellharden
   Installed package `shellharden v4.3.0` (executable `shellharden`)
warning: be sure to add `/root/aquaproj-aqua/pkgs/cargo/crates.io/shellharden/4.3.0/bin` to your PATH to be able to run the installed binaries
$ docker exec aqua-registry env AQUA_GOOS="linux" AQUA_GOARCH="amd64" aqua i
$ 

@suzuki-shunsuke
Copy link
Member

suzuki-shunsuke commented Nov 21, 2023

Yeah, this is a known issue.
Sorry for inconvenience.

The root cause is cmdx tasks share the same container aqua-registry with windows and others.
In these tasks aqua emulates other platforms by setting AQUA_GOOS and AQUA_GOARCH.
On Windows, aqua renames executable files if they don't have the suffix .exe because Windows require the suffix.
Tasks rename executable files so aqua can't find executable files on other than Windows.

In case of github_release packages, basically the installation path is different by platforms so there is no problem.
But in case of cargo packages, the installation path is same, so the issue occurs.

This is a known issue, but I left it for a while because there is no problem for almost all packages.
But of course I recognize we should fix the issue.

After having run aqua remove , testing will be successful.
If you know of a solution other than running aqua remove, could you tell me the solution?

Currently, removing the container is appropriate workaround.
The other way is to connect the container and remove the package.

cmdx con
aqua rm "<package name>"

To resolve the root cause, maybe we should create containers per platform.

@suzuki-shunsuke
Copy link
Member

@suzuki-shunsuke suzuki-shunsuke marked this pull request as ready for review November 23, 2023 01:39
@suzuki-shunsuke suzuki-shunsuke merged commit 1ad57cb into aquaproj:main Nov 23, 2023
17 checks passed
@hituzi-no-sippo hituzi-no-sippo deleted the test/add-support-for-cargo-packages branch November 23, 2023 03:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants