Don't download the whole Boost.CI Repo in functions.star

[Flamefire]

• Only download the required file(s).
• Move the command commands of all unix pipelines into a function to implement this only once

Test build at https://drone.cpp.al/boostorg/boost-ci/282

@sdarwin Please double check

BTW: Do you know of any way to conditionally include the functions.star from either the repo or local file so that we include the latest version from the PR instead if running on Boost.CI

[codecov]

Codecov Report

Merging #185 (a6ccc6c) into master (4657d39) will not change coverage.
The diff coverage is n/a.

❗ Current head a6ccc6c differs from pull request most recent head 0ca328e. Consider uploading reports for the commit 0ca328e to get more accurate results

Additional details and impacted files

@@            Coverage Diff            @@
##            master      #185   +/-   ##
=========================================
  Coverage   100.00%   100.00%           
=========================================
  Files            2         2           
  Lines           20        20           
  Branches        10        10           
=========================================
  Hits            20        20           

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 4657d39...0ca328e. Read the comment docs.

[sdarwin]

Please change wget to curl -s -S --retry 10 --create-dirs -L -o

conditionally include functions.star

this is the starlark language, it's based on python, and very similar to python. The last line of .drone.star runs that load command. Maybe it could check if the file functions.star exists, and then take a different action. Load it from a different place.
However, starlark has built-in security, and doesn't allow all files. It could very well get blocked. In the current situation, a copy of boost-ci has been placed on the drone server in a particular location, and that is where it is sourced from.

Another option, discussed in https://github.com/CPPAlliance/drone-ci/blob/master/docs/drone-ci.md , is to copy all the contents of functions.star directly into .drone.star.

It may be this pull request is not backwards compatible, since drone has been rolled out already to many repositories. Notice the load command at the end of .drone.star is loading particular functions. It's not loading "unix_common" though, so "unix_common" won't be available in .drone.star. it would require 50 or 100 pull requests to all the boost libraries which is probably not worth it.

On a different topic, if you are also reformatting .drone.star, take a look at https://github.com/boostorg/url/blob/develop/.drone.star

That could be the new starting template.

Include:

• 1 windows job
• 1 mac osx job
• 2 arm architecture jobs
• 1 or 2 s390x jobs
• 1 or 2 freebsd jobs
• updated versions of valgrind, asan, etc.

When there is so much variation and variety in the types of jobs in .drone.star, it might become more difficult to neatly and cleanly extrapolate or abstract that in functions. Just including all the parameters as-is, the way it's done now, seems ok. But if you are proposing new abstractions, we can consider them.

For now, Linux amd64 and arm64 are autoscaled, so there is no limit. The others such as s390x are dedicated machines. That is the reason for having 1 windows job in the template, for now, and not 5 or 10 windows jobs included in the template. Eventually we might be able to get autoscaling on windows.

[Flamefire]

Please change wget to curl -s -S --retry 10 --create-dirs -L -o

Done. Test build at https://drone.cpp.al/boostorg/boost-ci/289

Maybe it could check if the file functions.star exists, and then take a different action. Load it from a different place.

That is the idea. I just don't know how to do it as what is available in the language depends on the implementation, i.e. here it would be DroneCI specific and I wasn't able to find documentation on it. Do you have any?

In the current situation, a copy of boost-ci has been placed on the drone server in a particular location, and that is where it is sourced from.

Oh what? I thought it gets it from this repo directly. Again: Any docu on how the load behaves for Drone?

It's not loading "unix_common" though, so "unix_common" won't be available in .drone.star.

That shouldn't be an issue. It is an implementation detail only. So the load makes the name linux_cxx available but what that does and what it sees is up to that command.
At least if the Drone Starlark dialekt is close enough to Python/Bazel-Starlark.

On a different topic, if you are also reformatting .drone.star, take a look at https://github.com/boostorg/url/blob/develop/.drone.star

That could be the new starting template.

I wanted to move away the duplication like I did in the GHA template: E.g. no need to specify compiler, B2_TOOLSET and packages as only 1 is required. See #184 for discussion on that, just a teaser:
I'd like: job(compiler='gcc-12', cxxstd='11,17,20', os='ubuntu-22.04'), and have a working function for that ;-)

[sdarwin]

It is an implementation detail only.

Only an implementation detail. If you say so. :-)

How about this, when you're ready, check the new file into the master branch as functions2.star or functions.test.star. Alternatively, instead of checking it in, point me to a copy of the file, and I will manually copy it to the drone server. Either way, I will need to copy it to the drone server.

Then, after it's online, adjust a .drone.star file in a test branch to use it, as follows:

# from https://github.com/boostorg/boost-ci
load("@boost_ci//ci/drone/:functions.test.star", "linux_cxx","windows_cxx","osx_cxx","freebsd_cxx")

[sdarwin]

Any documentation about starlark?

Here are docs: https://docs.drone.io/pipeline/scripting/starlark/

This plugin is used: https://github.com/drone/drone-convert-starlark

They write: "DEPRECATED: native Starlark support was added to Drone core. This extension is no longer required and has been deprecated." However, the native support doesn't have the "load" command. Thus, server-side includes are missing. So we're still using the plugin.

[Flamefire]

A build will not be able to load() files in the same repo as the pipeline

So loading the updated file from here doesn't work :(

How about this, when you're ready, check the new file into the master branch as functions2.star or functions.test.star. > Alternatively, instead of checking it in, point me to a copy of the file, and I will manually copy it to the drone server. Either way, I will need to copy it to the drone server.

I'm done. Please take the file from this PR and give it a try :)

[Flamefire]

Thanks for fixing the C&P issue. On the compare I think bash allows == and = or is there a difference?

[sdarwin]

functions.test.star is on the server.

I think bash allows == and = or is there a difference?

I also thought bash allows either == or = so it was strange.

the error:

/bin/sh: 55: [: drone: unexpected operator

could be the bourne shell.

[Flamefire]

Looks good as far as I can tell: https://drone.cpp.al/boostorg/boost-ci/298

could be the bourne shell.

Ah yes, bash allows that but the initial script on Drone is run in plain shell

[Flamefire]

Oh wait!

[Flamefire]

I shouldn't do that late...

@sdarwin Can you install the latest version? I fixed the conditions and made the easier to read using plain if instead of checking for the inverse

[sdarwin]

The file is on the server as functions.test2.star Since work is still progressing on drone changes overall, there's no rush to merge, we can wait until things have settled down. it is looking pretty good.

[Flamefire]

Looks good so far, using it in Boost.Locale successfully.

[Flamefire]

Is this deployed yet (as the original functions.star)?

[sdarwin]

yes