fix: filter out Artifactory version & sort versions #402
Conversation
|
We running into the same issue, would it be possible to merge this? |
| @@ -73,8 +73,20 @@ remote_versions="$(curlw -sSf "${TFENV_REMOTE}/terraform/")" \ | |||
|
|
|||
| #log 'debug' "Remote versions available: ${remote_versions}"; # Even in debug mode this is too verbose | |||
|
|
|||
| # Remove Artifactory version | |||
| remote_versions="$(grep -v -E "Artifactory/[0-9]+\.[0-9]+\.[0-9]+" <<< "${remote_versions}")" | |||
There was a problem hiding this comment.
Instead of this, could we update the command to be sed -En 's#.*[a-z]*/([0-9]+\.[0-9]+\.[0-9]+(-(rc|beta|alpha|oci)-?[0-9]*)?).*#\1#p'
Then we wouldn't need line 80 either, or the calls to uniq, and it is more generic.
There was a problem hiding this comment.
I tested it, this doesn't work. It will only capture the Artifactory version.
You can check by yourself with the Artifactory sample page that I shared on #400 .
There was a problem hiding this comment.
Ah my bad! I guess I got too caught up in the regex rather than the actual goal.
sed -En 's#.*[a-z]*_([0-9]+\.[0-9]+\.[0-9]+(-(rc|beta|alpha|oci)-?[0-9]*)?).*#\1#p' should work.
There was a problem hiding this comment.
It doesn't capture anything now. I'm not sure to understand what you want to capture, but to me, the current PR implementation is rather OK:
- I exclude the particular case where we have the
Artifactory/that prefixes the version. - I grep all the versions that I can find (just like before).
- I process the behaviors depending on other env vars.
There was a problem hiding this comment.
I'm not sure why that sed command works on my machine with both BSD and GNU sed. If you replace -En with -rn, does it work?
I'm trying to capture only versions that are relevant to Terraform without hard coding the string 'terraform' (although doing this is probably fine atm as OpenTofu hasn't released anything yet) and without hard coding the string 'Artifactory' as if other mirrors are used, I could foresee a long chain of if/elses being created.
There was a problem hiding this comment.
It's not a problem of BSD or GNU sed.
If you take this Artifactory page as input:
Artifactory HTML page
<!DOCTYPE html>
<html>
<head><meta name="robots" content="noindex" />
<title>Index of hashicorp/terraform</title>
</head>
<body>
<h1>Index of hashicorp/terraform</h1>
<pre>Name Last modified Size</pre><hr/>
<pre><a href="../">../</a>
<a href="0.13.0/">0.13.0/</a> 15-Jun-2023 15:19 -
<a href="0.13.1/">0.13.1/</a> 15-Jun-2023 15:18 -
<a href="0.13.2/">0.13.2/</a> 15-Jun-2023 15:17 -
<a href="0.13.3/">0.13.3/</a> 15-Jun-2023 15:16 -
<a href="0.13.4/">0.13.4/</a> 15-Jun-2023 15:15 -
<a href="0.13.5/">0.13.5/</a> 15-Jun-2023 15:14 -
<a href="0.13.6/">0.13.6/</a> 15-Jun-2023 15:12 -
<a href="0.13.7/">0.13.7/</a> 15-Jun-2023 15:11 -
<a href="0.14.0/">0.14.0/</a> 15-Jun-2023 15:10 -
<a href="0.14.1/">0.14.1/</a> 15-Jun-2023 15:08 -
<a href="0.14.10/">0.14.10/</a> 15-Jun-2023 14:57 -
<a href="0.14.11/">0.14.11/</a> 15-Jun-2023 14:56 -
<a href="0.14.2/">0.14.2/</a> 15-Jun-2023 15:07 -
<a href="0.14.3/">0.14.3/</a> 15-Jun-2023 15:06 -
<a href="0.14.4/">0.14.4/</a> 15-Jun-2023 15:05 -
<a href="0.14.5/">0.14.5/</a> 15-Jun-2023 15:04 -
<a href="0.14.6/">0.14.6/</a> 15-Jun-2023 15:03 -
<a href="0.14.7/">0.14.7/</a> 15-Jun-2023 15:01 -
<a href="0.14.8/">0.14.8/</a> 15-Jun-2023 15:00 -
<a href="0.14.9/">0.14.9/</a> 15-Jun-2023 14:59 -
<a href="0.15.0/">0.15.0/</a> 15-Jun-2023 14:55 -
<a href="0.15.1/">0.15.1/</a> 15-Jun-2023 14:54 -
<a href="0.15.2/">0.15.2/</a> 15-Jun-2023 14:53 -
<a href="0.15.3/">0.15.3/</a> 15-Jun-2023 14:52 -
<a href="0.15.4/">0.15.4/</a> 15-Jun-2023 14:50 -
<a href="0.15.5/">0.15.5/</a> 15-Jun-2023 14:49 -
<a href="0.5.3/">0.5.3/</a> 07-Jun-2023 16:34 -
<a href="1.0.0/">1.0.0/</a> 15-Jun-2023 14:48 -
<a href="1.0.1/">1.0.1/</a> 15-Jun-2023 14:47 -
<a href="1.0.10/">1.0.10/</a> 15-Jun-2023 14:40 -
<a href="1.0.11/">1.0.11/</a> 15-Jun-2023 13:49 -
<a href="1.0.2/">1.0.2/</a> 15-Jun-2023 13:50 -
<a href="1.0.3/">1.0.3/</a> 15-Jun-2023 14:46 -
<a href="1.0.4/">1.0.4/</a> 15-Jun-2023 14:45 -
<a href="1.0.5/">1.0.5/</a> 15-Jun-2023 14:44 -
<a href="1.0.6/">1.0.6/</a> 15-Jun-2023 14:43 -
<a href="1.0.7/">1.0.7/</a> 15-Jun-2023 14:42 -
<a href="1.0.8/">1.0.8/</a> 15-Jun-2023 14:41 -
<a href="1.0.9/">1.0.9/</a> 15-Jun-2023 14:40 -
<a href="1.1.0/">1.1.0/</a> 15-Jun-2023 14:38 -
<a href="1.1.1/">1.1.1/</a> 15-Jun-2023 14:37 -
<a href="1.1.2/">1.1.2/</a> 15-Jun-2023 14:37 -
<a href="1.1.3/">1.1.3/</a> 15-Jun-2023 14:36 -
<a href="1.1.4/">1.1.4/</a> 15-Jun-2023 14:35 -
<a href="1.1.5/">1.1.5/</a> 15-Jun-2023 14:35 -
<a href="1.1.6/">1.1.6/</a> 15-Jun-2023 14:34 -
<a href="1.1.7/">1.1.7/</a> 24-Mar-2022 17:02 -
<a href="1.1.8/">1.1.8/</a> 22-Apr-2022 15:25 -
<a href="1.1.9/">1.1.9/</a> 22-Apr-2022 15:26 -
<a href="1.2.0/">1.2.0/</a> 19-May-2022 11:20 -
<a href="1.2.1/">1.2.1/</a> 24-May-2022 13:36 -
<a href="1.2.2/">1.2.2/</a> 13-Jun-2022 08:19 -
<a href="1.2.3/">1.2.3/</a> 17-Jun-2022 15:57 -
<a href="1.2.4/">1.2.4/</a> 29-Jun-2022 23:00 -
<a href="1.2.5/">1.2.5/</a> 13-Jul-2022 23:58 -
<a href="1.2.6/">1.2.6/</a> 28-Jul-2022 18:30 -
<a href="1.2.7/">1.2.7/</a> 17-Aug-2022 10:24 -
<a href="1.2.8/">1.2.8/</a> 24-Aug-2022 15:37 -
<a href="1.2.9/">1.2.9/</a> 10-Sep-2022 01:24 -
<a href="1.3.0/">1.3.0/</a> 21-Sep-2022 16:21 -
<a href="1.3.1/">1.3.1/</a> 28-Sep-2022 23:35 -
<a href="1.3.2/">1.3.2/</a> 07-Oct-2022 00:17 -
<a href="1.3.3/">1.3.3/</a> 19-Oct-2022 20:27 -
<a href="1.3.4/">1.3.4/</a> 02-Nov-2022 18:35 -
<a href="1.3.5/">1.3.5/</a> 19-Nov-2022 00:55 -
<a href="1.3.6/">1.3.6/</a> 30-Nov-2022 22:16 -
<a href="1.3.7/">1.3.7/</a> 04-Jan-2023 16:27 -
<a href="1.3.8/">1.3.8/</a> 10-Feb-2023 01:21 -
<a href="1.3.9/">1.3.9/</a> 16-Feb-2023 14:44 -
<a href="1.4.0/">1.4.0/</a> 08-Mar-2023 21:36 -
<a href="1.4.1/">1.4.1/</a> 15-Mar-2023 15:43 -
<a href="1.4.2/">1.4.2/</a> 16-Mar-2023 14:31 -
<a href="1.4.3/">1.4.3/</a> 30-Mar-2023 17:40 -
<a href="1.4.4/">1.4.4/</a> 30-Mar-2023 22:02 -
<a href="1.4.5/">1.4.5/</a> 14-Apr-2023 17:40 -
<a href="1.4.6/">1.4.6/</a> 27-Apr-2023 11:24 -
<a href="1.5.0/">1.5.0/</a> 12-Jun-2023 11:38 -
</pre>
<hr/><address style="font-size:small;">Artifactory/7.38.7 Server at XXX Port 8081</address></body></html>and considering the sed -En 's#.*[a-z]*_([0-9]+\.[0-9]+\.[0-9]+(-(rc|beta|alpha|oci)-?[0-9]*)?).*#\1#p',
you're trying to match something:
- that starts with
.*[a-z]*_, i.e. anything ending with an underscore_ - followed by a version number
([0-9]+\.[0-9]+\.[0-9]+(-(rc|beta|alpha|oci)-?[0-9]*)?).
There is nothing in the Artifactory HTML page that has a version number prefixed by an underscore _.
I can capture only stuff between <a ...>(version)/</a> but then, I assume that the page returned by HashiCorp, Artifactory or any other mirror MUST follow this format, which is more restrictive and brittle than the current implementation.
There was a problem hiding this comment.
Fair enough, my apologies. Feel free to mark this as resolved.
There was a problem hiding this comment.
Thanks @jack-parsons-bjss 🙏
Can we merge this PR then, if all is good?
|
The p flag for the s command of sed is meant to display the changed line, it doesn't seem possible to exclude the artifactory line and retrieve the version on other line in one command. So if you want to do it with one sed call, you should use two commands :
That did not remove duplicates, and i am not sure that will be more readable than the use of two grep calls. Changing the showed line 77 in the PR with |
|
the errors seems to be linked to golang/go#49425 |
|
the first go version without this bug is the 1.11, the first terraform release with 1.11+ compatibility instead of 1.10+ is the 0.11.9, i will upgrade the tests accordingly |
fix #400 #401