Concerns over sonicstorage.blob.core.windows.net downtime

[wdoekes]

Over the weekend, sonicstorage.blob.core.windows.net had been in private mode (as far as I can tell).
That meant that everyone who tried to build a SONiC image from friday until now, couldn't.

These tickets all concern those build failures:
#20749
#20737
#20685

In one of them, @liushilongbuaa writes "Fixed" without any explanation. In another, @Gfrom2016 says that "[public] access has been re-enabled".

As a community member, I'd very much like to know a few things:

• How did the access get toggled to private?
• What are the chances that this ever happens again?
• How can we guard against this? Both through technical means (mirrors, cache, fallback location at upstream) and through policy means (who gets to toggle this)?

We'd love to help test and develop SONiC as SDN core component, but we will be hesitant to invest time and money if we cannot depend on the build consistenly working. (In this case: depend on dependencies being available.)

For the debian components, we know that they will be available. And github objects we can certainly source from somewhere if needed. But for instance the broadcom files, like libsaibcm_11.2.13.1_amd64.deb, seem to be available only on the sonicstorage. (Where is its upstream?) This makes us fear that someone on the internet can just pull the plug and disable the builds without any moments notice.

Please take a moment and address our concerns.

Thank you!
Walter Doekes
OSSO B.V.

[prgeor]

@wangxin can you provide explanation on how this fixed?

[liushilongbuaa]

Some packages are stored on Azure(sonicstorage controlled by microsoft). Some components build refers these packages.
It depends on each component. Maybe you can download these files to local place for backup.

[wdoekes]

@liushilongbuaa I appreciate you thinking along, but downloading the necessary files separately is obviously something that has occurred to me already. Right now I'm rigidly holding on to all build artifacts/cache/downloads in case something goes wrong again.

For example, here's a Makefile that collects all libsaibcm*deb files I could find, just in case I need a different version:

.PHONY: all
all: \
        11.2.13.1 \
        11.2.12.1 \
...
libsaibcm_dnx-dev_$(BCMVER)_amd64.deb:
        wget https://sonicstorage.blob.core.windows.net/public/sai/sai-broadcom/$(BCMBRANCH)/$(BCMVER)/dnx/$@
endif

Makefile.txt <- download

But right now these are just band-aids. If things are down again we'll have to start a big digging operation to find/fix the build.

I can backup a Debian repo. I can backup Git projects. I cannot backup the sonicstorage, because I have no file listing. (The counter argument that I've downloaded the files already earlier doesn't hold true. If I change a build setting, a new download might just pop up.)

What we're still looking for is:

• A simple explanation of what went wrong. Did the intern press the is_private checkbox? Can they just do that?
• A procedural explanation of what is stored where/why. A file listing of the entire storage bucket helps. We're particularly interested in the origin of parts that are not source code: like libbcm. Where would we source updates if someone decides that the sonicstorage gets no updates?
• A technical solution, like a single HTTP_PROXY setting so we can use a local proxy to cache everything and not depend on sonicstorage being up. (This is something we can contribute to.)

Thanks! If you have the answer to either of these three questions, please share. (Don't wait to get them all.)

[liushilongbuaa]

@liushilongbuaa I appreciate you thinking along, but downloading the necessary files separately is obviously something that has occurred to me already. Right now I'm rigidly holding on to all build artifacts/cache/downloads in case something goes wrong again.

For example, here's a Makefile that collects all libsaibcm*deb files I could find, just in case I need a different version:

.PHONY: all
all: \
        11.2.13.1 \
        11.2.12.1 \
...
libsaibcm_dnx-dev_$(BCMVER)_amd64.deb:
        wget https://sonicstorage.blob.core.windows.net/public/sai/sai-broadcom/$(BCMBRANCH)/$(BCMVER)/dnx/$@
endif

Makefile.txt <- download

But right now these are just band-aids. If things are down again we'll have to start a big digging operation to find/fix the build.

I can backup a Debian repo. I can backup Git projects. I cannot backup the sonicstorage, because I have no file listing. (The counter argument that I've downloaded the files already earlier doesn't hold true. If I change a build setting, a new download might just pop up.)

What we're still looking for is:

• A simple explanation of what went wrong. Did the intern press the is_private checkbox? Can they just do that?
• A procedural explanation of what is stored where/why. A file listing of the entire storage bucket helps. We're particularly interested in the origin of parts that are not source code: like libbcm. Where would we source updates if someone decides that the sonicstorage gets no updates?
• A technical solution, like a single HTTP_PROXY setting so we can use a local proxy to cache everything and not depend on sonicstorage being up. (This is something we can contribute to.)

Thanks! If you have the answer to either of these three questions, please share. (Don't wait to get them all.)

1. We have internal security requirement on storage account settings. Change on security settings caused access issues.
2. You can file list here https://github.com/sonic-net/sonic-buildimage/blob/master/files/build/versions/default/versions-web
3. I don't know he reason to use sonictorage. Maybe you can search each url in Makefile and find the original PR to find the reason.

[wdoekes]

1. We have internal security requirement on storage account settings. Change on security settings caused access issues.

Not an entirely satisfactory answer, but I'll take it 😺

3. I don't know [the] reason to use [sonicstorage]. Maybe you can search each url in Makefile and find the original PR to find the reason.

Fair enough. I think I'm particularly interested in the broadcom binaries. I'll try and find someone who knows through other channels.

2. You can [find the] file list here https://github.com/sonic-net/sonic-buildimage/blob/master/files/build/versions/default/versions-web

Nice. That is definitely something.

I do wonder if that:

(a) is up to date -- seeing that ipmitool has versions that have changed several times, see #20195 #20836 ;

(b) includes everything -- seeing that the libbcm* I mentioned above (Makefile.txt) is not in there.

Can you elaborate on how this list is maintained/updated?

[liushilongbuaa]

@wdoekes , PR17878 involves build ipmtool locally.
PR20349 reverts the change.

libbcm* locates platform/broadcom/sai.mk.

[wdoekes]

@liushilongbuaa : thank you for your response. However, I was not asking for the particular PRs. My point was:

• if there is a list of files that are used: this list
• why is it not up to date?

I used the ipmitool example because I know that it has been updated recently, but the list has not.

The list has:

https://deb.debian.org/debian/pool/main/i/ipmitool/ipmitool_1.8.19-4.debian.tar.xz==aee97d7c05ca32815652bf899a029f9d
https://deb.debian.org/debian/pool/main/i/ipmitool/ipmitool_1.8.19-4.dsc==72f67264021488c0219de23b840d0336
https://deb.debian.org/debian/pool/main/i/ipmitool/ipmitool_1.8.19.orig.tar.gz==0aa41c99d93ce129cf00a9b8803ed8c9

But the PRs mentioned, do:

• Ipmitool bookworm: Fix and patch enterprise-numbers URL #17878 - add 1.8.19-4
• Upgrade ipmitool versions to 1.8.19-4+deb12u1 #20195 - replace 1.8.19-4 with 1.8.19-4+deb12u1
• Remove ipmitool compilation #20349 - remove 1.8.19-4+deb12u1

In the mean time, the list has not been updated to first include:

https://deb.debian.org/debian/pool/main/i/ipmitool/ipmitool_1.8.19-4+deb12u2.debian.tar.xz==1886c4054222f9b70617bc78656d0823

and then another commit to remove it.

For the ipmitool example the list was only updated by this PR: #18037 - [master] Upgrade SONiC package Versions

But, I think it should have been updated at every step when versions are changed (i.e. after every PR mentioned here). And because it isn't, the list isn't guaranteed to be correct or complete.

That is what I was referring to when I said "I wonder if [it] is up to date [..] and includes everything"

And yes, platform/broadcom/sai.mk is where the libsaibcm* can be found (I found those, see Makefile.txt above). But my point here is: if libsaibcm* is not in the list, what else is also not in the list?

---

TL;DR:

• Nice that there is a list;
• unfortunately the list is not complete:
  • it is not up to date;
  • it does not include everything.

So, it only partially solves my problem of not knowing which files I need.