Deleting / Re-creating Bucket With Versioning Does Not Enable Versioning through Web Console

[markive]

We are testing MinIO, and the first test we did was to create a bucket without versioning (to look at file structures on disk). Then we wanted to test versioning so we turned it on. We found that our bucket wasn't creating files with version IDs (versionID = null) when uploaded through the API.

When we created a new bucket with versioning turned on from the beginning everything was as expected.

We therefore deleted the original bucket (by deleting all objects and versions first) all in the console and re-created it through the console with versioning turned on. We then found we got the same incorrect behaviour. So it seems like we have 'bricked' that bucket name. I was able to confirm there is no reference to the original bucket name on the file storage.

Another weird behavior is that objects uploaded through the API have this problem, but objects uploaded through the console web app do get a version ID for the first object version.

The only way to resolve the issue is to manually create the bucket using the command line:
mc mb --with-versioning local/mark-voffon-test Bucket created successfully 'local/mark-voffon-test'.

Expected Behavior

When you re-create a bucket with versioning, it should work when using the console web app. It should have the same behavior as the command line.

Current Behavior

Objects behave like they were first uploaded to a bucket with versioning turned off, and on a second object upload you get a version ID.

The console web app isn't re-creating a deleted bucket with versioning turned on the same as the command line.

Video link demonstrating the issue: https://youtu.be/lw41JBWDMYU

Steps to Reproduce (for bugs)

1. Create bucket with versioning turned off.

2. Upload file through API (we used c# and node scripts to test)

3. You will have file with versionID=null

4. Turn on versioning for the bucket

5. Upload same file through API

6. You should have versionId set by MinIO

7. Upload new file through API

8. You will have file with versionID=null (this is wrong, bucket should version new files from now on).

9. Delete files and bucket completely through console web app

10. Re-create bucket with versioning turned on from the beginning

11. Follow steps above

12. Behavior is still incorrect

13. Delete files and bucket completely through console web app

14. Create same bucket with mb command with versioning enabled

15. Follow steps above

16. Everything works according to the documentation.

Regression

I assume yes
We are running the latest version as of a few days ago.

Your Environment

We are running this as a Dockerised container in Azure Container Apps

• MinIO version used (minio --version):
  minio version RELEASE.2024-11-07T00-52-20Z (commit-id=cefc43e4daa4cbb490ef6726ea374e26a93eb85e)
  Runtime: go1.23.3 linux/amd64
  License: GNU AGPLv3 - https://www.gnu.org/licenses/agpl-3.0.html
  Copyright: 2015-2024 MinIO, Inc.

• Operating System and version (uname -a):
  Linux rdrive-minio-ea--fh4gz0c-768ff885d5-8hxtt 5.15.164.1-1.cm2 Profiling endpoints for mcs #1 SMP Sun Aug 18 19:16:21 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux

[ramondeklein]

I have created the following script (it assumes alias test and will destroy the bucket test):

mc rb local/test --force
mc mb local/test
echo "test 1" | mc pipe local/test/test-object
mc version enable local/test
mc ls --versions local/test
echo "test 1" | mc pipe local/test/test-object
mc ls --versions local/test

This is the output:

Bucket created successfully `local/test`.
 0 B / ? ┃...┃7 bytes -> `local/test/test-object`
local/test versioning is enabled
[2024-12-05 11:19:35 CET]     7B STANDARD null v1 PUT test-object
 0 B / ? ┃...┃7 bytes -> `local/test/test-object`
[2024-12-05 11:19:35 CET]     7B STANDARD aafb8da3-d1dd-4ae8-af4e-dd25281eff37 v2 PUT test-object
[2024-12-05 11:19:35 CET]     7B STANDARD null v1 PUT test-object

Versioning works fine. I'm not even sure if you can actually disable versioning when uploading data, but could you tell me which SDK you are using and how you're uploading the object that isn't versioned? You also may want to post the trace mc admin trace -v <alias> when you upload the object.

[markive]

@ramondeklein to clarify, the issue I am experiencing only happens when the console web app is used, not the command line as per your test.

I have recorded a youtube video that illustrates the steps and where the inconsistency is: https://youtu.be/lw41JBWDMYU

Do let me know if you need any other information.

For my testing I am using the latest c# sdk for .net 8. But we have also tested the latest node sdk. We don't believe this is the issue, but there's something wrong with the console web app.

Do let me know if I've posted this issue to the wrong repo. Thanks!

[marktheunissen]

Hi @markive I've followed your steps and cannot reproduce, yes I used the console. Could you please repeat the steps usingmc instead of your custom code? See below. Also, you can follow the verbose server logs using mc admin trace -v ALIAS and look out for calls to PUT /bucket/?versioning= with XML bodies like <VersioningConfiguration><Status>Enabled</Status><ExcludeFolders>true</ExcludeFolders></VersioningConfiguration>

And also calls when you use your SDK code vs. mc client. Can you spot any difference to the call bodies?

Create bucket with versioning turned off. ✅
Upload file through API (we used c# and node scripts to test) ✅

mc cp sample-cradle.jpg local/mybucket/sample-cradle.jpg

You will have file with versionID=null ✅

❯ mc ls --versions local/mybucket/sample-cradle.jpg
[2024-12-06 16:50:49 AEDT] 142KiB STANDARD null v1 PUT sample-cradle.jpg

Turn on versioning for the bucket ✅

Done in console

Upload same file through API ✅

❯ mc cp sample-cradle.jpg local/mybucket/sample-cradle.jpg

You should have versionId set by MinIO ✅

❯ mc ls --versions local/mybucket/sample-cradle.jpg
[2024-12-06 16:51:18 AEDT] 142KiB STANDARD de72f05e-e9e4-44f2-b3f2-94377b963ae7 v2 PUT sample-cradle.jpg
[2024-12-06 16:50:49 AEDT] 142KiB STANDARD null v1 PUT sample-cradle.jpg

Upload new file through API ✅

❯ mc cp sample-cradle.jpg local/mybucket/sample-cradle-2.jpg

You will have file with versionID=null (this is wrong, bucket should version new files from now on). ❌

❯ mc ls --versions local/mybucket/sample-cradle-2.jpg
[2024-12-06 16:51:29 AEDT] 142KiB STANDARD c7581aae-8045-41d2-aeac-f971b656e842 v1 PUT sample-cradle-2.jpg

As you can see above the version id is correct.

[marktheunissen]

You can also compare the API calls made by the console and the calls made by the mc command in your mc admin trace and see that they are the same. In fact it's the same client library (minio-go) so I don't see how enabling versioning on the bucket in the console could make a difference.

[marktheunissen]

We are running this as a Dockerised container in Azure Container Apps

Just run it locally and see if you can reproduce on local machine with minio server ./data or whatever

[markive]

Hi @marktheunissen, thanks for looking into it. I don't have a linux system other than a containerized environment so I don't know how I can run mc locally to test that. I could run it under windows, would that be a fair test?

I think focusing on the sdk shouldn't be the issue (we also tried with Node). Since versionId is controlled by minio, it should not be possible to successfully store an object (200 response) with versioning turned on through any means which results in versionId=null.

This is the request made by the .Net SDK, it's a PUT always, and in fact all the SDKs minio provides don't support POST:

Here is the API request that the console web app makes, it uses a POST even when uploading new versions of an existing file:

[markive]

I think the weirdest part of the behavior we are seeing is, why when a bucket is re-created does versioning not work, but when it is created for the first time with versioning turned on does it work perfectly? That's very inconsistent behavior and I'm 100% using the console web app for those operations.

[marktheunissen]

You can run mc on Windows and it just connects to any MinIO service that can be reached on the network

[marktheunissen]

why when a bucket is re-created does versioning not work, but when it is created for the first time with versioning turned on does it work perfectly?

Is this reproducible through mc commands? Like mc mb to make a bucket, etc? If you could provide that it would be very helpful. It might be that some bucket metadata is not cleaned up but we would need a script or something that clearly shows this.