Unable to create sonatype nexus pod when using s3 bucket as pvc

[anchaubey]

/kind bug

NOTE: If this is a filesystem related bug, please take a look at the Mountpoint repo to submit a bug report

What happened?

I have created a nexus deployment where I am using s3 static provisioning feature https://github.com/awslabs/mountpoint-s3-csi-driver/blob/main/examples/kubernetes/static_provisioning/non_root.yaml

My pvc/pv config

---
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: s3-csi
provisioner: s3.csi.aws.com
parameters:
  mounter: rclone
  region: eu-west-1
  bucket: test-s3-blob-store
  storageClass: STANDARD
reclaimPolicy: Delete
volumeBindingMode: Immediate
---
apiVersion: v1
kind: PersistentVolume
metadata:
  name: pv-s3
spec:
  capacity:
    storage: 10Gi
  csi:
    driver: s3.csi.aws.com
    volumeHandle: bucketname/nexus-data
    volumeAttributes:
      bucketName: bucketname
  accessModes:
    - ReadWriteMany
  persistentVolumeReclaimPolicy: Retain
  storageClassName: s3-csi
  claimRef:
    namespace: namespace
    name: pvc-s3
  mountOptions:
    - allow-delete
    - region eu-west-1
    - allow-other
    - file-mode 0777
    - dir-mode 0777
    - allow-overwrite
    - uid=200
    - gid=200
    - prefix nexus-data/
  volumeMode: Filesystem
--- 
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: pvc-s3
  namespace: namespace
spec:
  accessModes:
    - ReadWriteMany
  storageClassName: s3-csi
  resources:
    requests:
      storage: 5Gi
  volumeName: pv-s3
---

What you expected to happen?

I am getting error once the pod is trying to launch as below:-

/sys/fs/cgroup/memory/memory.limit_in_bytes (No such file or directory)
java.lang.RuntimeException: /nexus-data/log/karaf.log (Invalid argument)

How to reproduce it (as minimally and precisely as possible)?

Anything else we need to know?:

Environment - EKS - 1.30

• Kubernetes version (use kubectl version):
• Driver version: latest

[anchaubey]

Just one more update that it's creating files in s3 bucket but still few directories are missing:-

The logs as below:-

OpenJDK 64-Bit Server VM warning: Unable to open cgroup memory limit file /sys/fs/cgroup/memory/memory.limit_in_bytes (No such file or directory)
java.lang.RuntimeException: /nexus-data/log/karaf.log (Invalid argument)
	at org.apache.karaf.main.util.BootstrapLogManager.getDefaultHandlerInternal(BootstrapLogManager.java:102)
	at org.apache.karaf.main.util.BootstrapLogManager.getDefaultHandlersInternal(BootstrapLogManager.java:137)
	at org.apache.karaf.main.util.BootstrapLogManager.getDefaultHandlers(BootstrapLogManager.java:70)
	at org.apache.karaf.main.util.BootstrapLogManager.configureLogger(BootstrapLogManager.java:75)
	at org.apache.karaf.main.Main.launch(Main.java:279)
	at org.sonatype.nexus.karaf.NexusMain.launch(NexusMain.java:120)
	at org.sonatype.nexus.karaf.NexusMain.main(NexusMain.java:52)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.exe4j.runtime.LauncherEngine.launch(LauncherEngine.java:85)
	at com.install4j.runtime.launcher.UnixLauncher.main(UnixLauncher.java:69)
Caused by: java.io.FileNotFoundException: /nexus-data/log/karaf.log (Invalid argument)
	at java.io.FileOutputStream.open0(Native Method)
	at java.io.FileOutputStream.open(FileOutputStream.java:270)
	at java.io.FileOutputStream.<init>(FileOutputStream.java:213)
	at org.apache.karaf.main.util.BootstrapLogManager$SimpleFileHandler.open(BootstrapLogManager.java:193)
	at org.apache.karaf.main.util.BootstrapLogManager$SimpleFileHandler.<init>(BootstrapLogManager.java:182)
	at org.apache.karaf.main.util.BootstrapLogManager.getDefaultHandlerInternal(BootstrapLogManager.java:100)
	... 12 more
java.lang.RuntimeException: /nexus-data/log/karaf.log (Invalid argument)
	at org.apache.karaf.main.util.BootstrapLogManager.getDefaultHandlerInternal(BootstrapLogManager.java:102)
	at org.apache.karaf.main.util.BootstrapLogManager.getDefaultHandlersInternal(BootstrapLogManager.java:137)
	at org.apache.karaf.main.util.BootstrapLogManager.getDefaultHandlers(BootstrapLogManager.java:70)
	at org.apache.karaf.main.util.BootstrapLogManager.configureLogger(BootstrapLogManager.java:75)
	at org.apache.karaf.main.KarafActivatorManager.<init>(KarafActivatorManager.java:48)
	at org.apache.karaf.main.Main.launch(Main.java:315)
	at org.sonatype.nexus.karaf.NexusMain.launch(NexusMain.java:120)
	at org.sonatype.nexus.karaf.NexusMain.main(NexusMain.java:52)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.exe4j.runtime.LauncherEngine.launch(LauncherEngine.java:85)
	at com.install4j.runtime.launcher.UnixLauncher.main(UnixLauncher.java:69)
Caused by: java.io.FileNotFoundException: /nexus-data/log/karaf.log (Invalid argument)
	at java.io.FileOutputStream.open0(Native Method)
	at java.io.FileOutputStream.open(FileOutputStream.java:270)
	at java.io.FileOutputStream.<init>(FileOutputStream.java:213)
	at org.apache.karaf.main.util.BootstrapLogManager$SimpleFileHandler.open(BootstrapLogManager.java:193)
	at org.apache.karaf.main.util.BootstrapLogManager$SimpleFileHandler.<init>(BootstrapLogManager.java:182)
	at org.apache.karaf.main.util.BootstrapLogManager.getDefaultHandlerInternal(BootstrapLogManager.java:100)
	... 13 more
Failed to write lock file: /nexus-data/lock
java.io.FileNotFoundException: /nexus-data/lock (Invalid argument)
	at java.io.RandomAccessFile.open0(Native Method)
	at java.io.RandomAccessFile.open(RandomAccessFile.java:316)
	at java.io.RandomAccessFile.<init>(RandomAccessFile.java:243)
	at org.sonatype.nexus.karaf.NexusFileLock.doLock(NexusFileLock.java:67)
	at org.sonatype.nexus.karaf.NexusFileLock.lock(NexusFileLock.java:54)
	at org.apache.karaf.main.Main.doMonitor(Main.java:425)
	at org.apache.karaf.main.Main.access$100(Main.java:78)
	at org.apache.karaf.main.Main$3.run(Main.java:409)
Nexus data directory already in use: /nexus-data

[unexge]

Hey @anchaubey, the CSI Driver only supports static provisioning at the moment, and using a StorageClass might cause unexpected behavior such as providing an unexpected PersistentVolume/S3-Bucket to a PVC. I'd suggest removing storage class and do one-to-one binding between your PVC and PV, see https://github.com/awslabs/mountpoint-s3-csi-driver/blob/main/docs/CONFIGURATION.md#static-provisioning.

For the issue, could you please share debug logs from Mountpoint by passing debug option to mountOptions of your PV? See https://github.com/awslabs/mountpoint-s3-csi-driver/blob/main/docs/LOGGING.md for more details on getting those logs.

[anchaubey]

Hello @unexge ,

Thanks for taking a look at the issue. Just fyi, I have tried removing storage class and just using the pv and pvc and with debug option. My concern is, if nexus or if all types of applications compatible to run with s3 bucket as pv.

----------------------
apiVersion: v1
kind: PersistentVolume
metadata:
  name: pv-s3
spec:
  capacity:
    storage: 10Gi
  csi:
    driver: s3.csi.aws.com
    volumeHandle: bucketname
    volumeAttributes:
      bucketName: bucketname
  accessModes:
    - ReadWriteMany
  persistentVolumeReclaimPolicy: Retain
  storageClassName: ""
  claimRef:
    namespace: namespace
    name: pvc-s3
  mountOptions:
    - allow-delete
    - region eu-west-1
    - allow-other
    - file-mode 0777
    - dir-mode 0777
    - allow-overwrite
    - uid=200
    - gid=200
    - prefix nexus-data/
    - debug
  volumeMode: Filesystem
--- 
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: pvc-s3
  namespace: namespace
spec:
  accessModes:
    - ReadWriteMany
  storageClassName: ""
  resources:
    requests:
      storage: 5Gi
  volumeName: pv-s3
---
-----------------------------------

This is my deployment file:

----------
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nexus
  namespace: namespace
  labels:
    app: nexus
spec:
  replicas: 1
  selector:
    matchLabels:
      app: nexus
  template:
    metadata:
      labels:
        app: nexus
    spec:
      containers:
        - name: nexus
          image: sonatype/nexus3:latest
          ports:
            - containerPort: 8081
          volumeMounts:
            - name: pv-s3
              mountPath: /nexus-data
      volumes:
        - name: pv-s3
          persistentVolumeClaim:
            claimName: pvc-s3

----------

Logs for your review:-

Oct 16 16:37:10 instance-name mount-s3[764101]: [DEBUG] unlink{req=90998 parent=7604 name="revision.location"}:head_object{id=43182 bucket=bucketname key=nexus-data/cache/bundle70/version0.0/revision.location}: mountpoint_s3_client::s3_crt_client: meta request finished duration=9.362943ms
Oct 16 16:37:10 instance-name mount-s3[764101]: [DEBUG] unlink{req=90998 parent=7604 name="revision.location"}:list_objects{id=43183 bucket=bucketname continued=false delimiter=/ max_keys=1 prefix=nexus-data/cache/bundle70/version0.0/revision.location/}: mountpoint_s3_client::s3_crt_client: S3 request finished request_type=Default crt_error=None http_status=200 range=None duration=23.038368ms ttfb=Some(22.573267ms) request_id=5MF76YGQTZ7TP06V
Oct 16 16:37:10 instance-name mount-s3[764101]: [DEBUG] unlink{req=90998 parent=7604 name="revision.location"}:list_objects{id=43183 bucket=bucketname continued=false delimiter=/ max_keys=1 prefix=nexus-data/cache/bundle70/version0.0/revision.location/}: mountpoint_s3_client::s3_crt_client: meta request finished duration=23.144546ms
Oct 16 16:37:10 instance-name mount-s3[764101]: [DEBUG] unlink{req=90998 parent=7604 name="revision.location"}: mountpoint_s3::inode: unlink on remote file will delete key nexus-data/cache/bundle70/version0.0/revision.location parent=7604 name="revision.location"
Oct 16 16:37:10 instance-name mount-s3[764101]: [DEBUG] unlink{req=90998 parent=7604 name="revision.location"}:delete_object{id=43184 bucket=bucketname key=nexus-data/cache/bundle70/version0.0/revision.location}: mountpoint_s3_client::s3_crt_client::delete_object: new request
Oct 16 16:37:10 instance-name mount-s3[764101]: [DEBUG] unlink{req=90998 parent=7604 name="revision.location"}:delete_object{id=43184 bucket=bucketname key=nexus-data/cache/bundle70/version0.0/revision.location}: mountpoint_s3_client::s3_crt_client: S3 request finished request_type=Default crt_error=None http_status=204 range=None duration=20.391065ms ttfb=Some(20.205786ms) request_id=5MF11F582V38H5ZE
Oct 16 16:37:10 instance-name mount-s3[764101]: [DEBUG] unlink{req=90998 parent=7604 name="revision.location"}:delete_object{id=43184 bucket=bucketname key=nexus-data/cache/bundle70/version0.0/revision.location}: mountpoint_s3_client::s3_crt_client: meta request finished duration=20.526109ms
Oct 16 16:37:10 instance-name mount-s3[764101]: [DEBUG] fuser::request: FUSE(91000) ino 0x0000000000001e49 FORGET nlookup 1
Oct 16 16:37:10 instance-name mount-s3[764101]: [DEBUG] fuser::request: FUSE(91002) ino 0x0000000000001db2 RMDIR name "version0.0"
Oct 16 16:37:10 instance-name mount-s3[764101]: [DEBUG] rmdir{req=91002 parent=7602 name="version0.0"}:head_object{id=43185 bucket=bucketname key=nexus-data/cache/bundle70/version0.0}: mountpoint_s3_client::s3_crt_client::head_object: new request
Oct 16 16:37:10 instance-name mount-s3[764101]: [DEBUG] rmdir{req=91002 parent=7602 name="version0.0"}:list_objects{id=43186 bucket=bucketname continued=false delimiter=/ max_keys=1 prefix=nexus-data/cache/bundle70/version0.0/}: mountpoint_s3_client::s3_crt_client::list_objects: new request
Oct 16 16:37:10 instance-name mount-s3[764101]: [DEBUG] rmdir{req=91002 parent=7602 name="version0.0"}:head_object{id=43185 bucket=bucketname key=nexus-data/cache/bundle70/version0.0}: mountpoint_s3_client::s3_crt_client: S3 request failed request_type=HeadObject crt_error=Some(Error(14343, "aws-c-s3: AWS_ERROR_S3_INVALID_RESPONSE_STATUS, Invalid response status from request")) http_status=404 range=None duration=9.239226ms ttfb=Some(9.016797ms) request_id=5MF4YKB8D4RFCB22
Oct 16 16:37:10 instance-name mount-s3[764101]: [DEBUG] rmdir{req=91002 parent=7602 name="version0.0"}:head_object{id=43185 bucket=bucketname key=nexus-data/cache/bundle70/version0.0}: mountpoint_s3_client::s3_crt_client:  duration=9.405017ms request_id=5MF4YKB8D4RFCB22 error=ServiceError(NotFound)meta request failed
Oct 16 16:37:10 instance-name mount-s3[764101]: [DEBUG] rmdir{req=91002 parent=7602 name="version0.0"}:head_object{id=43185 bucket=bucketname key=nexus-data/cache/bundle70/version0.0}: mountpoint_s3_client::s3_crt_client: meta request result: MetaRequestResult { response_status: 404, crt_error: Error(14343, "aws-c-s3: AWS_ERROR_S3_INVALID_RESPONSE_STATUS, Invalid response status from request"), error_response_headers: Some(Headers { inner: 0x7f8324020020 }), error_response_body: None }
Oct 16 16:37:10 instance-name mount-s3[764101]: [DEBUG] rmdir{req=91002 parent=7602 name="version0.0"}:list_objects{id=43186 bucket=bucketname continued=false delimiter=/ max_keys=1 prefix=nexus-data/cache/bundle70/version0.0/}: mountpoint_s3_client::s3_crt_client: S3 request finished request_type=Default crt_error=None http_status=200 range=None duration=10.529988ms ttfb=Some(9.514077ms) request_id=5MF9WYSK35MEKWYV
Oct 16 16:37:10 instance-name mount-s3[764101]: [DEBUG] rmdir{req=91002 parent=7602 name="version0.0"}:list_objects{id=43186 bucket=bucketname continued=false delimiter=/ max_keys=1 prefix=nexus-data/cache/bundle70/version0.0/}: mountpoint_s3_client::s3_crt_client: meta request finished duration=10.634356ms
Oct 16 16:37:10 instance-name mount-s3[764101]: [WARN] rmdir{req=91002 parent=7602 name="version0.0"}: mountpoint_s3::fuse: rmdir failed: inode error: file "version0.0" does not exist in parent inode 7602 (full key "nexus-data/cache/bundle70/")
Oct 16 16:37:10 instance-name mount-s3[764101]: [DEBUG] fuser::request: FUSE(91004) ino 0x0000000000001afc LOOKUP name "bundle8"
Oct 16 16:37:10 instance-name mount-s3[764101]: [DEBUG] lookup{req=91004 ino=6908 name="bundle8"}:head_object{id=43187 bucket=bucketname key=nexus-data/cache/bundle8}: mountpoint_s3_client::s3_crt_client::head_object: new request
Oct 16 16:37:10 instance-name mount-s3[764101]: [DEBUG] lookup{req=91004 ino=6908 name="bundle8"}:list_objects{id=43188 bucket=bucketname continued=false delimiter=/ max_keys=1 prefix=nexus-data/cache/bundle8/}: mountpoint_s3_client::s3_crt_client::list_objects: new request
Oct 16 16:37:10 instance-name mount-s3[764101]: [DEBUG] lookup{req=91004 ino=6908 name="bundle8"}:head_object{id=43187 bucket=bucketname key=nexus-data/cache/bundle8}: mountpoint_s3_client::s3_crt_client: S3 request failed request_type=HeadObject crt_error=Some(Error(14343, "aws-c-s3: AWS_ERROR_S3_INVALID_RESPONSE_STATUS, Invalid response status from request")) http_status=404 range=None duration=7.368226ms ttfb=Some(7.180246ms) request_id=5MF4BV27DSR03ENS
Oct 16 16:37:10 instance-name mount-s3[764101]: [DEBUG] lookup{req=91004 ino=6908 name="bundle8"}:head_object{id=43187 bucket=bucketname key=nexus-data/cache/bundle8}: mountpoint_s3_client::s3_crt_client:  duration=7.561713ms request_id=5MF4BV27DSR03ENS error=ServiceError(NotFound)meta request failed
Oct 16 16:37:10 instance-name mount-s3[764101]: [DEBUG] lookup{req=91004 ino=6908 name="bundle8"}:head_object{id=43187 bucket=bucketname key=nexus-data/cache/bundle8}: mountpoint_s3_client::s3_crt_client: meta request result: MetaRequestResult { response_status: 404, crt_error: Error(14343, "aws-c-s3: AWS_ERROR_S3_INVALID_RESPONSE_STATUS, Invalid response status from request"), error_response_headers: Some(Headers { inner: 0x7f8324043970 }), error_response_body: None }
Oct 16 16:37:10 instance-name mount-s3[764101]: [DEBUG] lookup{req=91004 ino=6908 name="bundle8"}:list_objects{id=43188 bucket=bucketname continued=false delimiter=/ max_keys=1 prefix=nexus-data/cache/bundle8/}: mountpoint_s3_client::s3_crt_client: S3 request finished request_type=Default crt_error=None http_status=200 range=None duration=19.527035ms ttfb=Some(12.810999ms) request_id=5MF34WNJ8PN0WX8K
Oct 16 16:37:10 instance-name mount-s3[764101]: [DEBUG] lookup{req=91004 ino=6908 name="bundle8"}:list_objects{id=43188 bucket=bucketname continued=false delimiter=/ max_keys=1 prefix=nexus-data/cache/bundle8/}: mountpoint_s3_client::s3_crt_client: meta request finished duration=19.687363ms
Oct 16 16:37:10 instance-name mount-s3[764101]: [DEBUG] fuser::request: FUSE(91006) ino 0x0000000000001b46 OPENDIR flags 0x18800
Oct 16 16:37:10 instance-name mount-s3[764101]: [DEBUG] fuser::request: FUSE(91008) ino 0x0000000000001b46 READDIRPLUS fh FileHandle(4551), offset 0, size 4096
Oct 16 16:37:10 instance-name mount-s3[764101]: [DEBUG] readdirplus{req=91008 ino=6982 fh=4551 offset=0}:list_objects{id=43189 bucket=bucketname continued=false delimiter=/ max_keys=1000 prefix=nexus-data/cache/bundle8/}: mountpoint_s3_client::s3_crt_client::list_objects: new request
Oct 16 16:37:10 instance-name mount-s3[764101]: [DEBUG] readdirplus{req=91008 ino=6982 fh=4551 offset=0}:list_objects{id=43189 bucket=bucketname continued=false delimiter=/ max_keys=1000 prefix=nexus-data/cache/bundle8/}: mountpoint_s3_client::s3_crt_client: S3 request finished request_type=Default crt_error=None http_status=200 range=None duration=12.1366ms ttfb=Some(11.208029ms) request_id=5MF03AY1T82XAZJ8
Oct 16 16:37:10 instance-name mount-s3[764101]: [DEBUG] readdirplus{req=91008 ino=6982 fh=4551 offset=0}:list_objects{id=43189 bucket=bucketname continued=false delimiter=/ max_keys=1000 prefix=nexus-data/cache/bundle8/}: mountpoint_s3_client::s3_crt_client: meta request finished duration=12.31884ms
Oct 16 16:37:10 instance-name mount-s3[764101]: [DEBUG] readdirplus{req=91008 ino=6982 fh=4551 offset=0}: mountpoint_s3::inode: inode needs to be recreated parent=6982 name="bundle.info" ino=6989
Oct 16 16:37:10 instance-name mount-s3[764101]: [DEBUG] fuser::request: FUSE(91010) ino 0x0000000000001b4d FORGET nlookup 1
Oct 16 16:37:10 instance-name mount-s3[764101]: [DEBUG] fuser::request: FUSE(91012) ino 0x0000000000001b46 READDIRPLUS fh FileHandle(4551), offset 4, size 4096
Oct 16 16:37:10 instance-name mount-s3[764101]: [DEBUG] fuser::request: FUSE(91014) ino 0x0000000000001b46 RELEASEDIR fh FileHandle(4551), flags 0x18800, flush false, lock owner None
Oct 16 16:37:10 instance-name mount-s3[764101]: [DEBUG] fuser::request: FUSE(91016) ino 0x0000000000001b46 UNLINK name "bundle.info"
Oct 16 16:37:10 instance-name mount-s3[764101]: [DEBUG] unlink{req=91016 parent=6982 name="bundle.info"}:head_object{id=43190 bucket=bucketname key=nexus-data/cache/bundle8/bundle.info}: mountpoint_s3_client::s3_crt_client::head_object: new request
Oct 16 16:37:10 instance-name mount-s3[764101]: [DEBUG] unlink{req=91016 parent=6982 name="bundle.info"}:list_objects{id=43191 bucket=bucketname continued=false delimiter=/ max_keys=1 prefix=nexus-data/cache/bundle8/bundle.info/}: mountpoint_s3_client::s3_crt_client::list_objects: new request
Oct 16 16:37:10 instance-name mount-s3[764101]: [DEBUG] unlink{req=91016 parent=6982 name="bundle.info"}:head_object{id=43190 bucket=bucketname key=nexus-data/cache/bundle8/bundle.info}: mountpoint_s3_client::s3_crt_client: S3 request finished request_type=HeadObject crt_error=None http_status=200 range=None duration=10.759079ms ttfb=Some(10.510469ms) request_id=5MF07WNXFCEV76FZ
Oct 16 16:37:10 instance-name mount-s3[764101]: [DEBUG] unlink{req=91016 parent=6982 name="bundle.info"}:head_object{id=43190 bucket=bucketname key=nexus-data/cache/bundle8/bundle.info}: mountpoint_s3_client::s3_crt_client: meta request finished duration=10.90643ms
Oct 16 16:37:10 instance-name mount-s3[764101]: [DEBUG] unlink{req=91016 parent=6982 name="bundle.info"}:list_objects{id=43191 bucket=bucketname continued=false delimiter=/ max_keys=1 prefix=nexus-data/cache/bundle8/bundle.info/}: mountpoint_s3_client::s3_crt_client: S3 request finished request_type=Default crt_error=None http_status=200 range=None duration=13.546949ms ttfb=Some(10.318488ms) request_id=5MF0NYGC5Y6P6AN5
Oct 16 16:37:10 instance-name mount-s3[764101]: [DEBUG] unlink{req=91016 parent=6982 name="bundle.info"}:list_objects{id=43191 bucket=bucketname continued=false delimiter=/ max_keys=1 prefix=nexus-data/cache/bundle8/bundle.info/}: mountpoint_s3_client::s3_crt_client: meta request finished duration=13.663924ms
Oct 16 16:37:10 instance-name mount-s3[764101]: [DEBUG] unlink{req=91016 parent=6982 name="bundle.info"}: mountpoint_s3::inode: unlink on remote file will delete key nexus-data/cache/bundle8/bundle.info parent=6982 name="bundle.info"
Oct 16 16:37:10 instance-name mount-s3[764101]: [DEBUG] unlink{req=91016 parent=6982 name="bundle.info"}:delete_object{id=43192 bucket=bucketname key=nexus-data/cache/bundle8/bundle.info}: mountpoint_s3_client::s3_crt_client::delete_object: new request
Oct 16 16:37:10 instance-name mount-s3[764101]: [DEBUG] unlink{req=91016 parent=6982 name="bundle.info"}:delete_object{id=43192 bucket=bucketname key=nexus-data/cache/bundle8/bundle.info}: mountpoint_s3_client::s3_crt_client: S3 request finished request_type=Default crt_error=None http_status=204 range=None duration=25.498169ms ttfb=Some(25.292989ms) request_id=5MF4Q1ZRX4ER6N61
Oct 16 16:37:10 instance-name mount-s3[764101]: [DEBUG] unlink{req=91016 parent=6982 name="bundle.info"}:delete_object{id=43192 bucket=bucketname key=nexus-data/cache/bundle8/bundle.info}: mountpoint_s3_client::s3_crt_client: meta request finished duration=25.641556ms
Oct 16 16:37:10 instance-name mount-s3[764101]: [DEBUG] fuser::request: FUSE(91018) ino 0x0000000000001e4a FORGET nlookup 1
Oct 16 16:37:10 instance-name mount-s3[764101]: [DEBUG] fuser::request: FUSE(91020) ino 0x0000000000001b46 GETATTR
Oct 16 16:37:10 instance-name mount-s3[764101]: [DEBUG] fuser::request: FUSE(91022) ino 0x0000000000001b48 OPENDIR flags 0x18800
Oct 16 16:37:10 instance-name mount-s3[764101]: [DEBUG] fuser::request: FUSE(91024) ino 0x0000000000001b48 READDIRPLUS fh FileHandle(4552), offset 0, size 4096
Oct 16 16:37:10 instance-name mount-s3[764101]: [DEBUG] readdirplus{req=91024 ino=6984 fh=4552 offset=0}:list_objects{id=43193 bucket=bucketname continued=false delimiter=/ max_keys=1000 prefix=nexus-data/cache/bundle8/version0.0/}: mountpoint_s3_client::s3_crt_client::list_objects: new request
Oct 16 16:37:10 instance-name mount-s3[764101]: [DEBUG] readdirplus{req=91024 ino=6984 fh=4552 offset=0}:list_objects{id=43193 bucket=bucketname continued=false delimiter=/ max_keys=1000 prefix=nexus-data/cache/bundle8/version0.0/}: mountpoint_s3_client::s3_crt_client: S3 request finished request_type=Default crt_error=None http_status=200 range=None duration=13.54011ms ttfb=Some(12.60853ms) request_id=5MF021GZ5Q946JW1
Oct 16 16:37:10 instance-name mount-s3[764101]: [DEBUG] readdirplus{req=91024 ino=6984 fh=4552 offset=0}:list_objects{id=43193 bucket=bucketname continued=false delimiter=/ max_keys=1000 prefix=nexus-data/cache/bundle8/version0.0/}: mountpoint_s3_client::s3_crt_client: meta request finished duration=13.688316ms
Oct 16 16:37:10 instance-name mount-s3[764101]: [DEBUG] readdirplus{req=91024 ino=6984 fh=4552 offset=0}: mountpoint_s3::inode: inode needs to be recreated parent=6984 name="revision.location" ino=6985
Oct 16 16:37:10 instance-name mount-s3[764101]: [DEBUG] fuser::request: FUSE(91026) ino 0x0000000000001b49 FORGET nlookup 1
Oct 16 16:37:10 instance-name mount-s3[764101]: [DEBUG] fuser::request: FUSE(91028) ino 0x0000000000001b48 READDIRPLUS fh FileHandle(4552), offset 3, size 4096
Oct 16 16:37:10 instance-name mount-s3[764101]: [DEBUG] fuser::request: FUSE(91030) ino 0x0000000000001b48 RELEASEDIR fh FileHandle(4552), flags 0x18800, flush false, lock owner None
Oct 16 16:37:10 instance-name mount-s3[764101]: [DEBUG] fuser::request: FUSE(91032) ino 0x0000000000001b48 UNLINK name "revision.location"
Oct 16 16:37:10 instance-name mount-s3[764101]: [DEBUG] unlink{req=91032 parent=6984 name="revision.location"}:head_object{id=43194 bucket=bucketname key=nexus-data/cache/bundle8/version0.0/revision.location}: mountpoint_s3_client::s3_crt_client::head_object: new request
Oct 16 16:37:10 instance-name mount-s3[764101]: [DEBUG] unlink{req=91032 parent=6984 name="revision.location"}:list_objects{id=43195 bucket=bucketname continued=false delimiter=/ max_keys=1 prefix=nexus-data/cache/bundle8/version0.0/revision.location/}: mountpoint_s3_client::s3_crt_client::list_objects: new request
Oct 16 16:37:10 instance-name mount-s3[764101]: [DEBUG] unlink{req=91032 parent=6984 name="revision.location"}:list_objects{id=43195 bucket=bucketname continued=false delimiter=/ max_keys=1 prefix=nexus-data/cache/bundle8/version0.0/revision.location/}: mountpoint_s3_client::s3_crt_client: S3 request finished request_type=Default crt_error=None http_status=200 range=None duration=9.669027ms ttfb=Some(8.771106ms) request_id=5MF19505QFPZ0K4M
Oct 16 16:37:10 instance-name mount-s3[764101]: [DEBUG] unlink{req=91032 parent=6984 name="revision.location"}:list_objects{id=43195 bucket=bucketname continued=false delimiter=/ max_keys=1 prefix=nexus-data/cache/bundle8/version0.0/revision.location/}: mountpoint_s3_client::s3_crt_client: meta request finished duration=9.813589ms
Oct 16 16:37:10 instance-name mount-s3[764101]: [DEBUG] unlink{req=91032 parent=6984 name="revision.location"}:head_object{id=43194 bucket=bucketname key=nexus-data/cache/bundle8/version0.0/revision.location}: mountpoint_s3_client::s3_crt_client: S3 request finished request_type=HeadObject crt_error=None http_status=200 range=None duration=10.088588ms ttfb=Some(9.887518ms) request_id=5MFBHQ7D3ZQ1QNXT
Oct 16 16:37:10 instance-name mount-s3[764101]: [DEBUG] unlink{req=91032 parent=6984 name="revision.location"}:head_object{id=43194 bucket=bucketname key=nexus-data/cache/bundle8/version0.0/revision.location}: mountpoint_s3_client::s3_crt_client: meta request finished duration=10.221582ms
Oct 16 16:37:10 instance-name mount-s3[764101]: [DEBUG] unlink{req=91032 parent=6984 name="revision.location"}: mountpoint_s3::inode: unlink on remote file will delete key nexus-data/cache/bundle8/version0.0/revision.location parent=6984 name="revision.location"
Oct 16 16:37:10 instance-name mount-s3[764101]: [DEBUG] unlink{req=91032 parent=6984 name="revision.location"}:delete_object{id=43196 bucket=bucketname key=nexus-data/cache/bundle8/version0.0/revision.location}: mountpoint_s3_client::s3_crt_client::delete_object: new request
Oct 16 16:37:10 instance-name mount-s3[764101]: [DEBUG] unlink{req=91032 parent=6984 name="revision.location"}:delete_object{id=43196 bucket=bucketname key=nexus-data/cache/bundle8/version0.0/revision.location}: mountpoint_s3_client::s3_crt_client: S3 request finished request_type=Default crt_error=None http_status=204 range=None duration=19.202184ms ttfb=Some(19.008115ms) request_id=5MF9NQW4QRRNFBA6
Oct 16 16:37:10 instance-name mount-s3[764101]: [DEBUG] unlink{req=91032 parent=6984 name="revision.location"}:delete_object{id=43196 bucket=bucketname key=nexus-data/cache/bundle8/version0.0/revision.location}: mountpoint_s3_client::s3_crt_client: meta request finished duration=19.352424ms
Oct 16 16:37:10 instance-name mount-s3[764101]: [DEBUG] fuser::request: FUSE(91034) ino 0x0000000000001e4b FORGET nlookup 1
Oct 16 16:37:10 instance-name mount-s3[764101]: [DEBUG] fuser::request: FUSE(91036) ino 0x0000000000001b46 RMDIR name "version0.0"
Oct 16 16:37:10 instance-name mount-s3[764101]: [DEBUG] rmdir{req=91036 parent=6982 name="version0.0"}:head_object{id=43197 bucket=bucketname key=nexus-data/cache/bundle8/version0.0}: mountpoint_s3_client::s3_crt_client::head_object: new request
Oct 16 16:37:10 instance-name mount-s3[764101]: [DEBUG] rmdir{req=91036 parent=6982 name="version0.0"}:list_objects{id=43198 bucket=bucketname continued=false delimiter=/ max_keys=1 prefix=nexus-data/cache/bundle8/version0.0/}: mountpoint_s3_client::s3_crt_client::list_objects: new request
Oct 16 16:37:10 instance-name mount-s3[764101]: [DEBUG] rmdir{req=91036 parent=6982 name="version0.0"}:head_object{id=43197 bucket=bucketname key=nexus-data/cache/bundle8/version0.0}: mountpoint_s3_client::s3_crt_client: S3 request failed request_type=HeadObject crt_error=Some(Error(14343, "aws-c-s3: AWS_ERROR_S3_INVALID_RESPONSE_STATUS, Invalid response status from request")) http_status=404 range=None duration=7.029355ms ttfb=Some(6.793135ms) request_id=5MF7PYH2KEDMH2SR
Oct 16 16:37:10 instance-name mount-s3[764101]: [DEBUG] rmdir{req=91036 parent=6982 name="version0.0"}:head_object{id=43197 bucket=bucketname key=nexus-data/cache/bundle8/version0.0}: mountpoint_s3_client::s3_crt_client:  duration=7.263797ms request_id=5MF7PYH2KEDMH2SR error=ServiceError(NotFound)meta request failed
Oct 16 16:37:10 instance-name mount-s3[764101]: [DEBUG] rmdir{req=91036 parent=6982 name="version0.0"}:head_object{id=43197 bucket=bucketname key=nexus-data/cache/bundle8/version0.0}: 

[anchaubey]

This is pod status while creation @unexge ,

Error: failed to create containerd task: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: error mounting "/var/lib/kubelet/pods/65dcb368-708d-4aee-accd-8c6740ec024b/volume-subpaths/nexus-nexus3-properties/nexus3/3" to rootfs at "/nexus-data/etc/fabric/nexus-store.properties": open /run/containerd/io.containerd.runtime.v2.task/k8s.io/nexus3/rootfs/nexus-data/etc/fabric/nexus-store.properties: operation not permitted: unknown