#2502 : mass nifti deletion update

[DuckflipXYZ]

Taking over Jean-Come works. Some known issues :

---

---

1 - PH :
Tested on qualif-ofsep with studyId = 20:
First time I executed the request on swagger, nothing happened
Second time, I got this error:

Whenever I refresh the page (ctrl + F5), I see the job starting over, deleting niftis then stop as the error above is met

Also tested with studyId = 21 (only one query this time) and the same thing happened

---

---

2 - PH :
Tested on ofsep-qualif:
Sent the request from swagger to delete niftis
It took something like 4 minutes until I saw the job start, but I sent that request for a study with more than 1Tb of niftis
Once the job finished, I wanted to check if the storage volume was updated but I kept having issues in ms-studies:

2024-11-27_14:28:53.158 WARN [phdauvergne(203)] c.z.hikari.pool.ProxyConnection - HikariPool-1 - Connection com.mysql.cj.jdbc.ConnectionImpl@1c9975a8 marked as broken because of SQLSTATE(08S01), ErrorCode(0)
com.mysql.cj.jdbc.exceptions.CommunicationsException: Communications link failure

The last packet successfully received from the server was 766 milliseconds ago. The last packet sent successfully to the server was 766 milliseconds ago.
at com.mysql.cj.jdbc.exceptions.SQLError.createCommunicationsException(SQLError.java:174)
at com.mysql.cj.jdbc.exceptions.SQLExceptionsMapping.translateException(SQLExceptionsMapping.java:64)
...
at org.shanoir.ng.study.service.StudyServiceImpl.findById(StudyServiceImpl.java:166)

Caused by: com.mysql.cj.exceptions.CJCommunicationsException: Communications link failure

The last packet successfully received from the server was 766 milliseconds ago. The last packet sent successfully to the server was 766 milliseconds ago.
at java.base/jdk.internal.reflect.DirectConstructorHandleAccessor.newInstance(Unknown Source)
at java.base/java.lang.reflect.Constructor.newInstanceWithCaller(Unknown Source)
at java.base/java.lang.reflect.Constructor.newInstance(Unknown Source)
at com.mysql.cj.exceptions.ExceptionFactory.createException(ExceptionFactory.java:61)
...
at com.mysql.cj.protocol.a.NativeProtocol.sendCommand(NativeProtocol.java:701)
at com.mysql.cj.protocol.a.NativeProtocol.sendQueryPacket(NativeProtocol.java:1050)
at com.mysql.cj.NativeSession.execSQL(NativeSession.java:660)
at com.mysql.cj.jdbc.ClientPreparedStatement.executeInternal(ClientPreparedStatement.java:889)
... 249 common frames omitted
Caused by: java.io.EOFException: Can not read response from server. Expected to read 4 bytes, read 0 bytes before connection was unexpectedly lost.
at com.mysql.cj.protocol.FullReadInputStream.readFully(FullReadInputStream.java:67)
at com.mysql.cj.protocol.a.SimplePacketReader.readHeaderLocal(SimplePacketReader.java:81)
at com.mysql.cj.protocol.a.SimplePacketReader.readHeader(SimplePacketReader.java:63)
at com.mysql.cj.protocol.a.SimplePacketReader.readHeader(SimplePacketReader.java:45)
at com.mysql.cj.protocol.a.TimeTrackingPacketReader.readHeader(TimeTrackingPacketReader.java:52)
at com.mysql.cj.protocol.a.TimeTrackingPacketReader.readHeader(TimeTrackingPacketReader.java:41)
at com.mysql.cj.protocol.a.MultiPacketReader.readHeader(MultiPacketReader.java:54)
at com.mysql.cj.protocol.a.MultiPacketReader.readHeader(MultiPacketReader.java:44)
at com.mysql.cj.protocol.a.NativeProtocol.readMessage(NativeProtocol.java:576)
... 254 common frames omitted
2024-11-27_14:28:53.292 ERROR [phdauvergne(203)] o.h.e.jdbc.spi.SqlExceptionHelper - Communications link failure

Then in ms-users:

org.springframework.transaction.CannotCreateTransactionException: Could not open JPA EntityManager for transaction
at org.springframework.orm.jpa.JpaTransactionManager.doBegin(JpaTransactionManager.java:466)
at org.springframework.transaction.support.AbstractPlatformTransactionManager.startTransaction(AbstractPlatformTransactionManager.java:531)
...
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:184)
at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:97)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:184)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:220)
at jdk.proxy2/jdk.proxy2.$Proxy184.findAll(Unknown Source)
at org.shanoir.ng.accessrequest.controller.AccessRequestApiController.findAllByAdminId(AccessRequestApiController.java:138)

And:

Caused by: org.hibernate.exception.JDBCConnectionException: Unable to acquire JDBC Connection [HikariPool-1 - Connection is not available, request timed out after 30000ms.] [n/a]
at org.hibernate.exception.internal.SQLExceptionTypeDelegate.convert(SQLExceptionTypeDelegate.java:51)
at org.hibernate.exception.internal.StandardSQLExceptionConverter.convert(StandardSQLExceptionConverter.java:58)
at org.hibernate.engine.jdbc.spi.SqlExceptionHelper.convert(SqlExceptionHelper.java:108)
at org.hibernate.engine.jdbc.spi.SqlExceptionHelper.convert(SqlExceptionHelper.java:94)
at org.hibernate.resource.jdbc.internal.LogicalConnectionManagedImpl.acquireConnectionIfNeeded(LogicalConnectionManagedImpl.java:116)
at org.hibernate.resource.jdbc.internal.LogicalConnectionManagedImpl.getPhysicalConnection(LogicalConnectionManagedImpl.java:143)
at org.springframework.orm.jpa.vendor.HibernateJpaDialect.beginTransaction(HibernateJpaDialect.java:164)
at org.springframework.orm.jpa.JpaTransactionManager.doBegin(JpaTransactionManager.java:420)
... 119 common frames omitted
Caused by: java.sql.SQLTransientConnectionException: HikariPool-1 - Connection is not available, request timed out after 30000ms.
at com.zaxxer.hikari.pool.HikariPool.createTimeoutException(HikariPool.java:696)
at com.zaxxer.hikari.pool.HikariPool.getConnection(HikariPool.java:181)
at com.zaxxer.hikari.pool.HikariPool.getConnection(HikariPool.java:146)
at com.zaxxer.hikari.HikariDataSource.getConnection(HikariDataSource.java:128)
at org.hibernate.engine.jdbc.connections.internal.DatasourceConnectionProviderImpl.getConnection(DatasourceConnectionProviderImpl.java:122)
at org.hibernate.internal.NonContextualJdbcConnectionAccess.obtainConnection(NonContextualJdbcConnectionAccess.java:46)
at org.hibernate.resource.jdbc.internal.LogicalConnectionManagedImpl.acquireConnectionIfNeeded(LogicalConnectionManagedImpl.java:113)
... 122 common frames omitted
Caused by: com.mysql.cj.jdbc.exceptions.CommunicationsException: Communications link failure

The last packet sent successfully to the server was 0 milliseconds ago. The driver has not received any packets from the server.
at com.mysql.cj.jdbc.exceptions.SQLError.createCommunicationsException(SQLError.java:174)
at com.mysql.cj.jdbc.exceptions.SQLExceptionsMapping.translateException(SQLExceptionsMapping.java:64)
at com.mysql.cj.jdbc.ConnectionImpl.createNewIO(ConnectionImpl.java:815)

I then redeployed, tested on studies with less niftis and it worked nicely.
Then I tried again on the study with 1Tb of nifti, the job stopped at 79% and ms-users threw:

I guess this is because of the duration of the request or something along those lines

---

---

Splitting transaction, currently testing

[DuckflipXYZ]

Tested in qualif with 3 study in a raw (ids : 20, 21, 32, 32 has 2T of nifti), no issue of server memory.
Please keep in mind that I had to talk to Anthony about the server memory issue, after what he gave more resources to the database container (or something like, I'm not sure), it gonna be required on prod too (probably).

Close this PR opened by Jean-Come when merging : #2508