Lahi Integration is facing repeated timeouts while fetching data from Avni server

[himeshr]

Motivation

Lahi Integration service running in Staging-integration environment is down for whole of 18 Dec 2023.
Its repeatedly failing with socketTimeout error, due to delay in response from Avni-server.

Acceptance Criteria

Lahi Integration service should sync entities from Glific to Avni (Lahiint organisation) successfully

Error Logs

2023-12-19 07:30:23,254 ERROR SpringTaskScheduler-5 [avni.client.AvniHttpClient] URI: https://app.avniproject.org/api/subjects?lastModifiedDateTime=1900-01-01T00:00:00.000Z&concepts=%7B%22Student%20contact%20number%22:%229873249733%22%7D&subjectType=Student, Errored Request body: null
2023-12-19 07:30:23,260 INFO  SpringTaskScheduler-5 [lahi.service.LahiIntegrationDataService] Updating integrating_entity_status with Mon Dec 18 12:48:24 UTC 2023 date
2023-12-19 07:30:23,260 INFO  SpringTaskScheduler-5 [avni_integration_service.util.HealthCheckService] Health check ping for slug:lahi status: FAILURE
2023-12-19 07:30:23,394 ERROR SpringTaskScheduler-5 [lahi.job.AvniLahiMainJob] Failed
org.springframework.web.client.ResourceAccessException: I/O error on GET request for "https://app.avniproject.org/api/subjects": Read timed out; nested exception is java.net.SocketTimeoutException: Read timed out
        at org.springframework.web.client.RestTemplate.doExecute(RestTemplate.java:785)
        at org.springframework.web.client.RestTemplate.execute(RestTemplate.java:751)
        at org.springframework.web.client.RestTemplate.exchange(RestTemplate.java:621)
        at org.avni_integration_service.avni.client.AvniHttpClient.getResponseEntity(AvniHttpClient.java:53)
        at org.avni_integration_service.avni.client.AvniHttpClient.get(AvniHttpClient.java:47)
        at org.avni_integration_service.avni.repository.AvniSubjectRepository.getSubjects(AvniSubjectRepository.java:50)
        at org.avni_integration_service.avni.repository.AvniSubjectRepository.getSubjects(AvniSubjectRepository.java:39)
        at org.avni_integration_service.lahi.repository.AvniStudentRepository.findMatchingStudents(AvniStudentRepository.java:37)
        at org.avni_integration_service.lahi.service.AvniStudentService.saveStudent(AvniStudentService.java:22)
        at org.avni_integration_service.lahi.worker.StudentWorker.processStudents(StudentWorker.java:47)
        at org.avni_integration_service.lahi.job.AvniLahiMainJob.execute(AvniLahiMainJob.java:49)
        at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54)
        at org.springframework.scheduling.concurrent.ReschedulingRunnable.run(ReschedulingRunnable.java:95)
        at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
        at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
        at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
        at java.base/java.lang.Thread.run(Thread.java:840)
Caused by: java.net.SocketTimeoutException: Read timed out
        at java.base/sun.nio.ch.NioSocketImpl.timedRead(NioSocketImpl.java:288)
        at java.base/sun.nio.ch.NioSocketImpl.implRead(NioSocketImpl.java:314)
        at java.base/sun.nio.ch.NioSocketImpl.read(NioSocketImpl.java:355)
        at java.base/sun.nio.ch.NioSocketImpl$1.read(NioSocketImpl.java:808)
        at java.base/java.net.Socket$SocketInputStream.read(Socket.java:966)
        at java.base/sun.security.ssl.SSLSocketInputRecord.read(SSLSocketInputRecord.java:484)
        at java.base/sun.security.ssl.SSLSocketInputRecord.readHeader(SSLSocketInputRecord.java:478)
        at java.base/sun.security.ssl.SSLSocketInputRecord.bytesInCompletePacket(SSLSocketInputRecord.java:70)
        at java.base/sun.security.ssl.SSLSocketImpl.readApplicationRecord(SSLSocketImpl.java:1465)
        at java.base/sun.security.ssl.SSLSocketImpl$AppInputStream.read(SSLSocketImpl.java:1069)
        at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:137)
        at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:153)
        at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:280)
        at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138)
        at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56)
        at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:259)
        at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:163)
        at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:157)
        at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:273)
        at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:125)
        at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272)
        at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186)

[petmongrels]

We should also make the rest template definition expilicit instead of @primary

[petmongrels]

after code review, need to deploy

[himeshr]

Deployed to Staging environment.