Skip to content

Commit

Permalink
[chore] Tokenizer improvements (#84)
Browse files Browse the repository at this point in the history
* [chore] Improved tokenizer logic to not discard untokenized receipts

* [chore] Updated unit tests

* [chore] Improved exception handling

* [chore] Improved exception handling

* [chore] Improved java doc

* [chore] Fixed Tokenizer response

* [chore] Implemented tokenizer service with retry

* [chore] Added retry config in env values

* [chore] Improved code to remove duplications

* [chore] Removed unused import

* [chore] Removed static declaration

* [chore] Changed retry env values
  • Loading branch information
svariant authored Nov 8, 2023
1 parent 1a8afa1 commit 6b6a0e9
Show file tree
Hide file tree
Showing 18 changed files with 658 additions and 94 deletions.
38 changes: 21 additions & 17 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -65,23 +65,27 @@ On terminal type:
then replace env variables with correct values
(if there is NO default value, the variable HAS to be defined)

| VARIABLE | USAGE | DEFAULT VALUE |
|---------------------------------------|----------------------------------------------------------------|:------------------------------------------------------:|
| `RECEIPT_QUEUE_CONN_STRING` | Connection string to the Receipt Queue | |
| `RECEIPT_QUEUE_TOPIC` | Topic name of the Receipt Queue | |
| `RECEIPT_QUEUE_DELAY` | Delay, in seconds, the visibility of the messages in the queue | "1" |
| `COSMOS_BIZ_EVENT_CONN_STRING` | Connection string to the BizEvent CosmosDB | |
| `COSMOS_RECEIPTS_CONN_STRING` | Connection string to the Receipt CosmosDB | |
| `COSMOS_RECEIPT_SERVICE_ENDPOINT` | Endpoint to the Receipt CosmosDB | |
| `COSMOS_RECEIPT_KEY` | Key to the Receipt CosmosDB | |
| `COSMOS_RECEIPT_DB_NAME` | Database name of the Receipt database in CosmosDB | |
| `COSMOS_RECEIPT_CONTAINER_NAME` | Container name of the Receipt container in CosmosDB | |
| `PDV_TOKENIZER_BASE_PATH` | PDV Tokenizer API base path | "https://api.uat.tokenizer.pdv.pagopa.it/tokenizer/v1" |
| `PDV_TOKENIZER_SEARCH_TOKEN_ENDPOINT` | PDV Tokenizer API search token endpoint | "/tokens/search" |
| `PDV_TOKENIZER_FIND_PII_ENDPOINT` | PDV Tokenizer API find pii endpoint | "/tokens/%s/pii" |
| `PDV_TOKENIZER_CREATE_TOKEN_ENDPOINT` | PDV Tokenizer API create token endpoint | "/tokens" |
| `PDV_TOKENIZER_SUBSCRIPTION_KEY` | API azure ocp apim subscription key | |
| `TOKENIZER_APIM_HEADER_KEY` | Tokenizer APIM header key | x-api-key |
| VARIABLE | USAGE | DEFAULT VALUE |
|---------------------------------------|-----------------------------------------------------------------------------------|:------------------------------------------------------:|
| `RECEIPT_QUEUE_CONN_STRING` | Connection string to the Receipt Queue | |
| `RECEIPT_QUEUE_TOPIC` | Topic name of the Receipt Queue | |
| `RECEIPT_QUEUE_DELAY` | Delay, in seconds, the visibility of the messages in the queue | "1" |
| `COSMOS_BIZ_EVENT_CONN_STRING` | Connection string to the BizEvent CosmosDB | |
| `COSMOS_RECEIPTS_CONN_STRING` | Connection string to the Receipt CosmosDB | |
| `COSMOS_RECEIPT_SERVICE_ENDPOINT` | Endpoint to the Receipt CosmosDB | |
| `COSMOS_RECEIPT_KEY` | Key to the Receipt CosmosDB | |
| `COSMOS_RECEIPT_DB_NAME` | Database name of the Receipt database in CosmosDB | |
| `COSMOS_RECEIPT_CONTAINER_NAME` | Container name of the Receipt container in CosmosDB | |
| `PDV_TOKENIZER_BASE_PATH` | PDV Tokenizer API base path | "https://api.uat.tokenizer.pdv.pagopa.it/tokenizer/v1" |
| `PDV_TOKENIZER_SEARCH_TOKEN_ENDPOINT` | PDV Tokenizer API search token endpoint | "/tokens/search" |
| `PDV_TOKENIZER_FIND_PII_ENDPOINT` | PDV Tokenizer API find pii endpoint | "/tokens/%s/pii" |
| `PDV_TOKENIZER_CREATE_TOKEN_ENDPOINT` | PDV Tokenizer API create token endpoint | "/tokens" |
| `PDV_TOKENIZER_SUBSCRIPTION_KEY` | API azure ocp apim subscription key | |
| `PDV_TOKENIZER_INITIAL_INTERVAL` | PDV Tokenizer initial interval for retry a request that fail with 429 status code | 200 |
| `PDV_TOKENIZER_MULTIPLIER` | PDV Tokenizer interval multiplier for subsequent request retry | 2.0 |
| `PDV_TOKENIZER_RANDOMIZATION_FACTOR` | PDV Tokenizer randomization factor for interval retry calculation | 0.6 |
| `PDV_TOKENIZER_MAX_RETRIES` | PDV Tokenizer max request retry | 3 |
| `TOKENIZER_APIM_HEADER_KEY` | Tokenizer APIM header key | x-api-key |

> to doc details about AZ fn config
> see [here](https://stackoverflow.com/questions/62669672/azure-functions-what-is-the-purpose-of-having-host-json-and-local-settings-jso)
Expand Down
4 changes: 4 additions & 0 deletions helm/values-dev.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -86,6 +86,10 @@ microservice-chart:
COSMOS_RECEIPT_DB_NAME: "db"
COSMOS_RECEIPT_CONTAINER_NAME: "receipts"
PDV_TOKENIZER_BASE_PATH: "https://api.uat.tokenizer.pdv.pagopa.it/tokenizer/v1"
PDV_TOKENIZER_INITIAL_INTERVAL: "200"
PDV_TOKENIZER_MULTIPLIER: "2.0"
PDV_TOKENIZER_RANDOMIZATION_FACTOR: "0.6"
PDV_TOKENIZER_MAX_RETRIES: "3"
ENABLE_ECS_CONSOLE: "true"
CONSOLE_LOG_THRESHOLD: "DEBUG"
CONSOLE_LOG_PATTERN: "%d{HH:mm:ss.SSS}[%thread]%-5level%logger{36}-%msg%n"
Expand Down
4 changes: 4 additions & 0 deletions helm/values-prod.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -86,6 +86,10 @@ microservice-chart:
COSMOS_RECEIPT_DB_NAME: "db"
COSMOS_RECEIPT_CONTAINER_NAME: "receipts"
PDV_TOKENIZER_BASE_PATH: "https://api.tokenizer.pdv.pagopa.it"
PDV_TOKENIZER_INITIAL_INTERVAL: "200"
PDV_TOKENIZER_MULTIPLIER: "2.0"
PDV_TOKENIZER_RANDOMIZATION_FACTOR: "0.6"
PDV_TOKENIZER_MAX_RETRIES: "3"
ENABLE_ECS_CONSOLE: "true"
CONSOLE_LOG_THRESHOLD: "DEBUG"
CONSOLE_LOG_PATTERN: "%d{HH:mm:ss.SSS}[%thread]%-5level%logger{36}-%msg%n"
Expand Down
4 changes: 4 additions & 0 deletions helm/values-uat.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -86,6 +86,10 @@ microservice-chart:
COSMOS_RECEIPT_DB_NAME: "db"
COSMOS_RECEIPT_CONTAINER_NAME: "receipts"
PDV_TOKENIZER_BASE_PATH: "https://api.uat.tokenizer.pdv.pagopa.it/tokenizer/v1"
PDV_TOKENIZER_INITIAL_INTERVAL: "200"
PDV_TOKENIZER_MULTIPLIER: "2.0"
PDV_TOKENIZER_RANDOMIZATION_FACTOR: "0.6"
PDV_TOKENIZER_MAX_RETRIES: "3"
ENABLE_ECS_CONSOLE: "true"
CONSOLE_LOG_THRESHOLD: "DEBUG"
CONSOLE_LOG_PATTERN: "%d{HH:mm:ss.SSS}[%thread]%-5level%logger{36}-%msg%n"
Expand Down
6 changes: 6 additions & 0 deletions pom.xml
Original file line number Diff line number Diff line change
Expand Up @@ -215,6 +215,12 @@
<version>2.6.1</version>
</dependency>

<dependency>
<groupId>io.github.resilience4j</groupId>
<artifactId>resilience4j-retry</artifactId>
<version>2.1.0</version>
</dependency>

</dependencies>

<build>
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -11,9 +11,7 @@
import it.gov.pagopa.receipt.pdf.datastore.entity.receipt.CartItem;
import it.gov.pagopa.receipt.pdf.datastore.entity.receipt.EventData;
import it.gov.pagopa.receipt.pdf.datastore.entity.receipt.Receipt;
import it.gov.pagopa.receipt.pdf.datastore.service.PDVTokenizerService;
import it.gov.pagopa.receipt.pdf.datastore.service.impl.BizEventToReceiptServiceImpl;
import it.gov.pagopa.receipt.pdf.datastore.service.impl.PDVTokenizerServiceImpl;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;

Expand All @@ -28,12 +26,15 @@
public class BizEventToReceipt {

private final Logger logger = LoggerFactory.getLogger(BizEventToReceipt.class);
private final BizEventToReceiptServiceImpl receiptService;

private final PDVTokenizerService pdvTokenizerService;

public BizEventToReceipt(){ this.pdvTokenizerService = new PDVTokenizerServiceImpl();}
public BizEventToReceipt() {
this.receiptService = new BizEventToReceiptServiceImpl();
}

BizEventToReceipt(PDVTokenizerService pdvTokenizerService){ this.pdvTokenizerService = pdvTokenizerService;}
public BizEventToReceipt(BizEventToReceiptServiceImpl receiptService) {
this.receiptService = receiptService;
}

/**
* This function will be invoked when an CosmosDB trigger occurs
Expand Down Expand Up @@ -83,32 +84,31 @@ public void processBizEventToReceipt(
// Retrieve receipt data from biz-event
for (BizEvent bizEvent : items) {

// Discard null biz events || not in status DONE || with totalNotice > 1
// Discard null biz events OR not in status DONE OR with totalNotice > 1
if (isBizEventInvalid(bizEvent, context)) {
discarder++;
continue;
}
try {
BizEventToReceiptServiceImpl service = new BizEventToReceiptServiceImpl();
Receipt receipt = new Receipt();

// Insert biz-event data into receipt
receipt.setEventId(bizEvent.getId());
Receipt receipt = new Receipt();

EventData eventData = new EventData();
if(bizEvent.getPayer() != null && bizEvent.getPayer().getEntityUniqueIdentifierValue() != null){
eventData.setPayerFiscalCode(
pdvTokenizerService.generateTokenForFiscalCode(bizEvent.getPayer().getEntityUniqueIdentifierValue())
);
}
if(bizEvent.getDebtor() != null && bizEvent.getDebtor().getEntityUniqueIdentifierValue() != null){
eventData.setDebtorFiscalCode(
pdvTokenizerService.generateTokenForFiscalCode(bizEvent.getDebtor().getEntityUniqueIdentifierValue())
);
}
// Insert biz-event data into receipt
receipt.setEventId(bizEvent.getId());

EventData eventData = new EventData();

try{
receiptService.tokenizeFiscalCodes(bizEvent, receipt, eventData);
} catch (Exception e){
logger.error("Error tokenizing receipt with bizEventId {}", bizEvent.getId(), e);
itemsDone.add(receipt);
continue;
}

try {
eventData.setTransactionCreationDate(
service.getTransactionCreationDate(bizEvent));
eventData.setAmount( bizEvent.getPaymentInfo() != null ?
receiptService.getTransactionCreationDate(bizEvent));
eventData.setAmount(bizEvent.getPaymentInfo() != null ?
bizEvent.getPaymentInfo().getAmount() : null);

CartItem item = new CartItem();
Expand All @@ -123,7 +123,7 @@ public void processBizEventToReceipt(
context.getFunctionName(), LocalDateTime.now(), bizEvent.getId(), bizEvent.getEventStatus());

// Send biz event as message to queue (to be processed from the other function)
service.handleSendMessageToQueue(bizEvent, receipt);
receiptService.handleSendMessageToQueue(bizEvent, receipt);

// Add receipt to items to be saved on CosmosDB
itemsDone.add(receipt);
Expand Down
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
package it.gov.pagopa.receipt.pdf.datastore.entity.receipt.enumeration;

public enum ReasonErrorCode {
ERROR_QUEUE(902), ERROR_BLOB_STORAGE(901), ERROR_PDF_ENGINE(0);
ERROR_QUEUE(902), ERROR_BLOB_STORAGE(901), ERROR_PDF_ENGINE(0), ERROR_PDV_TOKENIZER(701);

private int code;

Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,19 @@
package it.gov.pagopa.receipt.pdf.datastore.exception;

import lombok.Getter;

/**
* Thrown in case an unexpected error occur when invoking PDV Tokenizer service
*/
@Getter
public class PDVTokenizerUnexpectedException extends RuntimeException {

/**
* Constructs new exception with provided cause
*
* @param cause Exception causing the constructed one
*/
public PDVTokenizerUnexpectedException(Throwable cause) {
super(cause);
}
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,18 @@
package it.gov.pagopa.receipt.pdf.datastore.model.tokenizer;

import lombok.AllArgsConstructor;
import lombok.Builder;
import lombok.Data;
import lombok.NoArgsConstructor;

/**
* Model class for the error response of the PDV Tokenizer for status 403, 404, 429
*/
@Data
@Builder
@AllArgsConstructor
@NoArgsConstructor
public class ErrorMessage {

private String message;
}
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@
import java.util.List;

/**
* Model class for the error response of the PDV Tokenizer
* Model class for the error response of the PDV Tokenizer for status 400, 500
*/
@Data
@Builder
Expand Down
Original file line number Diff line number Diff line change
@@ -1,13 +1,37 @@
package it.gov.pagopa.receipt.pdf.datastore.service;

import com.fasterxml.jackson.core.JsonProcessingException;
import it.gov.pagopa.receipt.pdf.datastore.entity.event.BizEvent;
import it.gov.pagopa.receipt.pdf.datastore.entity.receipt.EventData;
import it.gov.pagopa.receipt.pdf.datastore.entity.receipt.Receipt;
import it.gov.pagopa.receipt.pdf.datastore.exception.PDVTokenizerException;

public interface BizEventToReceiptService {

/**
* Handles sending biz-events as message to queue and updates receipt's status
*
* @param bizEvent Biz-event from CosmosDB
* @param receipt Receipt to update
*/
void handleSendMessageToQueue(BizEvent bizEvent, Receipt receipt);

void handleError(Receipt receipt);

/**
* Retrieve conditionally the transaction creation date from biz-event
*
* @param bizEvent Biz-event from CosmosDB
* @return transaction date
*/
String getTransactionCreationDate(BizEvent bizEvent);

/**
* Calls PDVTokenizerService to tokenize the fiscal codes for both Debtor & Payer (if present)
*
* @param bizEvent BizEvent where fiscalCodes are stored
* @param receipt Receipt to update in case of errors
* @param eventData Event data to update with tokenized fiscalCodes
* @throws JsonProcessingException if an error occur when parsing input or output
* @throws PDVTokenizerException if an error occur when invoking the PDV Tokenizer
*/
void tokenizeFiscalCodes(BizEvent bizEvent, Receipt receipt, EventData eventData) throws JsonProcessingException, PDVTokenizerException;
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,40 @@
package it.gov.pagopa.receipt.pdf.datastore.service;

import com.fasterxml.jackson.core.JsonProcessingException;
import it.gov.pagopa.receipt.pdf.datastore.exception.PDVTokenizerException;

/**
* Service that wrap the {@link PDVTokenizerService} for adding retry logic for tokenizer responses with 429 status code
*/
public interface PDVTokenizerServiceRetryWrapper {

/**
* Call {@link PDVTokenizerService#getToken(String)} with retry on failure
*
* @param fiscalCode the fiscal code
* @return the token associated to the fiscal code
* @throws JsonProcessingException if an error occur when parsing input or output
* @throws PDVTokenizerException if an error occur when invoking the PDV Tokenizer
*/
String getTokenWithRetry(String fiscalCode) throws JsonProcessingException, PDVTokenizerException;

/**
* Call {@link PDVTokenizerService#getFiscalCode(String)} with retry on failure
*
* @param token the token
* @return the fiscal code associated to the provided token
* @throws JsonProcessingException if an error occur when parsing input or output
* @throws PDVTokenizerException if an error occur when invoking the PDV Tokenizer
*/
String getFiscalCodeWithRetry(String token) throws PDVTokenizerException, JsonProcessingException;

/**
* Call {@link PDVTokenizerService#generateTokenForFiscalCode(String)} with retry on failure
*
* @param fiscalCode the fiscal code
* @return the generated token
* @throws JsonProcessingException if an error occur when parsing input or output
* @throws PDVTokenizerException if an error occur when invoking the PDV Tokenizer
*/
String generateTokenForFiscalCodeWithRetry(String fiscalCode) throws PDVTokenizerException, JsonProcessingException;
}
Loading

0 comments on commit 6b6a0e9

Please sign in to comment.