Bug : Batch write using spark report items greater than 400KB

[maxmithun]

This method is defined as static so that mulitple tasks running in an executor will update the itemSize sharing the same variable. which incorrectly update and exceeds the size

static int getItemSizeBytes(Map<String, AttributeValue> item) {
   try {
     int itemSize = 0;
     for (Entry<String, AttributeValue> entry : item.entrySet()) {
       itemSize += entry.getKey() != null ? entry.getKey().getBytes(CHARACTER_ENCODING).length : 0;
       itemSize += entry.getValue() != null ? getAttributeSizeBytes(entry.getValue()) : 0;
     }
     return itemSize;
   } catch (UnsupportedEncodingException e) {
     throw new RuntimeException(e);
   }
 }

emr-dynamodb-connector/emr-dynamodb-hadoop/src/main/java/org/apache/hadoop/dynamodb/DynamoDBUtil.java

Line 141 in b9e5ae3

static int getItemSizeBytes(Map<String, AttributeValue> item) {

Called from method putBatch

int itemSizeBytes = DynamoDBUtil.getItemSizeBytes(item);

https://github.com/awslabs/emr-dynamodb-connector/blob/master/emr-dynamodb-hadoop/src/main/java/org/apache/hadoop/dynamodb/DynamoDBClient.java#L204

.