Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(drive): upload and download file #3872

Open
wants to merge 1 commit into
base: main
Choose a base branch
from
Open

feat(drive): upload and download file #3872

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
57 changes: 56 additions & 1 deletion connectors/google/google-drive/element-templates/google-drive-outbound-connector.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@
"keywords" : [ "create file", "create file from template", "create folder" ]
},
"documentationRef" : "https://docs.camunda.io/docs/components/connectors/out-of-the-box-connectors/googledrive/",
"version" : 3,
"version" : 4,
"category" : {
"id" : "connectors",
"name" : "Connectors"
Expand Down Expand Up @@ -163,6 +163,12 @@
}, {
"name" : "Create file from template",
"value" : "file"
}, {
"name" : "Upload file",
"value" : "upload"
}, {
"name" : "Download file",
"value" : "download"
} ]
}, {
"id" : "resource.name",
Expand All @@ -177,6 +183,11 @@
"name" : "resource.name",
"type" : "zeebe:input"
},
"condition" : {
"property" : "resource.type",
"oneOf" : [ "folder", "file" ],
"type" : "simple"
},
"type" : "String"
}, {
"id" : "resource.parent",
Expand All @@ -189,6 +200,11 @@
"name" : "resource.parent",
"type" : "zeebe:input"
},
"condition" : {
"property" : "resource.type",
"oneOf" : [ "folder", "file", "upload" ],
"type" : "simple"
},
"type" : "String"
}, {
"id" : "resource.additionalGoogleDriveProperties",
Expand Down Expand Up @@ -241,6 +257,45 @@
"type" : "simple"
},
"type" : "String"
}, {
"id" : "resource.downloadData.fileId",
"label" : "File ID",
"optional" : false,
"constraints" : {
"notEmpty" : true
},
"feel" : "optional",
"group" : "operationDetails",
"binding" : {
"name" : "resource.downloadData.fileId",
"type" : "zeebe:input"
},
"condition" : {
"property" : "resource.type",
"equals" : "download",
"type" : "simple"
},
"type" : "String"
}, {
"id" : "resource.uploadData.document",
"label" : "Document",
"description" : "Upload camunda document, <a href=\"https://docs.camunda.io/docs/apis-tools/camunda-api-rest/specifications/upload-document-alpha/\">see documentation</a>",
"optional" : false,
"constraints" : {
"notEmpty" : true
},
"feel" : "required",
"group" : "operationDetails",
"binding" : {
"name" : "resource.uploadData.document",
"type" : "zeebe:input"
},
"condition" : {
"property" : "resource.type",
"equals" : "upload",
"type" : "simple"
},
"type" : "String"
}, {
"id" : "resultVariable",
"label" : "Result variable",
Expand Down
57 changes: 56 additions & 1 deletion .../google/google-drive/element-templates/hybrid/google-drive-outbound-connector-hybrid.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@
"keywords" : [ "create file", "create file from template", "create folder" ]
},
"documentationRef" : "https://docs.camunda.io/docs/components/connectors/out-of-the-box-connectors/googledrive/",
"version" : 3,
"version" : 4,
"category" : {
"id" : "connectors",
"name" : "Connectors"
Expand Down Expand Up @@ -168,6 +168,12 @@
}, {
"name" : "Create file from template",
"value" : "file"
}, {
"name" : "Upload file",
"value" : "upload"
}, {
"name" : "Download file",
"value" : "download"
} ]
}, {
"id" : "resource.name",
Expand All @@ -182,6 +188,11 @@
"name" : "resource.name",
"type" : "zeebe:input"
},
"condition" : {
"property" : "resource.type",
"oneOf" : [ "folder", "file" ],
"type" : "simple"
},
"type" : "String"
}, {
"id" : "resource.parent",
Expand All @@ -194,6 +205,11 @@
"name" : "resource.parent",
"type" : "zeebe:input"
},
"condition" : {
"property" : "resource.type",
"oneOf" : [ "folder", "file", "upload" ],
"type" : "simple"
},
"type" : "String"
}, {
"id" : "resource.additionalGoogleDriveProperties",
Expand Down Expand Up @@ -246,6 +262,45 @@
"type" : "simple"
},
"type" : "String"
}, {
"id" : "resource.downloadData.fileId",
"label" : "File ID",
"optional" : false,
"constraints" : {
"notEmpty" : true
},
"feel" : "optional",
"group" : "operationDetails",
"binding" : {
"name" : "resource.downloadData.fileId",
"type" : "zeebe:input"
},
"condition" : {
"property" : "resource.type",
"equals" : "download",
"type" : "simple"
},
"type" : "String"
}, {
"id" : "resource.uploadData.document",
"label" : "Document",
"description" : "Upload camunda document, <a href=\"https://docs.camunda.io/docs/apis-tools/camunda-api-rest/specifications/upload-document-alpha/\">see documentation</a>",
"optional" : false,
"constraints" : {
"notEmpty" : true
},
"feel" : "required",
"group" : "operationDetails",
"binding" : {
"name" : "resource.uploadData.document",
"type" : "zeebe:input"
},
"condition" : {
"property" : "resource.type",
"equals" : "upload",
"type" : "simple"
},
"type" : "String"
}, {
"id" : "resultVariable",
"label" : "Result variable",
Expand Down
8 changes: 8 additions & 0 deletions ...tors/google/google-drive/src/main/java/io/camunda/connector/gdrive/GoogleDriveClient.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -51,4 +51,12 @@ public BatchUpdateDocumentResponse updateDocument(
throw new RuntimeException(e);
}
}

public Drive getDriveService() {
return driveService;
}

public Docs getDocsService() {
return docsService;
}
}
11 changes: 8 additions & 3 deletions ...rs/google/google-drive/src/main/java/io/camunda/connector/gdrive/GoogleDriveFunction.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@
import io.camunda.connector.api.annotation.OutboundConnector;
import io.camunda.connector.api.outbound.OutboundConnectorContext;
import io.camunda.connector.api.outbound.OutboundConnectorFunction;
import io.camunda.connector.gdrive.model.GoogleDriveResult;
import io.camunda.connector.gdrive.mapper.DocumentMapper;
import io.camunda.connector.gdrive.model.request.GoogleDriveRequest;
import io.camunda.connector.gdrive.supliers.GoogleDocsServiceSupplier;
import io.camunda.connector.generator.java.annotation.ElementTemplate;
Expand All @@ -25,7 +25,7 @@
@ElementTemplate.Metadata(
keywords = {"create file", "create file from template", "create folder"}),
inputDataClass = GoogleDriveRequest.class,
version = 3,
version = 4,
propertyGroups = {
@ElementTemplate.PropertyGroup(id = "authentication", label = "Authentication"),
@ElementTemplate.PropertyGroup(id = "operation", label = "Select operation"),
Expand Down Expand Up @@ -53,16 +53,21 @@ public GoogleDriveFunction(final GoogleDriveService service) {

@Override
public Object execute(final OutboundConnectorContext context) {

var request = context.bindVariables(GoogleDriveRequest.class);
service.setDocumentMapper(new DocumentMapper(context));

return executeConnector(request);
}

private GoogleDriveResult executeConnector(final GoogleDriveRequest request) {
private Object executeConnector(final GoogleDriveRequest request) {
LOGGER.debug("Executing my connector with request {}", request);

GoogleDriveClient drive =
new GoogleDriveClient(
GoogleDriveServiceSupplier.createDriveClientInstance(request.getAuthentication()),
GoogleDocsServiceSupplier.createDocsClientInstance(request.getAuthentication()));

return service.execute(drive, request.getResource());
}
}
80 changes: 78 additions & 2 deletions ...ors/google/google-drive/src/main/java/io/camunda/connector/gdrive/GoogleDriveService.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,37 +6,58 @@
*/
package io.camunda.connector.gdrive;

import com.google.api.client.http.ByteArrayContent;
import com.google.api.client.json.JsonParser;
import com.google.api.client.json.gson.GsonFactory;
import com.google.api.services.docs.v1.model.BatchUpdateDocumentResponse;
import com.google.api.services.docs.v1.model.Request;
import com.google.api.services.drive.Drive;
import com.google.api.services.drive.model.File;
import com.google.common.reflect.TypeToken;
import io.camunda.connector.gdrive.mapper.DocumentMapper;
import io.camunda.connector.gdrive.model.GoogleDriveResult;
import io.camunda.connector.gdrive.model.MimeTypeUrl;
import io.camunda.connector.gdrive.model.request.Resource;
import io.camunda.connector.gdrive.model.request.Template;
import io.camunda.connector.gdrive.model.request.Type;
import io.camunda.connector.gdrive.model.request.Variables;
import io.camunda.document.Document;
import io.camunda.google.supplier.GsonComponentSupplier;
import java.io.ByteArrayOutputStream;
import java.io.IOException;
import java.util.ArrayList;
import java.util.List;
import java.util.Optional;
import org.apache.commons.lang3.StringUtils;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;

public class GoogleDriveService {
private static final Logger LOGGER = LoggerFactory.getLogger(GoogleDriveService.class);
/**
* This constant is used to select different types of uploading (multipart or resumable). <a
* href="https://developers.google.com/drive/api/guides/manage-uploads"></a> <a
* href="https://developers.google.com/api-client-library/java/google-api-java-client/media-upload#implementation"></a>
*/
public static final long MAX_DIRECT_UPLOAD_FILE_SIZE_BYTES = 5_242_880L; // 5MB

public static final String IO_EXCEPTION_MESSAGE = "IO exception while %s a file";
private static final Logger LOGGER = LoggerFactory.getLogger(GoogleDriveService.class);
private final GsonFactory gsonFactory = GsonComponentSupplier.gsonFactoryInstance();

private DocumentMapper documentMapper;

public GoogleDriveService(DocumentMapper documentMapper) {
this.documentMapper = documentMapper;
}

public GoogleDriveService() {}
johnBgood marked this conversation as resolved.
Show resolved Hide resolved

public GoogleDriveResult execute(final GoogleDriveClient client, final Resource resource) {
public Object execute(final GoogleDriveClient client, final Resource resource) {
return switch (resource.type()) {
case FOLDER -> createFolder(client, resource);
case FILE -> createFile(client, resource);
case UPLOAD -> uploadFile(client, resource);
case DOWNLOAD -> downloadFile(client, resource);
};
}

Expand Down Expand Up @@ -116,4 +137,59 @@ private void updateWithRequests(
metaData.getMimeType());
}
}

private GoogleDriveResult uploadFile(final GoogleDriveClient client, final Resource resource) {
try {
var document = resource.uploadData().document();
File fileMetaData = prepareFileMetaData(document, resource.parent());

var content =
new ByteArrayContent(document.metadata().getContentType(), document.asByteArray());

Drive drive = client.getDriveService();
Drive.Files.Create createRequest = drive.files().create(fileMetaData, content);

if (document.metadata().getSize() > MAX_DIRECT_UPLOAD_FILE_SIZE_BYTES) {
createRequest.getMediaHttpUploader().setProgressListener(new LoggerProgressListener());
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't get this part, we have a MAX SIZE, but we can still upload the file if the size is exceeded? If so, we should rename the variable because it might be confusing

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Basically, this is done to pay tribute to the documentation. Just to show the difference between the two types uploading (multipart and resumable). But from a code point of view, everything except the listener is identical. https://developers.google.com/drive/api/guides/manage-uploads
https://developers.google.com/api-client-library/java/google-api-java-client/media-upload#implementation

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok I see, then maybe just a javadoc comment on MAX_DIRECT_UPLOAD_FILE_SIZE_BYTES explaining that beyond this size the upload is handled differently etc should be enough

}

File file = createRequest.execute();
return new GoogleDriveResult(file.getId(), MimeTypeUrl.getFileUrl(file.getId()));
} catch (IOException e) {
String msg = String.format(IO_EXCEPTION_MESSAGE, "uploading");
LOGGER.warn(msg, e);
throw new RuntimeException(msg, e);
}
}

private Document downloadFile(final GoogleDriveClient client, final Resource resource) {
Drive drive = client.getDriveService();
try {
String fileId = resource.downloadData().fileId();
File fileMetaData = drive.files().get(fileId).execute();
try (ByteArrayOutputStream outputStream = new ByteArrayOutputStream()) {
drive.files().get(fileId).executeMediaAndDownloadTo(outputStream);
return documentMapper.mapToDocument(outputStream.toByteArray(), fileMetaData);
}
} catch (IOException e) {
String msg = String.format(IO_EXCEPTION_MESSAGE, "downloading");
LOGGER.warn(msg);
throw new RuntimeException(msg, e);
}
}

private File prepareFileMetaData(Document document, String parent) {
File fileMetaData = new File();
fileMetaData.setName(document.metadata().getFileName());
Oleksiivanov marked this conversation as resolved.
Show resolved Hide resolved

Optional.ofNullable(parent)
.filter(StringUtils::isNotBlank)
.ifPresent(folder -> fileMetaData.setParents(List.of(folder)));

return fileMetaData;
}

public void setDocumentMapper(DocumentMapper documentMapper) {
this.documentMapper = documentMapper;
}
}
Loading