Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Test last branches merged #146

Merged
merged 44 commits into from
Dec 31, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
44 commits
Select commit Hold shift + click to select a range
ca320b5
w.i.p. RateLimit for Synchronizations
WilcoLouwerse Nov 26, 2024
34b2497
Merge branch 'feature/CONNECTOR-144/rate-limits' into feature/CONNECT…
WilcoLouwerse Nov 26, 2024
712c07d
Merge branch 'development' into feature/CONNECTOR-144/rate-limits-sync
WilcoLouwerse Nov 26, 2024
05f444c
Updated migration docblock
WilcoLouwerse Nov 26, 2024
8d67010
w.i.p. try to somehow set job nextrun when RateLimit has been reached
WilcoLouwerse Nov 26, 2024
6977d12
Merge branch 'development' into feature/CONNECTOR-144/rate-limits-sync
WilcoLouwerse Nov 26, 2024
927e629
Improvements for checking RateLimit during synchronization
WilcoLouwerse Nov 29, 2024
2eba4fd
Merge branch 'development' into feature/CONNECTOR-144/rate-limits-sync
WilcoLouwerse Nov 29, 2024
1fdc4bb
Merge branch 'development' into feature/CONNECTOR-144/rate-limits-sync
WilcoLouwerse Dec 5, 2024
ece68cd
Merge branch 'development' into feature/CONNECTOR-144/rate-limits-sync
WilcoLouwerse Dec 10, 2024
43f4ac7
Create logs for rate limit and tested setting the job nextrun correctly
WilcoLouwerse Dec 10, 2024
6843ff4
Fix timezone for setting job next run
WilcoLouwerse Dec 10, 2024
71e938e
Fix migrations & docblocks
WilcoLouwerse Dec 10, 2024
61d8007
Check conditions on a earlier moment
bbrands02 Dec 10, 2024
dda98f4
Added conditions in frontend
bbrands02 Dec 10, 2024
301376b
Remove unnecessary double key in unset
bbrands02 Dec 10, 2024
ed6a94d
Added sourceHashMapping column
bbrands02 Dec 10, 2024
cbef4f2
Cleanup and added hash mapping functionality
bbrands02 Dec 10, 2024
e6742bc
Added sourceHashMapping in frontend
bbrands02 Dec 10, 2024
98f3f47
Code cleanup, remove duplicates and use recursion instead of while loops
WilcoLouwerse Dec 10, 2024
07565d7
Small fix for syncaction stacktrace
WilcoLouwerse Dec 10, 2024
0846592
Small fix for syncaction stacktrace
WilcoLouwerse Dec 10, 2024
3a81969
Merge branch 'fix/AXCVDWOF-41/conditions-check-earlier' into feature/…
bbrands02 Dec 11, 2024
5c91953
Added code for deleting invalid or non existing objects
bbrands02 Dec 11, 2024
255352b
Added a findAllBySynchronization function
bbrands02 Dec 12, 2024
8131c2f
Fix conditions if empty frontend
bbrands02 Dec 12, 2024
eaeb4f6
Fixed deleting invalid or non existing objects
bbrands02 Dec 12, 2024
b1f61b1
Added docblocks
bbrands02 Dec 12, 2024
12d405a
Merge remote-tracking branch 'origin/feature/BEHEER-3021/fix-jobs' in…
bbrands02 Dec 13, 2024
bab8d12
Merge branch 'feature/BEHEER-3021/fix-jobs' into feature/CONNECTOR-14…
WilcoLouwerse Dec 13, 2024
6f7a403
Merge remote-tracking branch 'origin/feature/CONNECTOR-144/rate-limit…
bbrands02 Dec 13, 2024
afc9b28
Merge remote-tracking branch 'origin/feature/CONNECTOR-144/code-clean…
bbrands02 Dec 13, 2024
4d7720e
Merge remote-tracking branch 'origin/fix/AXCVDWOF-41/conditions-check…
bbrands02 Dec 13, 2024
0353f43
Merge remote-tracking branch 'origin/feature/CONNECTOR-164/added-hash…
bbrands02 Dec 13, 2024
923077e
Merge branch 'feature/CONNECTOR-61/delete-invalid-objects' into test-…
bbrands02 Dec 13, 2024
6971fd1
merge fix
bbrands02 Dec 13, 2024
7bbb47d
Fix ActionTask for ratelimit
WilcoLouwerse Dec 19, 2024
5eaa439
Make sure we never lower the job next run (when rate limit is reached)
WilcoLouwerse Dec 19, 2024
788c8bb
Merge remote-tracking branch 'origin/development' into test-last-bran…
bbrands02 Dec 20, 2024
2214752
Fix rate limit check and next endpoint check
bbrands02 Dec 20, 2024
ba7cdb0
fix test job result in frontend
bbrands02 Dec 20, 2024
8beef05
xxllnc woo config examples
bbrands02 Dec 20, 2024
fecfe87
format source.lastCall datetime
bbrands02 Dec 31, 2024
daf6add
SyncService subObjectId in endpoints
bbrands02 Dec 31, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@
"title": "omschrijving",
"summary": "zaaktypeomschrijving",
"description": "zaaktypeomschrijving",
"category": "{% if zaaktypecode|default %}{% set wooVerzoekenEnBesluiten = ['LP00000431', 'B1873', 'cherry'] %}{% set klachtoordelen = ['LP00000091', 'LP00001132', 'LP00000121', 'B0757', 'LP00000832', 'LP00001096'] %}{% if zaaktypecode in wooVerzoekenEnBesluiten %}{{ 'Woo-verzoeken en -besluiten' }}{% elseif zaaktypecode in klachtoordelen %}{{ 'Klachtoordelen' }}{% endif %}{% endif %}",
"category": "{% if zaaktypecode|default %}{% set wooVerzoekenEnBesluiten = ['LP00000431', 'B1873'] %}{% set klachtoordelen = ['LP00000091', 'LP00001132', 'LP00000121', 'B0757', 'LP00000832', 'LP00001096'] %}{% if zaaktypecode in wooVerzoekenEnBesluiten %}{{ 'Woo-verzoeken en -besluiten' }}{% elseif zaaktypecode in klachtoordelen %}{{ 'Klachtoordelen' }}{% endif %}{% endif %}",
"published": "startdatum",
"modified": "{{ 'now'|date('H:i:sTm-d-Y') }}",
"attachments": "[{% if files|default %}{% for file in files %} { {% if file['titel']|default %}\"title\": \"{{ file['titel'] }}\",{% endif %}\"labels\": [\"{{ 'Informatieverzoek' }}\"],{% if file['formaat']|default %}\"extension\": \"{{ file['formaat']|split('/')|last }}\",\"type\": \"{{ file['formaat'] }}\",{% endif %}{% if file['inhoud']|default and file['formaat']|default %}\"accessUrl\": \"data:{{ file['formaat'] }};base64,{{ file.inhoud }}\"{% endif %} }{{ loop.last ? '' : ',' }} {% endfor %}{% endif %}]",
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,7 @@
"sourceId": "1",
"sourceType": "api",
"sourceHash": "",
"sourceHashMapping": "1",
"sourceTargetMapping": "1",
"sourceConfig": {
"idPosition": "identificatie",
Expand All @@ -13,6 +14,7 @@
"headers": [],
"query.startdatum__gte": "2024-08-01",
"query.einddatum__lt": "2025-01-01",
"query.maximaleVertrouwelijkheidaanduiding": "openbaar",
"usesPagination": "false",
"extraDataConfigs.0.staticEndpoint": "/tlb/zaaksysteem/api/v1/zaken/{{ originId }}/informatieobjecten",
"extraDataConfigs.0.mergeExtraData": "true",
Expand All @@ -21,5 +23,17 @@
"extraDataConfigs.0.extraDataConfigPerResult.staticEndpoint": "/tlb/zaaksysteem/api/v1/informatieobjecten/{{ originId }}"
},
"targetId": "1/1",
"targetType": "register/schema"
"targetType": "register/schema",
"conditions": {
"if": [
{
"or": [
{ "in": [{ "var": "zaaktypecode" }, ["LP00000431", "B1873"]] },
{ "in": [{ "var": "zaaktypecode" }, ["LP00000091", "LP00001132", "LP00000121", "B0757", "LP00000832", "LP00001096"]] }
]
},
true,
false
]
}
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,22 @@
{
"name": "Xxllnc document to attachment",
"version": "0.0.1",
"mapping": {
"title": "filename",
"labels": "[{{ label }}]",
"size": "size",
"type": "mimetype",
"extension": "{{ filename|split('.')|last }}",
"shareUrl": "{ \"accessUrl\": \"https://urlvanzaaksysteem/{{ caseId }}/document/{{ uuid }}/download\",\"downloadUrl\": \"https://zaaksysteem-accept.noordwijk.nl/api/case/{{ caseId }}/document/{{ uuid }}/download\",\"shareUrl\": \"https://zaaksysteem-accept.noordwijk.nl/api/case/{{ caseId }}/document/{{ uuid }}/download\", {% if filename|default %}\"filename\": \"filename\",\"extension\": \"{{ filename|split('.')|last }}\",{% endif %}\"source\": \"1\" }"
},
"unset": [],
"cast": {
"title": "unsetIfValue==filename",
"labels": "jsonToArray",
"size": "unsetIfValue==size",
"type": "unsetIfValue==mimetype",
"extension": "unsetIfValue==",
"shareUrl": "jsonToArray"
},
"passThrough": false
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,29 @@
{
"name": "Xxllnc zoek endpoint to Publication",
"version": "0.0.1",
"mapping": {
"title": "{{ values['case.subject'] }}",
"summary": "{{ values['attribute.woo_samenvatting'] }}",
"description": "{{ values['attribute.woo_beschrijving'] }}",
"category": "{{ values['attribute.woo_categorie'] }}",
"featured": false,
"status": "Published",
"attachmentCount": 0,
"published": "{{ values['attribute.woo_publicatiedatum'] }}",
"modified": "{{ \"now\"|date(\"Y-m-d\\\\TH:i:s.v\\\\Z\") }}",
"license": "eupl2",
"catalog": "1bcb99cd-7e47-4242-b7a7-ac7d36fe5d5e",
"attachments": "[{% set fileCount = 0 %}{% for file in values['attribute.woo_publicatie'] %}{% if fileCount > 0 %}, {% endif %}{% set file = file|merge({'label': 'Publicatie', 'caseId': id}) %}{{ executeMapping(2, file)|json_encode }}{% set fileCount = fileCount + 1 %}{% endfor %}{% for file in values['attribute.woo_inventarisatielijst'] %}{% if fileCount > 0 %}, {% endif %}{% set file = file|merge({'label': 'Inventarisatielijst', 'caseId': id}) %}{{ executeMapping(2, file)|json_encode }}{% set fileCount = fileCount + 1 %}{% endfor %}{% for file in values['attribute.woo_informatieverzoek'] %}{% if fileCount > 0 %}, {% endif %}{% set file = file|merge({'label': 'Informatieverzoek', 'caseId': id}) %}{{ executeMapping(2, file)|json_encode }}{% set fileCount = fileCount + 1 %}{% endfor %}{% for file in values['attribute.woo_besluit'] %}{% if fileCount > 0 %}, {% endif %}{% set file = file|merge({'label': 'Besluit', 'caseId': id}) %}{{ executeMapping(2, file)|json_encode }}{% set fileCount = fileCount + 1 %}{% endfor %}]"
},
"unset": [],
"cast": {
"title": "unsetIfValue==",
"summary": "unsetIfValue==",
"description": "unsetIfValue==",
"category": "unsetIfValue==",
"published": "unsetIfValue==",
"modified": "unsetIfValue==",
"attachments": "jsonToArray"
},
"passThrough": false
}
18 changes: 18 additions & 0 deletions configurations/xxllnc-zoekendpoint-woo/sources/xxllnc-v1.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,18 @@
{
"name": "Xxllnc zaaksysteem",
"version": "0.0.1",
"location": "",
"isEnabled": true,
"type": "api",
"auth": "none",
"authorizationPassthroughMethod": "header",
"configuration": {
"headers.API-KEY": "",
"headers.API-Interface-ID": ""
},
"endpointsConfig": [],
"logRetention": 3600,
"errorRetention": 86400,
"objectCount": 0,
"test": false
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,31 @@
{
"name": "Xxllnc zoek endpoint to Publications",
"description": "",
"version": "0.0.1",
"sourceId": "1",
"sourceType": "api",
"sourceTargetMapping": "1",
"sourceConfig": {
"idPosition": "id",
"resultsPosition": "result",
"endpoint": "/public_search/x/search",
"headers": [],
"query": []
},
"currentPage": 1,
"targetId": "1/1",
"targetType": "register/schema",
"conditions": {
"and": [
{
"var": "values['attribute.woo_categorie']"
},
{
"var": "values['case.subject']"
},
{
"var": "values['attribute.woo_publicatiedatum']"
}
]
}
}
11 changes: 10 additions & 1 deletion lib/Action/SynchronizationAction.php
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,7 @@
use OCA\OpenConnector\Service\SynchronizationService;
use OCA\OpenConnector\Db\SynchronizationMapper;
use OCA\OpenConnector\Db\SynchronizationContractMapper;
use Symfony\Component\HttpKernel\Exception\TooManyRequestsHttpException;

/**
* This action handles the synchronization of data from the source to the target.
Expand Down Expand Up @@ -70,7 +71,15 @@ public function run(array $argument = []): array
$response['stackTrace'][] = 'Doing the synchronization';
try {
$objects = $this->synchronizationService->synchronize($synchronization);
} catch (Exception $e) {
} catch (TooManyRequestsHttpException $e) {
$response['level'] = 'WARNING';
$response['stackTrace'][] = $response['message'] = 'Stopped synchronization: ' . $e->getMessage();
if (isset($e->getHeaders()['X-RateLimit-Reset']) === true) {
$response['nextRun'] = $e->getHeaders()['X-RateLimit-Reset'];
$response['stackTrace'][] = 'Returning X-RateLimit-Reset header to update Job nextRun: ' . $response['nextRun'];
}
return $response;
} catch (Exception $e) {
$response['level'] = 'ERROR';
$response['stackTrace'][] = $response['message'] = 'Failed to synchronize: ' . $e->getMessage();
return $response;
Expand Down
25 changes: 17 additions & 8 deletions lib/Controller/SynchronizationsController.php
Original file line number Diff line number Diff line change
Expand Up @@ -256,17 +256,26 @@ public function test(int $id): JSONResponse

// Try to synchronize
try {
$logAndContractArray = $this->synchronizationService->synchronize(synchronization: $synchronization, isTest: true);
$logAndContractArray = $this->synchronizationService->synchronize(
synchronization: $synchronization,
isTest: true
);

// Return the result as a JSON response
return new JSONResponse(data: $logAndContractArray, statusCode: 200);
} catch (Exception $e) {
// If synchronizaiton fails, return an error response
return new JSONResponse([
'error' => 'Synchronization error',
'message' => $e->getMessage()
], 400);
}
// Check if getHeaders method exists and use it if available
$headers = method_exists($e, 'getHeaders') ? $e->getHeaders() : [];

return new JSONResponse($resultFromTest, 200);
// If synchronization fails, return an error response
return new JSONResponse(
data: [
'error' => 'Synchronization error',
'message' => $e->getMessage()
],
statusCode: $e->getCode() ?? 400,
headers: $headers
);
}
}
}
Loading
Loading