Skip to content

Commit

Permalink
Merge pull request #2 from php-etl/feature/optimize-upload-capacity
Browse files Browse the repository at this point in the history 
Optimize upload memory consumption
  • Loading branch information
@JoMessina
JoMessina authored Nov 26, 2024
2 parents 6801270 + 5f9d38c commit 47acd3a
Show file tree
Hide file tree
Showing 21 changed files with 223 additions and 91 deletions.
2 changes: 1 addition & 1 deletion .github/workflows/phpstan-5.yaml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@ jobs:
php_version: '8.2'

- name: PHPStan
uses: php-actions/phpstan@v2
uses: php-actions/phpstan@v3
with:
path: src/
level: 5
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/phpstan-6.yaml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@ jobs:
php_version: '8.2'

- name: PHPStan
uses: php-actions/phpstan@v2
uses: php-actions/phpstan@v3
with:
path: src/
level: 6
Expand Down
1 change: 1 addition & 0 deletions .php-cs-fixer.dist.php
View file
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,7 @@
return (new PhpCsFixer\Config())
->setRiskyAllowed(true)
->setRules([
'@PHP82Migration' => true,
'@PHP81Migration' => true,
'@PHP80Migration:risky' => true,
'@PHPUnit84Migration:risky' => true,
Expand Down
23 changes: 13 additions & 10 deletions composer.json
View file
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,15 @@
"name": "php-etl/prestashop-plugin",
"description": "Plugin for Prestashop Extractor and Loader",
"type": "gyroscops-plugin",
"license": "MIT",
"authors": [
{
"name": "Grégory Planchat",
"email": "[email protected]"
}
],
"minimum-stability": "dev",
"prefer-stable": true,
"require": {
"php": "^8.2",
"php-etl/configurator-contracts": "0.8.*",
Expand All @@ -12,17 +21,11 @@
},
"require-dev": {
"php-etl/phpunit-extension": "0.5.*",
"rector/rector": "^0.15"
"rector/rector": "^0.15",
"php-etl/bucket-contracts": "0.2.*",
"php-etl/bucket": "*",
"phpstan/phpstan": "^1.10"
},
"license": "MIT",
"authors": [
{
"name": "Grégory Planchat",
"email": "[email protected]"
}
],
"minimum-stability": "dev",
"prefer-stable": true,
"autoload": {
"psr-4": {
"Kiboko\\Plugin\\Prestashop\\": "src/"
Expand Down
72 changes: 62 additions & 10 deletions composer.lock
View file

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

12 changes: 6 additions & 6 deletions phpunit.xml
View file
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
<?xml version="1.0"?>
<phpunit
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:noNamespaceSchemaLocation="https://schema.phpunit.de/10.1/phpunit.xsd"
xsi:noNamespaceSchemaLocation="https://schema.phpunit.de/10.0/phpunit.xsd"
backupGlobals="true"
colors="false"
processIsolation="false"
Expand All @@ -21,12 +21,12 @@
<directory>tests/functional/</directory>
</testsuite>
</testsuites>
<php>
<ini name="allow_url_include" value="1"/>
</php>
<source>
<coverage>
<include>
<directory suffix=".php">src</directory>
</include>
</source>
</coverage>
<php>
<ini name="allow_url_include" value="1"/>
</php>
</phpunit>
2 changes: 1 addition & 1 deletion src/Builder/Capacity/All.php
View file
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@

final class All implements Builder
{
private null|Node\Expr|Node\Identifier $endpoint = null;
private Node\Expr|Node\Identifier|null $endpoint = null;

public function __construct(

Check failure on line 17 in src/Builder/Capacity/All.php

View workflow job for this annotation

GitHub Actions / phpstan-7 

Method Kiboko\Plugin\Prestashop\Builder\Capacity\All::__construct() has parameter $options with no value type specified in iterable type array.

Check failure on line 17 in src/Builder/Capacity/All.php

View workflow job for this annotation

GitHub Actions / phpstan-5 

Method Kiboko\Plugin\Prestashop\Builder\Capacity\All::__construct() has parameter $options with no value type specified in iterable type array.

Check failure on line 17 in src/Builder/Capacity/All.php

View workflow job for this annotation

GitHub Actions / phpstan-8 

Method Kiboko\Plugin\Prestashop\Builder\Capacity\All::__construct() has parameter $options with no value type specified in iterable type array.

Check failure on line 17 in src/Builder/Capacity/All.php

View workflow job for this annotation

GitHub Actions / phpstan-8 

Method Kiboko\Plugin\Prestashop\Builder\Capacity\All::__construct() has parameter $options with no value type specified in iterable type array.

Check failure on line 17 in src/Builder/Capacity/All.php

View workflow job for this annotation

GitHub Actions / phpstan-5 

Method Kiboko\Plugin\Prestashop\Builder\Capacity\All::__construct() has parameter $options with no value type specified in iterable type array.

Check failure on line 17 in src/Builder/Capacity/All.php

View workflow job for this annotation

GitHub Actions / phpstan-7 

Method Kiboko\Plugin\Prestashop\Builder\Capacity\All::__construct() has parameter $options with no value type specified in iterable type array.
private readonly ExpressionLanguage $interpreter,
Expand Down
8 changes: 4 additions & 4 deletions src/Builder/Capacity/Create.php
View file
Original file line number Diff line number Diff line change
Expand Up @@ -9,8 +9,8 @@

final class Create implements Builder
{
private null|Node\Expr|Node\Identifier $endpoint = null;
private null|Node\Expr $data = null;
private Node\Expr|Node\Identifier|null $endpoint = null;
private ?Node\Expr $data = null;

public function __construct(public array $options = [])

Check failure on line 15 in src/Builder/Capacity/Create.php

View workflow job for this annotation

GitHub Actions / phpstan-7 

Method Kiboko\Plugin\Prestashop\Builder\Capacity\Create::__construct() has parameter $options with no value type specified in iterable type array.

Check failure on line 15 in src/Builder/Capacity/Create.php

View workflow job for this annotation

GitHub Actions / phpstan-5 

Method Kiboko\Plugin\Prestashop\Builder\Capacity\Create::__construct() has parameter $options with no value type specified in iterable type array.

Check failure on line 15 in src/Builder/Capacity/Create.php

View workflow job for this annotation

GitHub Actions / phpstan-8 

Method Kiboko\Plugin\Prestashop\Builder\Capacity\Create::__construct() has parameter $options with no value type specified in iterable type array.

Check failure on line 15 in src/Builder/Capacity/Create.php

View workflow job for this annotation

GitHub Actions / phpstan-8 

Method Kiboko\Plugin\Prestashop\Builder\Capacity\Create::__construct() has parameter $options with no value type specified in iterable type array.

Check failure on line 15 in src/Builder/Capacity/Create.php

View workflow job for this annotation

GitHub Actions / phpstan-5 

Method Kiboko\Plugin\Prestashop\Builder\Capacity\Create::__construct() has parameter $options with no value type specified in iterable type array.

Check failure on line 15 in src/Builder/Capacity/Create.php

View workflow job for this annotation

GitHub Actions / phpstan-7 

Method Kiboko\Plugin\Prestashop\Builder\Capacity\Create::__construct() has parameter $options with no value type specified in iterable type array.
{
Expand Down Expand Up @@ -80,7 +80,7 @@ public function getNode(): Node
var: new Node\Expr\Variable('line'),
expr: new Node\Expr\Yield_(
value: new Node\Expr\New_(
class: new Node\Name\FullyQualified(name: 'Kiboko\Component\Bucket\AcceptanceResultBucket'),
class: new Node\Name\FullyQualified(name: \Kiboko\Component\Bucket\AcceptanceResultBucket::class),
args: [

Check warning on line 84 in src/Builder/Capacity/Create.php

View workflow job for this annotation

GitHub Actions / infection 

Escaped Mutant for Mutator "ArrayItemRemoval":

--- Original
+++ New
@@ @@
         foreach ($this->options as $key => $value) {
             $options[] = new Node\Expr\ArrayItem(value: new Node\Scalar\String_($value), key: new Node\Scalar\String_($key));
         }
-        return new Node\Stmt\While_(cond: new Node\Expr\ConstFetch(new Node\Name('true')), stmts: [new Node\Stmt\TryCatch(stmts: [new Node\Stmt\Expression(new Node\Expr\Assign(var: new Node\Expr\Variable('result'), expr: new Node\Expr\Ternary(new Node\Expr\MethodCall(new Node\Expr\MethodCall(var: new Node\Expr\PropertyFetch(var: new Node\Expr\Variable('this'), name: new Node\Identifier('client')), name: $this->endpoint), new Node\Identifier('create'), args: [new Node\Arg(value: $this->data), new Node\Arg(new Node\Expr\Array_($options))]), if: null, else: new Node\Expr\Array_()))), new Node\Stmt\Expression(expr: new Node\Expr\Assign(var: new Node\Expr\Variable('line'), expr: new Node\Expr\Yield_(value: new Node\Expr\New_(class: new Node\Name\FullyQualified(name: \Kiboko\Component\Bucket\AcceptanceResultBucket::class), args: [new Node\Arg(new Node\Expr\FuncCall(new Node\Name('array_merge'), [new Node\Expr\Variable('line'), new Node\Expr\Variable('result')]))]))))], catches: [new Node\Stmt\Catch_(types: [new Node\Name\FullyQualified('Exception')], var: new Node\Expr\Variable('exception'), stmts: [new Node\Stmt\Expression(expr: new Node\Expr\MethodCall(var: new Node\Expr\PropertyFetch(var: new Node\Expr\Variable('this'), name: 'logger'), name: new Node\Identifier('error'), args: [new Node\Arg(value: new Node\Expr\MethodCall(var: new Node\Expr\Variable('exception'), name: new Node\Identifier('getMessage'))), new Node\Arg(value: new Node\Expr\Array_(items: [new Node\Expr\ArrayItem(value: new Node\Expr\Variable('exception'), key: new Node\Scalar\String_('exception')), new Node\Expr\ArrayItem(value: new Node\Expr\Variable('line'), key: new Node\Scalar\String_('item'))], attributes: ['kind' => Node\Expr\Array_::KIND_SHORT]))])), new Node\Stmt\Expression(expr: new Node\Expr\Assign(var: new Node\Expr\Variable('line'), expr: new Node\Expr\Yield_(value: new Node\Expr\New_(class: new Node\Name\FullyQualified(name: \Kiboko\Component\Bucket\RejectionResultBucket::class), args: [new Node\Arg(value: new Node\Expr\Variable('exception')), new Node\Arg(value: new Node\Expr\Variable('line'))]))))])])]);
+        return new Node\Stmt\While_(cond: new Node\Expr\ConstFetch(new Node\Name('true')), stmts: [new Node\Stmt\TryCatch(stmts: [new Node\Stmt\Expression(new Node\Expr\Assign(var: new Node\Expr\Variable('result'), expr: new Node\Expr\Ternary(new Node\Expr\MethodCall(new Node\Expr\MethodCall(var: new Node\Expr\PropertyFetch(var: new Node\Expr\Variable('this'), name: new Node\Identifier('client')), name: $this->endpoint), new Node\Identifier('create'), args: [new Node\Arg(value: $this->data), new Node\Arg(new Node\Expr\Array_($options))]), if: null, else: new Node\Expr\Array_()))), new Node\Stmt\Expression(expr: new Node\Expr\Assign(var: new Node\Expr\Variable('line'), expr: new Node\Expr\Yield_(value: new Node\Expr\New_(class: new Node\Name\FullyQualified(name: \Kiboko\Component\Bucket\AcceptanceResultBucket::class), args: []))))], catches: [new Node\Stmt\Catch_(types: [new Node\Name\FullyQualified('Exception')], var: new Node\Expr\Variable('exception'), stmts: [new Node\Stmt\Expression(expr: new Node\Expr\MethodCall(var: new Node\Expr\PropertyFetch(var: new Node\Expr\Variable('this'), name: 'logger'), name: new Node\Identifier('error'), args: [new Node\Arg(value: new Node\Expr\MethodCall(var: new Node\Expr\Variable('exception'), name: new Node\Identifier('getMessage'))), new Node\Arg(value: new Node\Expr\Array_(items: [new Node\Expr\ArrayItem(value: new Node\Expr\Variable('exception'), key: new Node\Scalar\String_('exception')), new Node\Expr\ArrayItem(value: new Node\Expr\Variable('line'), key: new Node\Scalar\String_('item'))], attributes: ['kind' => Node\Expr\Array_::KIND_SHORT]))])), new Node\Stmt\Expression(expr: new Node\Expr\Assign(var: new Node\Expr\Variable('line'), exp

Check warning on line 84 in src/Builder/Capacity/Create.php

View workflow job for this annotation

GitHub Actions / infection 

Escaped Mutant for Mutator "ArrayItemRemoval":

--- Original
+++ New
@@ @@
         foreach ($this->options as $key => $value) {
             $options[] = new Node\Expr\ArrayItem(value: new Node\Scalar\String_($value), key: new Node\Scalar\String_($key));
         }
-        return new Node\Stmt\While_(cond: new Node\Expr\ConstFetch(new Node\Name('true')), stmts: [new Node\Stmt\TryCatch(stmts: [new Node\Stmt\Expression(new Node\Expr\Assign(var: new Node\Expr\Variable('result'), expr: new Node\Expr\Ternary(new Node\Expr\MethodCall(new Node\Expr\MethodCall(var: new Node\Expr\PropertyFetch(var: new Node\Expr\Variable('this'), name: new Node\Identifier('client')), name: $this->endpoint), new Node\Identifier('create'), args: [new Node\Arg(value: $this->data), new Node\Arg(new Node\Expr\Array_($options))]), if: null, else: new Node\Expr\Array_()))), new Node\Stmt\Expression(expr: new Node\Expr\Assign(var: new Node\Expr\Variable('line'), expr: new Node\Expr\Yield_(value: new Node\Expr\New_(class: new Node\Name\FullyQualified(name: \Kiboko\Component\Bucket\AcceptanceResultBucket::class), args: [new Node\Arg(new Node\Expr\FuncCall(new Node\Name('array_merge'), [new Node\Expr\Variable('line'), new Node\Expr\Variable('result')]))]))))], catches: [new Node\Stmt\Catch_(types: [new Node\Name\FullyQualified('Exception')], var: new Node\Expr\Variable('exception'), stmts: [new Node\Stmt\Expression(expr: new Node\Expr\MethodCall(var: new Node\Expr\PropertyFetch(var: new Node\Expr\Variable('this'), name: 'logger'), name: new Node\Identifier('error'), args: [new Node\Arg(value: new Node\Expr\MethodCall(var: new Node\Expr\Variable('exception'), name: new Node\Identifier('getMessage'))), new Node\Arg(value: new Node\Expr\Array_(items: [new Node\Expr\ArrayItem(value: new Node\Expr\Variable('exception'), key: new Node\Scalar\String_('exception')), new Node\Expr\ArrayItem(value: new Node\Expr\Variable('line'), key: new Node\Scalar\String_('item'))], attributes: ['kind' => Node\Expr\Array_::KIND_SHORT]))])), new Node\Stmt\Expression(expr: new Node\Expr\Assign(var: new Node\Expr\Variable('line'), expr: new Node\Expr\Yield_(value: new Node\Expr\New_(class: new Node\Name\FullyQualified(name: \Kiboko\Component\Bucket\RejectionResultBucket::class), args: [new Node\Arg(value: new Node\Expr\Variable('exception')), new Node\Arg(value: new Node\Expr\Variable('line'))]))))])])]);
+        return new Node\Stmt\While_(cond: new Node\Expr\ConstFetch(new Node\Name('true')), stmts: [new Node\Stmt\TryCatch(stmts: [new Node\Stmt\Expression(new Node\Expr\Assign(var: new Node\Expr\Variable('result'), expr: new Node\Expr\Ternary(new Node\Expr\MethodCall(new Node\Expr\MethodCall(var: new Node\Expr\PropertyFetch(var: new Node\Expr\Variable('this'), name: new Node\Identifier('client')), name: $this->endpoint), new Node\Identifier('create'), args: [new Node\Arg(value: $this->data), new Node\Arg(new Node\Expr\Array_($options))]), if: null, else: new Node\Expr\Array_()))), new Node\Stmt\Expression(expr: new Node\Expr\Assign(var: new Node\Expr\Variable('line'), expr: new Node\Expr\Yield_(value: new Node\Expr\New_(class: new Node\Name\FullyQualified(name: \Kiboko\Component\Bucket\AcceptanceResultBucket::class), args: []))))], catches: [new Node\Stmt\Catch_(types: [new Node\Name\FullyQualified('Exception')], var: new Node\Expr\Variable('exception'), stmts: [new Node\Stmt\Expression(expr: new Node\Expr\MethodCall(var: new Node\Expr\PropertyFetch(var: new Node\Expr\Variable('this'), name: 'logger'), name: new Node\Identifier('error'), args: [new Node\Arg(value: new Node\Expr\MethodCall(var: new Node\Expr\Variable('exception'), name: new Node\Identifier('getMessage'))), new Node\Arg(value: new Node\Expr\Array_(items: [new Node\Expr\ArrayItem(value: new Node\Expr\Variable('exception'), key: new Node\Scalar\String_('exception')), new Node\Expr\ArrayItem(value: new Node\Expr\Variable('line'), key: new Node\Scalar\String_('item'))], attributes: ['kind' => Node\Expr\Array_::KIND_SHORT]))])), new Node\Stmt\Expression(expr: new Node\Expr\Assign(var: new Node\Expr\Variable('line'), exp
new Node\Arg(
new Node\Expr\FuncCall(
Expand Down Expand Up @@ -144,7 +144,7 @@ class: new Node\Name\FullyQualified(name: 'Kiboko\Component\Bucket\AcceptanceRes
expr: new Node\Expr\Yield_(
value: new Node\Expr\New_(
class: new Node\Name\FullyQualified(
name: 'Kiboko\Component\Bucket\RejectionResultBucket'
name: \Kiboko\Component\Bucket\RejectionResultBucket::class
),
args: [
new Node\Arg(
Expand Down
2 changes: 1 addition & 1 deletion src/Builder/Capacity/Get.php
View file
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@

final class Get implements Builder
{
private null|Node\Expr|Node\Identifier $endpoint = null;
private Node\Expr|Node\Identifier|null $endpoint = null;

public function __construct(

Check failure on line 17 in src/Builder/Capacity/Get.php

View workflow job for this annotation

GitHub Actions / phpstan-7 

Method Kiboko\Plugin\Prestashop\Builder\Capacity\Get::__construct() has parameter $options with no value type specified in iterable type array.

Check failure on line 17 in src/Builder/Capacity/Get.php

View workflow job for this annotation

GitHub Actions / phpstan-5 

Method Kiboko\Plugin\Prestashop\Builder\Capacity\Get::__construct() has parameter $options with no value type specified in iterable type array.

Check failure on line 17 in src/Builder/Capacity/Get.php

View workflow job for this annotation

GitHub Actions / phpstan-8 

Method Kiboko\Plugin\Prestashop\Builder\Capacity\Get::__construct() has parameter $options with no value type specified in iterable type array.

Check failure on line 17 in src/Builder/Capacity/Get.php

View workflow job for this annotation

GitHub Actions / phpstan-8 

Method Kiboko\Plugin\Prestashop\Builder\Capacity\Get::__construct() has parameter $options with no value type specified in iterable type array.

Check failure on line 17 in src/Builder/Capacity/Get.php

View workflow job for this annotation

GitHub Actions / phpstan-5 

Method Kiboko\Plugin\Prestashop\Builder\Capacity\Get::__construct() has parameter $options with no value type specified in iterable type array.

Check failure on line 17 in src/Builder/Capacity/Get.php

View workflow job for this annotation

GitHub Actions / phpstan-7 

Method Kiboko\Plugin\Prestashop\Builder\Capacity\Get::__construct() has parameter $options with no value type specified in iterable type array.
private readonly ExpressionLanguage $interpreter,
Expand Down
8 changes: 4 additions & 4 deletions src/Builder/Capacity/Update.php
View file
Original file line number Diff line number Diff line change
Expand Up @@ -9,8 +9,8 @@

final class Update implements Builder
{
private null|Node\Expr|Node\Identifier $endpoint = null;
private null|Node\Expr $data = null;
private Node\Expr|Node\Identifier|null $endpoint = null;
private ?Node\Expr $data = null;

public function __construct(public array $options = [])

Check failure on line 15 in src/Builder/Capacity/Update.php

View workflow job for this annotation

GitHub Actions / phpstan-7 

Method Kiboko\Plugin\Prestashop\Builder\Capacity\Update::__construct() has parameter $options with no value type specified in iterable type array.

Check failure on line 15 in src/Builder/Capacity/Update.php

View workflow job for this annotation

GitHub Actions / phpstan-5 

Method Kiboko\Plugin\Prestashop\Builder\Capacity\Update::__construct() has parameter $options with no value type specified in iterable type array.

Check failure on line 15 in src/Builder/Capacity/Update.php

View workflow job for this annotation

GitHub Actions / phpstan-8 

Method Kiboko\Plugin\Prestashop\Builder\Capacity\Update::__construct() has parameter $options with no value type specified in iterable type array.

Check failure on line 15 in src/Builder/Capacity/Update.php

View workflow job for this annotation

GitHub Actions / phpstan-8 

Method Kiboko\Plugin\Prestashop\Builder\Capacity\Update::__construct() has parameter $options with no value type specified in iterable type array.

Check failure on line 15 in src/Builder/Capacity/Update.php

View workflow job for this annotation

GitHub Actions / phpstan-5 

Method Kiboko\Plugin\Prestashop\Builder\Capacity\Update::__construct() has parameter $options with no value type specified in iterable type array.

Check failure on line 15 in src/Builder/Capacity/Update.php

View workflow job for this annotation

GitHub Actions / phpstan-7 

Method Kiboko\Plugin\Prestashop\Builder\Capacity\Update::__construct() has parameter $options with no value type specified in iterable type array.
{
Expand Down Expand Up @@ -80,7 +80,7 @@ public function getNode(): Node
var: new Node\Expr\Variable('line'),
expr: new Node\Expr\Yield_(
value: new Node\Expr\New_(
class: new Node\Name\FullyQualified(name: 'Kiboko\Component\Bucket\AcceptanceResultBucket'),
class: new Node\Name\FullyQualified(name: \Kiboko\Component\Bucket\AcceptanceResultBucket::class),
args: [
new Node\Arg(
new Node\Expr\FuncCall(
Expand Down Expand Up @@ -144,7 +144,7 @@ class: new Node\Name\FullyQualified(name: 'Kiboko\Component\Bucket\AcceptanceRes
expr: new Node\Expr\Yield_(
value: new Node\Expr\New_(
class: new Node\Name\FullyQualified(
name: 'Kiboko\Component\Bucket\RejectionResultBucket'
name: \Kiboko\Component\Bucket\RejectionResultBucket::class
),
args: [
new Node\Arg(
Expand Down
Loading

0 comments on commit 47acd3a

Please sign in to comment.