Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

batch size after first occurence #67

Open
DnD-Magnum opened this issue May 18, 2022 · 0 comments
Open

batch size after first occurence #67

DnD-Magnum opened this issue May 18, 2022 · 0 comments

Comments

@DnD-Magnum
Copy link

DnD-Magnum commented May 18, 2022

Hello,

When you use batch tranformer option in your pipeline configuration, you have an error on the batch size after the first occurence.
For exemple , with the batch size to 100 :

-
  batch:
    merge:
      size: 100

Your first batch will contain 100 items, but the other batches will contain 101 items.

The generated code :

public function transform() : \Generator
        {
            $this->storage = [];
            $line = yield;
            $count = 1;
            while (true) {
                if ($count++ >= 100) {
                    $count = 0;
                    $this->storage[] = $line;
                    $line = (yield new \Kiboko\Component\Bucket\AcceptanceResultBucket($this->storage));
                    $this->storage = [];
                } else {
                    $this->storage[] = $line;
                    $line = (yield new \Kiboko\Component\Bucket\EmptyResultBucket());
                }
            }
        }

Geoffrey

gplanchat added a commit that referenced this issue May 20, 2022
#67 batch: fix batch size after first occurrence
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant