Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Restructure for better large bucket support (Fixes #138, #128. #100, #80, #54, and #14) #249

Open
wants to merge 30 commits into
base: main
Choose a base branch
from
Open

Restructure for better large bucket support (Fixes #138, #128. #100, #80, #54, and #14) #249

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
30 commits
Select commit Hold shift + click to select a range
c32c13f
Restructure for better large bucket support
ph Apr 7, 2016
d15376f
Add spec_helper
eherot Nov 3, 2023
a38b42a
Use aws-sdk-s3 gem exclusively
eherot Nov 3, 2023
e75a421
Arbitrary change to trigger a build
eherot Nov 3, 2023
9d85332
Update specs to match new code
eherot Nov 3, 2023
c6f2896
Another arbitrary change
eherot Nov 16, 2023
2c53790
Slight logic optimization in the processing policy validator
eherot Nov 20, 2023
e3baf7a
Make SinceDB bookkeeping optional
eherot Nov 20, 2023
619dd5d
Add sincedb spec tests
eherot Nov 20, 2023
d31f5d9
Test our additions to the poller
eherot Nov 20, 2023
69d919c
Skip integration tests if AWS keys are not available
eherot Nov 20, 2023
7f5d4b0
Further changes to accommodate our code changes
eherot Nov 20, 2023
b421e58
[sincedb.start_bookkeeping] Remove unnecessary raise (was added for t…
eherot Jan 29, 2024
04b1f8f
Print the size of the sincedb on load
eherot Jan 26, 2024
461e5a1
Remove SinceDB before re-seeding
eherot Jan 26, 2024
2873d24
Add debugging to the stream decompression process
eherot Jan 29, 2024
53cd961
Pass gzip_pattern to remote_file constructor correctly
eherot Jan 29, 2024
1924137
[stream_downloader.fetch] Call logger using correct name
eherot Jan 29, 2024
05b4a65
Version bump 3.9.0
eherot Jan 29, 2024
3d014e4
Remove "puts" added during debugging
eherot Jan 30, 2024
09d2a03
Better explain sincedb_expire_secs option
eherot Feb 1, 2024
a3e0832
Comment out overly verbose "Waiting for new work"
eherot Feb 1, 2024
f00df11
Warn when key order doesn't match mtime order
eherot Feb 1, 2024
927dfdd
remote_file: only delete if exists
eherot Feb 1, 2024
979a077
Report the number of objects collected during instantiation
eherot Feb 2, 2024
1767452
during processing retry eventually on IOErrors and immediately on Err…
eherot Feb 1, 2024
b4ff6c8
Abandon batch_size vs last fetch count check and only use limit(batch…
eherot Feb 2, 2024
0d77c0f
Log "File downloaded" events as INFO
eherot Feb 6, 2024
c6845ab
Shell out to zcat to decompress s3 file stream
eherot Feb 6, 2024
2aae3bc
Version bump 3.9.0-inv1
eherot Feb 6, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -115,3 +115,5 @@ Programming is not a required skill. Whatever you've seen about open source and
It is more important to the community that you are able to contribute.

For more information about contributing, see the [CONTRIBUTING](https://github.com/elastic/logstash/blob/master/CONTRIBUTING.md) file.

Arbitrary change to trigger a build (2)
Loading