Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[filebeat][gcs] - Added input metrics #41505

Merged
merged 18 commits into from
Nov 18, 2024
Merged

[filebeat][gcs] - Added input metrics #41505

merged 18 commits into from
Nov 18, 2024
+301 −32

Conversation

ShourieG
Copy link
Contributor

@ShourieG ShourieG commented Nov 4, 2024 •
edited
Loading

Type of change

  • Enhancement

Proposed commit message

Added input metrics along with some basic tests.

Checklist

  • My code follows the style guidelines of this project
  • I have commented my code, particularly in hard-to-understand areas
    - [] I have made corresponding changes to the documentation
    - [ ] I have made corresponding change to the default configuration files
  • I have added tests that prove my fix is effective or that my feature works
  • I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

NOTE

Ignore linting errors

Disruptive User Impact

None

Author's Checklist

  • [ ]

How to test this PR locally

Related issues

Use cases

Screenshots

Logs

@ShourieG ShourieG requested a review from a team as a code owner November 4, 2024 13:11
@botelastic botelastic bot added the needs_team Indicates that the issue/PR needs a Team:* label label Nov 4, 2024
@mergify Mergify
Copy link
Contributor

mergify bot commented Nov 4, 2024

This pull request does not have a backport label.
If this is a bug or security fix, could you label this PR @ShourieG? 🙏.
For such, you'll need to label your PR with:

  • The upcoming major version of the Elastic Stack
  • The upcoming minor version of the Elastic Stack (if you're not pushing a breaking change)

To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

  • backport-8./d is the label to automatically backport to the 8./d branch. /d is the digit

@mergify Mergify
Copy link
Contributor

mergify bot commented Nov 4, 2024

backport-8.x has been added to help with the transition to the new branch 8.x.
If you don't need it please use backport-skip label and remove the backport-8.x label.

@mergify mergify bot added the backport-8.x Automated backport to the 8.x branch with mergify label Nov 4, 2024
@ShourieG ShourieG added the Team:Security-Service Integrations Security Service Integrations Team label Nov 4, 2024
@elasticmachine
Copy link
Collaborator

Pinging @elastic/security-service-integrations (Team:Security-Service Integrations)

@botelastic botelastic bot removed the needs_team Indicates that the issue/PR needs a Team:* label label Nov 4, 2024
@ShourieG ShourieG added Filebeat Filebeat needs_team Indicates that the issue/PR needs a Team:* label labels Nov 4, 2024
@botelastic botelastic bot removed the needs_team Indicates that the issue/PR needs a Team:* label label Nov 4, 2024
@ShourieG ShourieG added enhancement needs_team Indicates that the issue/PR needs a Team:* label labels Nov 4, 2024
@botelastic botelastic bot removed the needs_team Indicates that the issue/PR needs a Team:* label label Nov 4, 2024
@ShourieG ShourieG added needs_team Indicates that the issue/PR needs a Team:* label input:GCS labels Nov 4, 2024
@botelastic botelastic bot removed the needs_team Indicates that the issue/PR needs a Team:* label label Nov 4, 2024
andrewkroh
andrewkroh reviewed Nov 4, 2024
View reviewed changes
Copy link
Member

@andrewkroh andrewkroh left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The docs for the input will need updated to include a Metrics section with table.

x-pack/filebeat/input/gcs/metrics.go Outdated Show resolved Hide resolved
x-pack/filebeat/input/gcs/metrics.go Outdated Show resolved Hide resolved
x-pack/filebeat/input/gcs/scheduler.go Outdated Show resolved Hide resolved
x-pack/filebeat/input/gcs/scheduler.go Outdated Show resolved Hide resolved
x-pack/filebeat/input/gcs/metrics.go Show resolved Hide resolved
x-pack/filebeat/input/gcs/metrics.go Show resolved Hide resolved
@mergify Mergify
Copy link
Contributor

mergify bot commented Nov 4, 2024

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b gcs/metrics upstream/gcs/metrics
git merge upstream/main
git push upstream gcs/metrics

@ShourieG
Copy link
Contributor Author

ShourieG commented Nov 7, 2024

@andrewkroh, I've addressed all the PR suggestions.

andrewkroh
andrewkroh reviewed Nov 7, 2024
View reviewed changes
x-pack/filebeat/input/gcs/metrics.go Show resolved Hide resolved
x-pack/filebeat/input/gcs/metrics.go Outdated Show resolved Hide resolved
x-pack/filebeat/input/gcs/metrics.go
gcsObjectsListedTotal: monitoring.NewUint(reg, "gcs_objects_listed_total"),
gcsBytesProcessedTotal: monitoring.NewUint(reg, "gcs_bytes_processed_total"),
gcsEventsCreatedTotal: monitoring.NewUint(reg, "gcs_events_created_total"),
gcsFailedJobsTotal: monitoring.NewUint(reg, "gcs_failed_jobs_total"),
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a gauge.

gcs_failed_jobs_gauge

Copy link
Contributor Author

@ShourieG ShourieG Nov 8, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this should not be a gauge, I'll remove the Dec(). FailedJobsTotal - ExpiredFailedJobsTotal = Recovered Jobs, at any given point of time.

x-pack/filebeat/input/gcs/job.go Outdated Show resolved Hide resolved
x-pack/filebeat/docs/inputs/input-gcs.asciidoc Outdated Show resolved Hide resolved
x-pack/filebeat/docs/inputs/input-gcs.asciidoc Outdated Show resolved Hide resolved
x-pack/filebeat/input/gcs/job.go Outdated Show resolved Hide resolved
@ShourieG
Copy link
Contributor Author

ShourieG commented Nov 8, 2024

@andrewkroh, I've addressed all the latest suggestions.

chemamartinez
chemamartinez approved these changes Nov 18, 2024
View reviewed changes
Copy link
Contributor

@chemamartinez chemamartinez left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@ShourieG ShourieG merged commit 821697d into elastic:main Nov 18, 2024
20 of 22 checks passed
mergify bot pushed a commit that referenced this pull request Nov 18, 2024
@ShourieG @mergify
[filebeat][gcs] - Added input metrics (#41505)
941bb0d 
(cherry picked from commit 821697d)
@mergify mergify bot mentioned this pull request Nov 18, 2024
Merged
4 tasks
@ShourieG ShourieG deleted the gcs/metrics branch November 18, 2024 15:51
ShourieG added a commit that referenced this pull request Nov 19, 2024
@mergify @ShourieG
[filebeat][gcs] - Added input metrics (#41505) (#41663)
6372b59 
(cherry picked from commit 821697d)

Co-authored-by: ShourieG <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@andrewkroh andrewkroh andrewkroh left review comments

@chemamartinez chemamartinez chemamartinez approved these changes

Assignees

@ShourieG ShourieG

Labels
backport-8.x Automated backport to the 8.x branch with mergify enhancement Filebeat Filebeat input:GCS Team:Security-Service Integrations Security Service Integrations Team
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

[filebeat][gcs] - Add input metrics to filebeat gcs input
4 participants
@ShourieG @elasticmachine @andrewkroh @chemamartinez