Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Logs refactoring to fit ELASTIC stack #498

Merged
merged 1 commit into from
Nov 1, 2024
Merged

Logs refactoring to fit ELASTIC stack #498

merged 1 commit into from
Nov 1, 2024

Conversation

ThanKarab
Copy link
Member

Changelog:

  • All loggers now have the same formatter.
  • Added node identifier to all loggers (controller included)
  • Refactored flower loggers to use the framework log level of the service.
  • Minor log removals.
2024-10-29 14:01:01,798 - ERROR - monetdb_facade.error_handling(157) - [dementia] - [exareme2-LOCALWORKER] - [localworker1] - [668221305] - Error occurred: Exception type: '<class 'OSError'>' and exception message: 'Connection refused'
2024-10-29 15:45:43,077 - INFO - worker_landscape_aggregator.start(357) - [dementia] - [exareme2-CONTROLLER] - [controller] - [BACKGROUND] - WorkerLandscapeAggregator started.
2024-10-29 15:48:17,213 - INFO - error_handlers.handle_bad_user_input(53) - [dementia] - [exareme2-CONTROLLER] - [controller] - [BACKGROUND] - Request Error. Type: 'BadUserInput' Message: 'Data model 'dementia:0.1' does not exist.'
2024-10-29 15:48:44,422 - INFO - federation_info_logs.log_datamodel_added(35) - [dementia] - [exareme2-CONTROLLER] - [controller] - [BACKGROUND] - Datamodel 'longitudinal_dementia:0.1' was added.
2024-10-29 16:04:49,961 - INFO - federation_info_logs.log_experiment_execution(15) - [dementia] - [exareme2-CONTROLLER] - [controller] - [603784292] - Experiment with request id '603784292' and context id '603784293' is starting algorithm 'kmeans', touching datasets 'edsd0,edsd1,edsd2,edsd3,edsd4,edsd5,edsd6,edsd7,edsd8,edsd9' on local workers 'localworker1' with parameters '{'k': 4, 'tol': 0.0001, 'maxiter': 100}'.
2024-10-29 16:04:54,637 - INFO - controller.exec_algorithm(856) - [dementia] - [exareme2-CONTROLLER] - [controller] - [603784292] - Finished execution->  algorithm_name='kmeans' with request_id='603784292'

@ThanKarab ThanKarab marked this pull request as draft October 29, 2024 15:04
@ThanKarab ThanKarab force-pushed the dev/logging_refactoring_for_elk branch 4 times, most recently from 6130b4e to fece1a8 Compare October 29, 2024 22:22
@ThanKarab ThanKarab marked this pull request as ready for review October 30, 2024 08:45
KFilippopolitis
KFilippopolitis approved these changes Nov 1, 2024
View reviewed changes
Copy link
Contributor

@KFilippopolitis KFilippopolitis left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please ensure consistency in the logging format by standardizing the capitalization. The log entries should either be in all uppercase or all lowercase to avoid confusion.

For example:

[exareme2-LOCALWORKER] vs. [exareme2-localworker]
[exareme2-CONTROLLER] vs. [exareme2-controller]

Currently, some logs use [exareme2-CONTROLLER] while others use [exareme2-controller]. Please decide on one convention (all caps or all lowercase) and apply it consistently across all logs.

exareme2/algorithms/exareme2/kmeans.py Show resolved Hide resolved
exareme2/algorithms/exareme2/udfgen/udfio.py Outdated Show resolved Hide resolved
exareme2/algorithms/flower/__init__.py Outdated Show resolved Hide resolved
exareme2/worker/utils/celery_app.py Show resolved Hide resolved
@ThanKarab ThanKarab force-pushed the dev/logging_refactoring_for_elk branch from fece1a8 to 1ff8a53 Compare November 1, 2024 11:45
@ThanKarab
Logs refactoring to fit ELASTIC stack
388bf15 
FEDERATION env variable is now required in both controller and worker services.
NODE_IDENTIFIER env variable is needed in the controller service.
All loggers now have the same formatter.
Added node identifier to all loggers (controller included)
Refactored flower loggers to use the framework log level of the service.
Minor log removals.
@ThanKarab ThanKarab force-pushed the dev/logging_refactoring_for_elk branch from 1ff8a53 to 388bf15 Compare November 1, 2024 14:02
@ThanKarab ThanKarab merged commit ad94560 into master Nov 1, 2024
7 checks passed
@ThanKarab ThanKarab deleted the dev/logging_refactoring_for_elk branch November 1, 2024 15:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@KFilippopolitis KFilippopolitis KFilippopolitis approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@ThanKarab @KFilippopolitis