Implement centralised logging

[lukebond]

We want all logs for all services to be tail-able together and individually, from the command line and also displayable within the UI. Searching would also be good.

From @sublimino in #19:

one-command cluster monitoring (i.e. journal -f on all units, got some fleet jiggerypokery to do this as there are silly TTY complications)

Is https://github.com/gliderlabs/logspout an option?

[lukebond]

I'd like to settle on a solution here. logspout looks good to me but not sure where best to send the logs.

@sublimino: not sure how the choice of tool (eg. logspout) relates to your above-mentioned "fleet jiggerypokery". can you outline what you have in mind?

[sublimino]

logspout -> elasticsearch-based kibana is the nicest way of doing things,
my fleet jiggerypokery is to tail all the services as they start and
interpolate them in the console. i.e.

$ pazctl cluster-log
[host-1] [service-a] bing
[host-2] [service-b] bang
[host-1] [service-c] bong

(this is essentially a sneaky melding of the output of fleetctl journal -f service-a fleetctl journal -f service-b fleetctl journal -f service-c)

This would be useful for cluster startup when developing, for example (much
quicker feedback than going through kibana).

An extant paz cluster would probably not need that debugging as operational
issues would be accessible via kibana.

^^^ This is a reflection of my bias for CLI-based tools

On 9 April 2015 at 11:27, Luke Bond [email protected] wrote:

I'd like to settle on a solution here. logspout looks good to me but not
sure where best to send the logs.

@sublimino https://github.com/sublimino: not sure how the choice of
tool (eg. logspout) relates to your above-mentioned "fleet jiggerypokery".
can you outline what you have in mind?

—
Reply to this email directly or view it on GitHub
#31 (comment).

[lukebond]

Thanks.

Sounds like that is part of the CLI and not "centralised logging". If you
agree then we should create an issue in the relevant CLI repository.

Regarding the centralised solution, I'd say we have the following work to
do:

• create unit files in the "paz" repo for kibana etc (is this one unit or
  multiple units?)
• ensure these units get installed by default alongside paz
• modify cloud config to configure log stash on each node
• add logging tab to the UI somewhere

Sound about right?

On Thursday, 9 April 2015, Andrew Martin [email protected] wrote:

logspout -> elasticsearch-based kibana is the nicest way of doing things,
my fleet jiggerypokery is to tail all the services as they start and
interpolate them in the console. i.e.

$ pazctl cluster-log
[host-1] [service-a] bing
[host-2] [service-b] bang
[host-1] [service-c] bong

(this is essentially a sneaky melding of the output of fleetctl journal -f service-a fleetctl journal -f service-b fleetctl journal -f service-c)

This would be useful for cluster startup when developing, for example (much
quicker feedback than going through kibana).

An extant paz cluster would probably not need that debugging as operational
issues would be accessible via kibana.

^^^ This is a reflection of my bias for CLI-based tools

On 9 April 2015 at 11:27, Luke Bond <[email protected]
javascript:_e(%7B%7D,'cvml','[email protected]');> wrote:

I'd like to settle on a solution here. logspout looks good to me but not
sure where best to send the logs.

@sublimino https://github.com/sublimino: not sure how the choice of
tool (eg. logspout) relates to your above-mentioned "fleet
jiggerypokery".
can you outline what you have in mind?

—
Reply to this email directly or view it on GitHub
#31 (comment).

—
Reply to this email directly or view it on GitHub
#31 (comment).

[sublimino]

a) Agreed!
b) looks good

Probably worth having individual units for clarity/reporting, but no strong feels either way.

[lukebond]

Thanks @sublimino

I'll split up into separate issues.