Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Gathering Docker log files with fluentd, sending to Elasticsearch, viewing with Kibana #1562

Merged
merged 0 commits into from
Oct 6, 2014
Merged

Gathering Docker log files with fluentd, sending to Elasticsearch, viewing with Kibana #1562

merged 0 commits into from
Oct 6, 2014

Conversation

satnam6502
Copy link
Contributor

This pull request includes changes to the Salt setup to add a fluentd instance to each node. This fluentd instance will try to send the logs it has gather from the local Dockerfiles to port 9200 of the enclosing machine (e.g. a GCE VM).

The contrib/loggimng directory has an example scenario that contains a simple web server (pod specification, service specification), Elasticsearch (pod and service specification) and Kibana (pod and service specification). It also contains the Dockerfile used for the fluentd manifest that is deployed by salt.

@satnam6502 satnam6502 changed the title Salt specification for adding fluentd to each node. Gathering Docker log files with fluentd, sending to Elasticsearch, viewing with Kibana Oct 3, 2014
@thockin
Copy link
Member

thockin commented Oct 3, 2014

I'll review in depth later, but I am not sure we want to push fluentd to
all nodes in all cases. Not yet, anyway.

On Thu, Oct 2, 2014 at 11:43 PM, Satnam Singh [email protected]
wrote:

This pull request includes changes to the Salt setup to add a fluentd
instance to each node. This fluentd instance will try to send the logs it
has gather from the local Dockerfiles to port 9200 of the enclosing machine
(e.g. a GCE VM). The assumption is at some point an instance of
Elasticsearch at port 9200 will be available (e.g. by running a Kubernetes

service).

You can merge this Pull Request by running

git pull https://github.com/satnam6502/kubernetes master

Or view, comment on, or merge it at:

#1562
Commit Summary

  • Salt specification for adding fluentd to each node.

File Changes

Patch Links:

Reply to this email directly or view it on GitHub
#1562.

@satnam6502
Copy link
Contributor Author

I was thinking about opening a discussion thread about this.
In the current setup I am not sure how we specify an "optional" piece of
base level functionality.

I was also wondering how to plug in a different collector e.g. using other
log ingestion APIs instead of using Elasticsearch.
Simple idea #1: write a program that listens to port 9200 (as
Elasticsearch), consumes the JSON log messages, and then does the Right
Thing with them (e.g. send them to some database etc.).

Satnam

On 3 October 2014 08:54, Tim Hockin [email protected] wrote:

I'll review in depth later, but I am not sure we want to push fluentd to
all nodes in all cases. Not yet, anyway.

On Thu, Oct 2, 2014 at 11:43 PM, Satnam Singh [email protected]
wrote:

This pull request includes changes to the Salt setup to add a fluentd
instance to each node. This fluentd instance will try to send the logs it
has gather from the local Dockerfiles to port 9200 of the enclosing
machine
(e.g. a GCE VM). The assumption is at some point an instance of
Elasticsearch at port 9200 will be available (e.g. by running a
Kubernetes

service).

You can merge this Pull Request by running

git pull https://github.com/satnam6502/kubernetes master

Or view, comment on, or merge it at:

#1562
Commit Summary

  • Salt specification for adding fluentd to each node.

File Changes

Patch Links:

Reply to this email directly or view it on GitHub
#1562.

—
Reply to this email directly or view it on GitHub
#1562 (comment)
.

Satnam Singh
Email: [email protected] [email protected]
Cell (USA): +1 408 656 4590
Office: +1 650 214 4488
Mobile (UK): +44 7979 648412

@jbeda
Copy link
Contributor

jbeda commented Oct 3, 2014

We can make this optional by defining a saltstack pillar value and conditionally including it or not in top.sls based on that pillar. We would then be able to create that pillar value or not based on env variables/options when kube-up is called. This is all pretty salt specific though. The mechanism for someone using fleet or something to install kubernetes will be different.

@satnam6502
Copy link
Contributor Author

Other options to consider:

  1. Initially, just have a fluentd image as part of every pod specification that wants to have logging. This would log, as before, to an Elasticsearch Kubernetes service. Clunky, and it involves redundant instances of fluentd when there is more than one pod scheduled on a node that needs logging. We would also have to adjust how the fluentd config is set up so each fluentd instances gathers only the logs of the peer images in the pod (which no doubt will involve further yuk). Can't say I am very excited about this.
  2. We could still fire up a fluentd container on each node as part of the manifest but have it "turned off by default". Then if any pod needs it then fluentd can start collecting. Perhaps the user can specify this by including an image we provide which -- when run -- simply actives fluentd and then goes away. Then we only pay the overhead of fluentd when we need it. For bonus points we could detect when there are no pods running that need logging --- in which casse fluent can be turned off again. All of this is of course from the perspective of logging user applications. We should also think about how we want to collect the logs of the actual Kebernetes infrastructure.

@thockin
Copy link
Member

thockin commented Oct 5, 2014

Turning on fluentd is really a cluster-admin decision - it isn't really
localized unless you put it in every pod, as you say, and it is clunky (as
you say). It's sort of all or nothing.

Given that, I really think it's something that cluster-setup should offer,
but not demand, or else we go the route of scheduling "one on every
machine" pods (which I argued against) - which would make this easier to
add to a cluster ex post facto.

On Fri, Oct 3, 2014 at 10:02 AM, Satnam Singh [email protected]
wrote:

Other options to consider:

Initially, just have a fluentd image as part of every pod
specification that want to have logging. This would log, as before, the an
Elasticsearch Kubernetes *service. Clunky, and it involves
redundant instances of fluentd when there is more than one pod scheduled on
a node that needs logging. We would also have to adjust how the fluentd
config is set up so each fluentd instances gathers only the logs of the
peer images in the pod (which no doubt will involve further yuk). Can't say
I am very excited about this.
2.

We could still fire up a fluentd container on each node as part of the
manifest but have it "turned off by default". Then if any pod need it then
fluentd can start collecting. Perhaps the user can specify this by
including an image we provide which -- when run -- simply actives fluentd
and then goes away. Then we only pay the overhead of fluentd when we need
it. For bonus points we could detect when there are no pods running that
need logging --- in which casse fluent can be turned off again. All of this
is of course from the perspective of logging user applications. We should
also think about how we want to collected the logs of the actual Kebernetes
infrastructure.

Reply to this email directly or view it on GitHub
#1562 (comment)
.

@satnam6502
Copy link
Contributor Author

I suspect you are rubbing salt into my wounds.

On Sat, Oct 4, 2014 at 10:20 PM, Tim Hockin [email protected]
wrote:

Turning on fluentd is really a cluster-admin decision - it isn't really
localized unless you put it in every pod, as you say, and it is clunky (as
you say). It's sort of all or nothing.

Given that, I really think it's something that cluster-setup should offer,
but not demand, or else we go the route of scheduling "one on every
machine" pods (which I argued against) - which would make this easier to
add to a cluster ex post facto.

On Fri, Oct 3, 2014 at 10:02 AM, Satnam Singh [email protected]
wrote:

Other options to consider:

Initially, just have a fluentd image as part of every pod
specification that want to have logging. This would log, as before, the
an
Elasticsearch Kubernetes *service. Clunky, and it involves
redundant instances of fluentd when there is more than one pod scheduled
on
a node that needs logging. We would also have to adjust how the fluentd
config is set up so each fluentd instances gathers only the logs of the
peer images in the pod (which no doubt will involve further yuk). Can't
say
I am very excited about this.
2.

We could still fire up a fluentd container on each node as part of the
manifest but have it "turned off by default". Then if any pod need it
then
fluentd can start collecting. Perhaps the user can specify this by
including an image we provide which -- when run -- simply actives fluentd
and then goes away. Then we only pay the overhead of fluentd when we need
it. For bonus points we could detect when there are no pods running that
need logging --- in which casse fluent can be turned off again. All of
this
is of course from the perspective of logging user applications. We should
also think about how we want to collected the logs of the actual
Kebernetes
infrastructure.

Reply to this email directly or view it on GitHub
<
#1562 (comment)

.

—
Reply to this email directly or view it on GitHub
#1562 (comment)
.

@thockin
Copy link
Member

thockin commented Oct 6, 2014

I don't know if you're actually unhappy with that reply or not - too dry
for me? :) or maybe you're making a salt joke?

On Sun, Oct 5, 2014 at 3:06 AM, Satnam Singh [email protected]
wrote:

I suspect you are rubbing salt into my wounds.

On Sat, Oct 4, 2014 at 10:20 PM, Tim Hockin [email protected]
wrote:

Turning on fluentd is really a cluster-admin decision - it isn't really
localized unless you put it in every pod, as you say, and it is clunky
(as
you say). It's sort of all or nothing.

Given that, I really think it's something that cluster-setup should
offer,
but not demand, or else we go the route of scheduling "one on every
machine" pods (which I argued against) - which would make this easier to
add to a cluster ex post facto.

On Fri, Oct 3, 2014 at 10:02 AM, Satnam Singh [email protected]

wrote:

Other options to consider:

Initially, just have a fluentd image as part of every pod
specification that want to have logging. This would log, as before,
the
an
Elasticsearch Kubernetes *service. Clunky, and it involves
redundant instances of fluentd when there is more than one pod
scheduled
on
a node that needs logging. We would also have to adjust how the
fluentd
config is set up so each fluentd instances gathers only the logs of
the
peer images in the pod (which no doubt will involve further yuk).
Can't
say
I am very excited about this.
2.

We could still fire up a fluentd container on each node as part of the
manifest but have it "turned off by default". Then if any pod need it
then
fluentd can start collecting. Perhaps the user can specify this by
including an image we provide which -- when run -- simply actives
fluentd
and then goes away. Then we only pay the overhead of fluentd when we
need
it. For bonus points we could detect when there are no pods running
that
need logging --- in which casse fluent can be turned off again. All of
this
is of course from the perspective of logging user applications. We
should
also think about how we want to collected the logs of the actual
Kebernetes
infrastructure.

Reply to this email directly or view it on GitHub
<

#1562 (comment)

.

Reply to this email directly or view it on GitHub
<
https://github.com/GoogleCloudPlatform/kubernetes/pull/1562#issuecomment-57927055>

.

Reply to this email directly or view it on GitHub
#1562 (comment)
.

@satnam6502
Copy link
Contributor Author

I am always looking for a cheap shot.

S

On Mon, Oct 6, 2014 at 11:34 AM, Tim Hockin [email protected]
wrote:

I don't know if you're actually unhappy with that reply or not - too dry
for me? :) or maybe you're making a salt joke?

On Sun, Oct 5, 2014 at 3:06 AM, Satnam Singh [email protected]
wrote:

I suspect you are rubbing salt into my wounds.

On Sat, Oct 4, 2014 at 10:20 PM, Tim Hockin [email protected]
wrote:

Turning on fluentd is really a cluster-admin decision - it isn't really
localized unless you put it in every pod, as you say, and it is clunky
(as
you say). It's sort of all or nothing.

Given that, I really think it's something that cluster-setup should
offer,
but not demand, or else we go the route of scheduling "one on every
machine" pods (which I argued against) - which would make this easier
to
add to a cluster ex post facto.

On Fri, Oct 3, 2014 at 10:02 AM, Satnam Singh <
[email protected]>

wrote:

Other options to consider:

Initially, just have a fluentd image as part of every pod
specification that want to have logging. This would log, as before,
the
an
Elasticsearch Kubernetes *service. Clunky, and it involves
redundant instances of fluentd when there is more than one pod
scheduled
on
a node that needs logging. We would also have to adjust how the
fluentd
config is set up so each fluentd instances gathers only the logs of
the
peer images in the pod (which no doubt will involve further yuk).
Can't
say
I am very excited about this.
2.

We could still fire up a fluentd container on each node as part of
the
manifest but have it "turned off by default". Then if any pod need it
then
fluentd can start collecting. Perhaps the user can specify this by
including an image we provide which -- when run -- simply actives
fluentd
and then goes away. Then we only pay the overhead of fluentd when we
need
it. For bonus points we could detect when there are no pods running
that
need logging --- in which casse fluent can be turned off again. All
of
this
is of course from the perspective of logging user applications. We
should
also think about how we want to collected the logs of the actual
Kebernetes
infrastructure.

Reply to this email directly or view it on GitHub
<

#1562 (comment)

.

Reply to this email directly or view it on GitHub
<

#1562 (comment)

.

Reply to this email directly or view it on GitHub
<
#1562 (comment)

.

—
Reply to this email directly or view it on GitHub
#1562 (comment)
.

@satnam6502
Copy link
Contributor Author

I will abandon this PR and replace it with another one that has a ELK loggic scenario where there is a fluentd in the pod specification rather than baking it in with salt.

@brendandburns brendandburns merged commit e26bdd0 into kubernetes:master Oct 6, 2014
seans3 pushed a commit to seans3/kubernetes that referenced this pull request Apr 10, 2019
@k8s-ci-robot
Merge pull request kubernetes#1562 from cofyc/typofix
1045006 
fix typos in storage docs
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees

@thockin thockin

Labels
area/introspection
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
5 participants
@satnam6502 @thockin @jbeda @brendandburns @bgrant0607