Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(Kubernetes): Add JMX exporter containers to all DataHub components #2251

Merged
merged 9 commits into from
Mar 31, 2021
Merged

feat(Kubernetes): Add JMX exporter containers to all DataHub components #2251

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
9 commits
Select commit Hold shift + click to select a range
30f6088
Add JMX_OPTS and JAVA_OPTS environment variables to the docker images
Mar 17, 2021
d6bd0b3
Add JMX Exporter to datahub-frontend
Mar 17, 2021
bf81011
Add JMX Exporter to datahub-gms
Mar 18, 2021
b822ca2
Add JMX Exporter to datahub-mae-consumer
Mar 18, 2021
7e72fdd
Add JMX Exporter to datahub-mce-consumer
Mar 18, 2021
04dce79
Fix volume definition
Mar 18, 2021
7d9632e
Add entry for jmx exporter activation to the READMEs. Comment out the…
Mar 18, 2021
2578a2c
Remove unused variable in values.yaml
Mar 18, 2021
57d251c
jmxexporter container as an helm library chart
Mar 25, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions contrib/kubernetes/datahub/charts/datahub-frontend/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,7 @@ Current chart version is `0.2.0`
|-----|------|---------|-------------|
| affinity | object | `{}` | |
| datahub.play.mem.buffer.size | string | `"10MB"` | |
| exporters.jmx.enabled | boolean | false | |
| extraEnvs | Extra [environment variables][] which will be appended to the `env:` definition for the container | `[]` |
| extraVolumes | Templatable string of additional `volumes` to be passed to the `tpl` function | "" |
| extraVolumeMounts | Templatable string of additional `volumeMounts` to be passed to the `tpl` function | "" |
Expand Down
17 changes: 17 additions & 0 deletions contrib/kubernetes/datahub/charts/datahub-frontend/templates/config-jmx-exporter.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,17 @@
{{- if .Values.exporters.jmx.enabled }}
apiVersion: v1
kind: ConfigMap
metadata:
name: {{ include "datahub-frontend.fullname" . }}-config-jmx-exporter
labels:
{{- include "datahub-frontend.labels" . | nindent 4 }}
data:
config.yml: |-
hostPort: localhost:{{ .Values.env.JMXPORT }}
lowercaseOutputName: {{ .Values.exporters.jmx.config.lowercaseOutputName }}
lowercaseOutputLabelNames: {{ .Values.exporters.jmx.config.lowercaseOutputLabelNames }}
rules:
{{ .Values.exporters.jmx.config.rules | toYaml | indent 6 }}
ssl: false
startDelaySeconds: {{ .Values.exporters.jmx.config.startDelaySeconds }}
{{- end }}
8 changes: 8 additions & 0 deletions contrib/kubernetes/datahub/charts/datahub-frontend/templates/deployment.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -29,6 +29,11 @@ spec:
{{- if .Values.extraVolumes }}
{{ toYaml .Values.extraVolumes | nindent 8 }}
{{- end }}
{{- if .Values.exporters.jmx.enabled }}
- name: config-jmx-exporter
configMap:
name: {{ include "datahub-frontend.fullname" . }}-config-jmx-exporter
{{- end }}
initContainers:
{{- if .Values.extraInitContainers }}
{{- .Values.extraInitContainers | toYaml | nindent 6 }}
Expand Down Expand Up @@ -74,6 +79,9 @@ spec:
{{- end }}
resources:
{{- toYaml .Values.resources | nindent 12 }}

{{- include "datahub-jmxexporter.container" . }}

{{- with .Values.nodeSelector }}
nodeSelector:
{{- toYaml . | nindent 8 }}
Expand Down
46 changes: 46 additions & 0 deletions contrib/kubernetes/datahub/charts/datahub-frontend/values.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -89,6 +89,52 @@ tolerations: []

affinity: {}

env:
JMXPORT: 1099

exporters:
jmx:
enabled: false
# image:
# repository: bitnami/jmx-exporter
# tag: 0.15.0
# pullPolicy: IfNotPresent
# config:
# lowercaseOutputName: true
# lowercaseOutputLabelNames: true
# rules:
# - pattern: ".*"
# startDelaySeconds: 30
# env: {}
# resources: {}
# path: /metrics
# ports:
# jmxxp:
# containerPort: 5556
# protocol: TCP
# livenessProbe:
# httpGet:
# path: /metrics
# port: jmxxp
# initialDelaySeconds: 30
# periodSeconds: 15
# timeoutSeconds: 60
# failureThreshold: 8
# successThreshold: 1
# readinessProbe:
# httpGet:
# path: /metrics
# port: jmxxp
# initialDelaySeconds: 30
# periodSeconds: 15
# timeoutSeconds: 60
# failureThreshold: 8
# successThreshold: 1
# serviceMonitor:
# interval: 30s
# scrapeTimeout: 30s
# scheme: http

datahub:
play:
mem:
Expand Down
1 change: 1 addition & 0 deletions contrib/kubernetes/datahub/charts/datahub-gms/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,7 @@ Current chart version is `0.2.0`
| Key | Type | Default | Description |
|-----|------|---------|-------------|
| affinity | object | `{}` | |
| exporters.jmx.enabled | boolean | false | |
| extraEnvs | Extra [environment variables][] which will be appended to the `env:` definition for the container | `[]` |
| extraVolumes | Templatable string of additional `volumes` to be passed to the `tpl` function | "" |
| extraVolumeMounts | Templatable string of additional `volumeMounts` to be passed to the `tpl` function | "" |
Expand Down
17 changes: 17 additions & 0 deletions contrib/kubernetes/datahub/charts/datahub-gms/templates/config-jmx-exporter.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,17 @@
{{- if .Values.exporters.jmx.enabled }}
apiVersion: v1
kind: ConfigMap
metadata:
name: {{ include "datahub-gms.fullname" . }}-config-jmx-exporter
labels:
{{- include "datahub-gms.labels" . | nindent 4 }}
data:
config.yml: |-
hostPort: localhost:{{ .Values.env.JMXPORT }}
lowercaseOutputName: {{ .Values.exporters.jmx.config.lowercaseOutputName }}
lowercaseOutputLabelNames: {{ .Values.exporters.jmx.config.lowercaseOutputLabelNames }}
rules:
{{ .Values.exporters.jmx.config.rules | toYaml | indent 6 }}
ssl: false
startDelaySeconds: {{ .Values.exporters.jmx.config.startDelaySeconds }}
{{- end }}
8 changes: 8 additions & 0 deletions contrib/kubernetes/datahub/charts/datahub-gms/templates/deployment.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -36,6 +36,11 @@ spec:
defaultMode: 256
secretName: {{ .Values.global.credentialsAndCertsSecrets.name }}
{{- end }}
{{- if .Values.exporters.jmx.enabled }}
- name: config-jmx-exporter
configMap:
name: {{ include "datahub-gms.fullname" . }}-config-jmx-exporter
{{- end }}
{{- if .Values.extraVolumes }}
{{ toYaml .Values.extraVolumes | nindent 8 }}
{{- end }}
Expand Down Expand Up @@ -114,6 +119,9 @@ spec:
{{- end }}
resources:
{{- toYaml .Values.resources | nindent 12 }}

{{- include "datahub-jmxexporter.container" . }}

{{- with .Values.nodeSelector }}
nodeSelector:
{{- toYaml . | nindent 8 }}
Expand Down
45 changes: 45 additions & 0 deletions contrib/kubernetes/datahub/charts/datahub-gms/values.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -88,6 +88,51 @@ tolerations: []

affinity: {}

env:
JMXPORT: 1099

exporters:
jmx:
enabled: false
# image:
# repository: bitnami/jmx-exporter
# tag: 0.15.0
# pullPolicy: IfNotPresent
# config:
# lowercaseOutputName: true
# rules:
# - pattern: ".*"
# startDelaySeconds: 30
# env: {}
# resources: {}
# path: /metrics
# ports:
# jmxxp:
# containerPort: 5556
# protocol: TCP
# livenessProbe:
# httpGet:
# path: /metrics
# port: jmxxp
# initialDelaySeconds: 30
# periodSeconds: 15
# timeoutSeconds: 60
# failureThreshold: 8
# successThreshold: 1
# readinessProbe:
# httpGet:
# path: /metrics
# port: jmxxp
# initialDelaySeconds: 30
# periodSeconds: 15
# timeoutSeconds: 60
# failureThreshold: 8
# successThreshold: 1
# serviceMonitor:
# interval: 30s
# scrapeTimeout: 30s
# scheme: http

#This section is useful if we are installing this chart separately for testing
# helm install datahub-gms datahub-gms/
global:
Expand Down
21 changes: 21 additions & 0 deletions contrib/kubernetes/datahub/charts/datahub-jmxexporter/Chart.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,21 @@
apiVersion: v2
name: datahub-jmxexporter
description: A Helm chart for Kubernetes

# A chart can be either an 'application' or a 'library' chart.
#
# Application charts are a collection of templates that can be packaged into versioned archives
# to be deployed.
#
# Library charts provide useful utilities or functions for the chart developer. They're included as
# a dependency of application charts to inject those utilities and functions into the rendering
# pipeline. Library charts do not define any templates and therefore cannot be deployed.
type: library

# This is the chart version. This version number should be incremented each time you make changes
# to the chart and its templates, including the app version.
version: 0.2.1

# This is the version number of the application being deployed. This version number should be
# incremented each time you make changes to the application.
appVersion: 0.3.1
32 changes: 32 additions & 0 deletions contrib/kubernetes/datahub/charts/datahub-jmxexporter/templates/_container.tpl
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,32 @@
{{- define "datahub-jmxexporter.container" -}}
{{- if .Values.exporters.jmx.enabled }}
- name: jmx-exporter
image: "{{ .Values.exporters.jmx.image.repository }}:{{ .Values.exporters.jmx.image.tag }}"
imagePullPolicy: {{ .Values.exporters.jmx.image.pullPolicy }}
args: ["{{ .Values.exporters.jmx.ports.jmxxp.containerPort}}", "/opt/jmx_exporter/config.yml"]
ports:
{{- range $key, $port := .Values.exporters.jmx.ports }}
- name: {{ $key }}
{{ toYaml $port | indent 14 }}
{{- end }}
livenessProbe:
{{ toYaml .Values.exporters.jmx.livenessProbe | indent 12 }}
readinessProbe:
{{ toYaml .Values.exporters.jmx.readinessProbe | indent 12 }}
env:
- name: SERVICE_PORT
value: {{ .Values.exporters.jmx.ports.jmxxp.containerPort | quote }}
{{- with .Values.exporters.jmx.env }}
{{- range $key, $value := . }}
- name: {{ $key | upper | replace "." "_" }}
value: {{ $value | quote }}
{{- end }}
{{- end }}
resources:
{{ toYaml .Values.exporters.jmx.resources | indent 12 }}
volumeMounts:
- name: config-jmx-exporter
mountPath: /opt/jmx_exporter/config.yml
subPath: config.yml
{{- end }}
{{- end }}
1 change: 1 addition & 0 deletions contrib/kubernetes/datahub/charts/datahub-mae-consumer/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,7 @@ Current chart version is `0.2.0`
| Key | Type | Default | Description |
|-----|------|---------|-------------|
| affinity | object | `{}` | |
| exporters.jmx.enabled | boolean | false | |
| extraEnvs | Extra [environment variables][] which will be appended to the `env:` definition for the container | `[]` |
| extraVolumes | Templatable string of additional `volumes` to be passed to the `tpl` function | "" |
| extraVolumeMounts | Templatable string of additional `volumeMounts` to be passed to the `tpl` function | "" |
Expand Down
17 changes: 17 additions & 0 deletions contrib/kubernetes/datahub/charts/datahub-mae-consumer/templates/config-jmx-exporter.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,17 @@
{{- if .Values.exporters.jmx.enabled }}
apiVersion: v1
kind: ConfigMap
metadata:
name: {{ include "datahub-mae-consumer.fullname" . }}-config-jmx-exporter
labels:
{{- include "datahub-mae-consumer.labels" . | nindent 4 }}
data:
config.yml: |-
hostPort: localhost:{{ .Values.env.JMXPORT }}
lowercaseOutputName: {{ .Values.exporters.jmx.config.lowercaseOutputName }}
lowercaseOutputLabelNames: {{ .Values.exporters.jmx.config.lowercaseOutputLabelNames }}
rules:
{{ .Values.exporters.jmx.config.rules | toYaml | indent 6 }}
ssl: false
startDelaySeconds: {{ .Values.exporters.jmx.config.startDelaySeconds }}
{{- end }}
8 changes: 8 additions & 0 deletions contrib/kubernetes/datahub/charts/datahub-mae-consumer/templates/deployment.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -36,6 +36,11 @@ spec:
defaultMode: 256
secretName: {{ .Values.global.credentialsAndCertsSecrets.name }}
{{- end }}
{{- if .Values.exporters.jmx.enabled }}
- name: config-jmx-exporter
configMap:
name: {{ include "datahub-mae-consumer.fullname" . }}-config-jmx-exporter
{{- end }}
{{- if .Values.extraVolumes }}
{{ toYaml .Values.extraVolumes | nindent 8 }}
{{- end }}
Expand Down Expand Up @@ -97,6 +102,9 @@ spec:
{{- end }}
resources:
{{- toYaml .Values.resources | nindent 12 }}

{{- include "datahub-jmxexporter.container" . }}

{{- with .Values.nodeSelector }}
nodeSelector:
{{- toYaml . | nindent 8 }}
Expand Down
45 changes: 45 additions & 0 deletions contrib/kubernetes/datahub/charts/datahub-mae-consumer/values.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -89,6 +89,51 @@ tolerations: []

affinity: {}

env:
JMXPORT: 1099

exporters:
jmx:
enabled: false
# image:
# repository: bitnami/jmx-exporter
# tag: 0.15.0
# pullPolicy: IfNotPresent
# config:
# lowercaseOutputName: true
# rules:
# - pattern: ".*"
# startDelaySeconds: 30
# env: {}
# resources: {}
# path: /metrics
# ports:
# jmxxp:
# containerPort: 5556
# protocol: TCP
# livenessProbe:
# httpGet:
# path: /metrics
# port: jmxxp
# initialDelaySeconds: 30
# periodSeconds: 15
# timeoutSeconds: 60
# failureThreshold: 8
# successThreshold: 1
# readinessProbe:
# httpGet:
# path: /metrics
# port: jmxxp
# initialDelaySeconds: 30
# periodSeconds: 15
# timeoutSeconds: 60
# failureThreshold: 8
# successThreshold: 1
# serviceMonitor:
# interval: 30s
# scrapeTimeout: 30s
# scheme: http

global:
elasticsearch:
host: "elasticsearch"
Expand Down
1 change: 1 addition & 0 deletions contrib/kubernetes/datahub/charts/datahub-mce-consumer/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,7 @@ Current chart version is `0.2.0`
| Key | Type | Default | Description |
|-----|------|---------|-------------|
| affinity | object | `{}` | |
| exporters.jmx.enabled | boolean | false | |
| extraEnvs | Extra [environment variables][] which will be appended to the `env:` definition for the container | `[]` |
| extraVolumes | Templatable string of additional `volumes` to be passed to the `tpl` function | "" |
| extraVolumeMounts | Templatable string of additional `volumeMounts` to be passed to the `tpl` function | "" |
Expand Down
17 changes: 17 additions & 0 deletions contrib/kubernetes/datahub/charts/datahub-mce-consumer/templates/config-jmx-exporter.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,17 @@
{{- if .Values.exporters.jmx.enabled }}
apiVersion: v1
kind: ConfigMap
metadata:
name: {{ include "datahub-mce-consumer.fullname" . }}-config-jmx-exporter
labels:
{{- include "datahub-mce-consumer.labels" . | nindent 4 }}
data:
config.yml: |-
hostPort: localhost:{{ .Values.env.JMXPORT }}
lowercaseOutputName: {{ .Values.exporters.jmx.config.lowercaseOutputName }}
lowercaseOutputLabelNames: {{ .Values.exporters.jmx.config.lowercaseOutputLabelNames }}
rules:
{{ .Values.exporters.jmx.config.rules | toYaml | indent 6 }}
ssl: false
startDelaySeconds: {{ .Values.exporters.jmx.config.startDelaySeconds }}
{{- end }}
Loading