How to config email alert in using grafana and prometheus-operator

Question

I installed prometheus-operator (include prometheus/alertmanager/grafana) via helm. Then I access Grafana UI and config alert via email. When I click send an email test, I got the message “ SMTP not configured, check your grafana.ini config file’s [smtp] section”

But I don’t know where the grafana.ini to can change in this case.

[root@k8s-master ~]# kubectl get pods --all-namespaces
NAMESPACE     NAME                                                      READY   STATUS    RESTARTS   AGE
kube-system   calico-kube-controllers-5bbc8f45cb-nlqgh                  1/1     Running   1          15h
kube-system   calico-node-lk2j5                                         1/1     Running   1          15h
kube-system   calico-node-v6wzs                                         1/1     Running   1          15h
kube-system   calico-node-zfh5r                                         1/1     Running   1          15h
kube-system   coredns-5c98db65d4-79c2g                                  1/1     Running   1          15h
kube-system   coredns-5c98db65d4-bqj7g                                  1/1     Running   1          15h
kube-system   etcd-k8s-master                                           1/1     Running   1          15h
kube-system   kube-apiserver-k8s-master                                 1/1     Running   1          15h
kube-system   kube-controller-manager-k8s-master                        1/1     Running   2          15h
kube-system   kube-proxy-8qmdt                                          1/1     Running   1          15h
kube-system   kube-proxy-qwgbc                                          1/1     Running   1          15h
kube-system   kube-proxy-vhqjd                                          1/1     Running   1          15h
kube-system   kube-scheduler-k8s-master                                 1/1     Running   1          15h
monitoring    alertmanager-prometheus-operator-alertmanager-0           2/2     Running   3          15h
monitoring    prometheus-operator-grafana-64848fc9bb-dbnwc              2/2     Running   3          15h
monitoring    prometheus-operator-kube-state-metrics-5d46566c59-ck4np   1/1     Running   2          15h
monitoring    prometheus-operator-operator-64dcc7bfc-lpdj6              2/2     Running   2          15h
monitoring    prometheus-operator-prometheus-node-exporter-ns4kg        1/1     Running   1          15h
monitoring    prometheus-operator-prometheus-node-exporter-tdhwq        1/1     Running   2          15h
monitoring    prometheus-operator-prometheus-node-exporter-xt8z9        1/1     Running   2          15h
monitoring    prometheus-prometheus-operator-prometheus-0               3/3     Running   4          15h

Answer 1

You will be able to override this configuration, using helm variables, thanks to alertmanager.config key. This key convert yaml into configuration for alertmanager, so you can use every alertmanager configuration. You should probably also change grafana.ini configuration to configure smtp into grafana (test seems to use that configuration). You can check this configuration in Grafana via "Server admin" > "Settings", search "smtp".

As a reference, you can do something like the following for alertmanager :

    helm upgrade --install prometheus stable/prometheus-operator \
-f helm/prometheus-operator.yml \
-f helm/grafana-custom.staging.yml \
--set-string alertmanager.config.global.smtp_smarthost="my.smtp.tld:465" \
--set-string alertmanager.config.global.smtp_auth_username="[email protected]" \
--set-string alertmanager.config.global.smtp_from="[email protected]" \
--set-string alertmanager.config.global.smtp_auth_password="MyAmazingPassword" \
--set-string grafana.'grafana\.ini'.smtp.enabled=true \
--set-string grafana.'grafana\.ini'.smtp.host="my.smtp.tld:465" \
--set-string grafana.'grafana\.ini'.smtp.from_address="[email protected]" \
--set-string grafana.'grafana\.ini'.smtp.user="[email protected]" \
--set-string grafana.'grafana\.ini'.smtp.password="MyAmazingPassword"