After talking with SREs looking for help and guidance (https://wikimedia.slack.com/archives/C055QGPTC69/p1725291450945039), we have some extra context about this task:

• We should instrument our code using prom-client
  • We can implement the default metrics and we can also add others
• We have to build a specific endpoint for metrics. Something like /metrics (using same port as the webapp or a different one)
• At this moment monitoring is disabled in the MPIC helmfile for both environments, staging and production
• There is already a grafana dashboard that shows some metrics for MPIC through envoy: https://grafana.wikimedia.org/d/b1jttnFMz/envoy-telemetry-k8s?forceLogin&orgId=1&var-app=All&var-datasource=thanos&var-destination=All&var-kubernetes_namespace=mpic&var-prometheus=k8s-dse&var-site=eqiad
  • We will need a separate dashboard if we add any additional metric using prom-client
  • It's easy to copy items from an existing dashboard to create a new one and we can also use Grizzly to define a new one as code (Observability team could help us with this)

@Sfaci @cjming Related MR for prom-client setup. The things setup here as per discussion
prom-client.collectDefaultMetrics() function will automatically collect standard metrics like CPU, memory usage, and more.
duration of each HTTP request and histogram.
/metrics endpoint

Change #1070649 had a related patch set uploaded (by Santiago Faci; author: Santiago Faci):

[operations/deployment-charts@master] Metrics Platform Instrument Configurator: Enabling prometheus monitoring for MPIC

https://gerrit.wikimedia.org/r/1070649

@Sfaci @cjming Public dashboard is setup here https://grafana-rw.wikimedia.org/d/ee2057f3-eb34-45a7-a48b-489e3ff0b2ec/mpic?orgId=1

Just for tracking purposes I will put here the MR where we have implemented the prometheus client library change for MPIC : https://gitlab.wikimedia.org/repos/data-engineering/mpic/-/merge_requests/96. It's already merged

Change #1070649 merged by jenkins-bot:

[operations/deployment-charts@master] Metrics Platform Instrument Configurator: Enabling prometheus monitoring for MPIC

https://gerrit.wikimedia.org/r/1070649

Change #1070869 had a related patch set uploaded (by Santiago Faci; author: Santiago Faci):

[operations/deployment-charts@master] MPIC: Deploying to staging a new release (v0.1.4)

https://gerrit.wikimedia.org/r/1070869

Change #1070869 merged by jenkins-bot:

[operations/deployment-charts@master] MPIC: Deploying to staging a new release (v0.1.4)

https://gerrit.wikimedia.org/r/1070869

Monitoring code is already implemented and deploy and the dashboard is already alive, but some tuning is needed to show properly all the metrics. We have based the current dashboard on an existing one for AQS and metric names are pretty different. At this time we are working on that.

According to some comments that Ben has put after merging https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/1070649, we should move the monitoring configuration from the chart to the helm files for the next deployment. That's also pending.

Current dashboard is available at https://grafana-rw.wikimedia.org/d/ee2057f3-eb34-45a7-a48b-489e3ff0b2ec/mpic?orgId=1
@SGupta-WMF Could you take a look at it? I have been tuning a bit expressions and queries and it seems that everything is working now

After being reviewed we will be able to deploy on production and fix something regarding the monitoring kubernetes configuration according to something that Ben suggested. That change is already prepared (see gerritbot message below)

Change #1070977 had a related patch set uploaded (by Santiago Faci; author: Santiago Faci):

[operations/deployment-charts@master] MPIC: Moving monitoring configuration from chart to helmfiles

https://gerrit.wikimedia.org/r/1070977

Change #1070977 merged by jenkins-bot:

[operations/deployment-charts@master] MPIC: Moving monitoring configuration from chart to helmfiles

https://gerrit.wikimedia.org/r/1070977

Change #1071580 had a related patch set uploaded (by Santiago Faci; author: Santiago Faci):

[operations/deployment-charts@master] MPIC: Deploying a new release v0.1.5 to staging

https://gerrit.wikimedia.org/r/1071580

Change #1071580 merged by jenkins-bot:

[operations/deployment-charts@master] MPIC: Deploying a new release v0.1.5 to staging

https://gerrit.wikimedia.org/r/1071580

@Sfaci The dashboard looks good to me , I largely reviewed the mpic-next service. Thanks!

Cool! Thanks @SGupta-WMF
Next and final step will be deploying MPIC to production to enable there this new feature (and prepare the dashboard for that environment as well). I'll get on with it.

Change #1073152 had a related patch set uploaded (by Santiago Faci; author: Santiago Faci):

[operations/deployment-charts@master] MPIC: New deployment (v0.1.5) to production

https://gerrit.wikimedia.org/r/1073152

Change #1073152 merged by jenkins-bot:

[operations/deployment-charts@master] MPIC: New deployment (v0.1.5) to production

https://gerrit.wikimedia.org/r/1073152

@brouberol Just in case you need to know this, monitoring is already enabled for MPIC on staging and production environments. The dashboard is available https://grafana-rw.wikimedia.org/d/ee2057f3-eb34-45a7-a48b-489e3ff0b2ec/mpic?orgId=1