In T372648#10073513, @SLyngshede-WMF wrote:

It's probably enough to bump the default timeout as a quick fix. I'll take a look.

Deployed with:

helmfile -e staging-codfw -l name=cfssl-issuer-crds -l name=cfssl-issuer -i apply --context 5

@Ottomata could you please verify the above/patches to unblock the hardware refresh of kafka-main?

To unblock the kafka hardware replacements, I've added an alternative readiness probe to the chart which will be used when .Values.main_app.conf.test_events evaluates to false. It just does a GET /_info which is what we do for other servicerunner services as well IMHO. AIUI schema loading happens async anyways, so the service should be ready as soon as the routing is set up, right?

For the record: The CoreDNS Pods are, like all other Pods, reachable by their Pod IP. So going through a debug Pod to resolve the PTRs is not really necessary.
Regarding the IPs of Pods not listed in Endpoints: These could be added to CoreDNS via the https://github.com/coredns/kubepods plugin as well.

All of these have been reimaged with raid10-6dev now, thanks!

From T371423#10068548 I did:

All of these have been reimaged with raid10-6dev now, thanks!

In T372445#10064393, @colewhite wrote:

I can't recommend querying OpenSearch directly. Logs-api was made for services needing log integration (e.g. deploys), and not really for one-off queries. Browsing/filtering/aggregating the data is best accomplished using OpenSearch Dashboards found here: https://logstash.wikimedia.org

In T371423#10064707, @JMeybohm wrote:

They all failed because the installer tried to bring up some old mdadm arrays and failed doing so, Maybe they where on the new disks, or it is because the disk numbering changed with the new disks added, idk.
What worked is zapping GPT/MBR from a debian installer shell and re(PXE)booting into the installer:

dd if=/dev/zero of=/dev/sda bs=512 count=34
dd if=/dev/zero of=/dev/sda bs=512 count=34 seek=$((`blockdev --getsz /dev/sda` - 34))
dd if=/dev/zero of=/dev/sdb bs=512 count=34
dd if=/dev/zero of=/dev/sdb bs=512 count=34 seek=$((`blockdev --getsz /dev/sdb` - 34))
dd if=/dev/zero of=/dev/sdc bs=512 count=34
dd if=/dev/zero of=/dev/sdc bs=512 count=34 seek=$((`blockdev --getsz /dev/sdc` - 34))
dd if=/dev/zero of=/dev/sdd bs=512 count=34
dd if=/dev/zero of=/dev/sdd bs=512 count=34 seek=$((`blockdev --getsz /dev/sdd` - 34))
dd if=/dev/zero of=/dev/sde bs=512 count=34
dd if=/dev/zero of=/dev/sde bs=512 count=34 seek=$((`blockdev --getsz /dev/sde` - 34))
dd if=/dev/zero of=/dev/sdf bs=512 count=34
dd if=/dev/zero of=/dev/sdf bs=512 count=34 seek=$((`blockdev --getsz /dev/sdf` - 34))

In T371422#10064622, @VRiley-WMF wrote:

I have allocated these drives added these SSDs to specified servers. Please test it out and let us know if there are any issues. Thank you!

They all failed because the installer tried to bring up some old mdadm arrays and failed doing so, Maybe they where on the new disks, or it is because the disk numbering changed with the new disks added, idk.
What worked is zapping GPT/MBR from a debian installer shell and re(PXE)booting into the installer:

dd if=/dev/zero of=/dev/sda bs=512 count=34
dd if=/dev/zero of=/dev/sda bs=512 count=34 seek=$((`blockdev --getsz /dev/sda` - 34))
dd if=/dev/zero of=/dev/sdb bs=512 count=34
dd if=/dev/zero of=/dev/sdb bs=512 count=34 seek=$((`blockdev --getsz /dev/sdb` - 34))
dd if=/dev/zero of=/dev/sdc bs=512 count=34
dd if=/dev/zero of=/dev/sdc bs=512 count=34 seek=$((`blockdev --getsz /dev/sdc` - 34))
dd if=/dev/zero of=/dev/sdd bs=512 count=34
dd if=/dev/zero of=/dev/sdd bs=512 count=34 seek=$((`blockdev --getsz /dev/sdd` - 34))
dd if=/dev/zero of=/dev/sde bs=512 count=34
dd if=/dev/zero of=/dev/sde bs=512 count=34 seek=$((`blockdev --getsz /dev/sde` - 34))
dd if=/dev/zero of=/dev/sdf bs=512 count=34
dd if=/dev/zero of=/dev/sdf bs=512 count=34 seek=$((`blockdev --getsz /dev/sdf` - 34))

The errors you get suggest that the command was not quoted properly, please double check. The examples linked in the wiki assume you have SSH tunneled port 9200 to localhost, so they use localhost:9200 everywhere. Replace that with https://logs-api.svc.eqiad.wmnet, check your quoting and you should be good.

Oh, wait. 0.9GB - I totally misread. That is obviously not okay :D Tried rescanning the drives without luck. I would assume it's broken.

In T371423#10061804, @Jhancock.wm wrote:

I located the missing disk and reseated it. it's showing as having a size of 0.94 GB. Not sure if it's bad or needs to be reformatted. lmk and I'll see if I can find a replacement.

@Jhancock.wm could you please check kafka-main2010 again? After trying to re-image I now only see 5 disks in iDRAC.

In T371667#10058085, @brouberol wrote:

Yes, that's perfect.

Correct. Anything that is at least as big as the ~900G of the four currently installed SSDs will be fine. Thanks!

Just to be extra sure, you want the following to be removed:

• stable/cloudnative-pg-operator-0.2.0.tgz
• stable/cloudnative-pg-operator-crds-0.1.0.tgz
• stable/cluster-0.1.0.tgz
• stable/cluster-0.1.1.tgz
• stable/cluster-0.1.2.tgz
• stable/cluster-0.1.3.tgz
• stable/cluster-0.1.4.tgz
• stable/cluster-0.1.5.tgz
• stable/cluster-0.1.6.tgz

One thing I've noticed is that kafka-main2010 seems to have a different disk then all the others (all others are 1.7T models):

Generally speaking throttling is not an issue (as long as availability/latency targets are still met) but more a measure against processes going rough (so it's very common and kind of desired to have containers throttled). AIUI the referenced task the throttling was not the issue in this case as well. So as long as there is no actual problem, I would refrain from trying to "fix" all throttling or alert on it - as it's also not clearly actionable.

With how the prometheus service discovery currently works (e.g scraping every container port by default) we do have a large number of "okay to be down" targets, so an alert like this will produce quite some alerts. It's also pretty common for pods to go away, which might produce a flurry of alerts as well.

For T337928: cfssl-issuer: Generate Kubernetes Events I tried to vendor our v1.6.1 which fails badly as etcd seemed to have renamed a bunch of their libraries at some time. To circumvent this I've imported upstream v1.6.5 via gbp into https://gitlab.wikimedia.org/repos/sre/cfssl/-/tree/upstream/1.6.5?ref_type=tags, rebased our patches onto that in https://gitlab.wikimedia.org/repos/sre/cfssl/-/tree/merge_1.6.5 and created a branch that includes the patches (https://gitlab.wikimedia.org/repos/sre/cfssl/-/tree/wmf-1.6.5?ref_type=heads) for cfssl-issuer to include

The nodes are not in service, so no need to schedule a maint-window from our side. Feel free to choose a time that suits you best.

The nodes are not in service, so no need to schedule a maint-window from our side. Feel free to choose a time that suits you best.

Regarding the throttling: Maybe it would help to set GOMAXPROCS to something sensible/related to the CPU limit (see https://wikitech.wikimedia.org/wiki/Kubernetes/Resource_requests_and_limits, https://github.com/uber-go/automaxprocs/tree/master)

In T368366#9998706, @elukey wrote:

I used this horrible bash script to get a breakdown of image versions deployed on a given cluster:

for ns in `kubectl get ns | cut -d " " -f 1 | grep -v NAME`; do echo -e "\nnamespace: $ns\n"; kubectl get pods -n $ns -o jsonpath="{.items[*].spec['initContainers', 'containers'][*].image}" |tr -s '[[:space:]]' '\n' |sort |uniq -c; done

This is still painful but keeping a note in here anyway :)

Implementation is (I think) completed. There is a README which hopefully explains what my intentions where and how the policies can be used to create something PSS like but with exceptions.

I have updated https://grafana-rw.wikimedia.org/d/b1jttnFMz/envoy-telemetry-k8s which is now loading the namespace variable via kube_namespace_created and I also flipped the variable order from (app, namespace) to (namespace, app) which greatly improves loading time of all variables after namespace (and therefore the dashboard as a whole)

I think this would be nice to make and keep things a bit more tidy 👍

In T368523#9990103, @Lucas_Werkmeister_WMDE wrote:

I was hoping to only change how the Termbox connects to Wikidata, not the other direction (Termbox being called by Wikidata) – your comment sounds like it would affect both? (Maybe it’s not possible to separate them?) Would that mean changing the wmgWikibaseSSRTermboxServerUrl as well (and maybe a new hieradata/common/service.yaml and/or hieradata/common/profile/services_proxy/envoy.yaml entry in Puppet)?

The policy parser tool has moved to https://gitlab.wikimedia.org/repos/sre/kyverno-policy-parser

Something like that, yes. Although you will need to use a mesh.public_port different from the one used for the "not test" release, as the nodePort would clash otherwise. See https://wikitech.wikimedia.org/wiki/Kubernetes/Service_ports

I might be missing something obvious here (sorry for not mentioning earlier), but why does termbox test not use the service mesh to connect to mw-api-int-ro.discovery.wmnet like the other termbox releases do? That would leave the TLS part completely to envoy, removing the requirement to provide NODE_EXTRA_CA_CERTS etc.

In T368523#9969013, @Lucas_Werkmeister_WMDE wrote:

Aha, “unable to get local issuer certificate”:

lucaswerkmeister-wmde@deploy1002 /srv/deployment-charts/helmfile.d/services/termbox $ kube_env termbox staging
lucaswerkmeister-wmde@deploy1002 /srv/deployment-charts/helmfile.d/services/termbox $ kubectl get pods
NAME                              READY   STATUS    RESTARTS   AGE
termbox-staging-c74b86488-hwm58   3/3     Running   0          11m
termbox-test-d78f78c6d-k5xnb      2/2     Running   0          11m
lucaswerkmeister-wmde@deploy1002 /srv/deployment-charts/helmfile.d/services/termbox $ kubectl logs termbox-test-d78f78c6d-k5xnb termbox-test --tail=1
{"name":"wikibase-termbox","hostname":"termbox-test-d78f78c6d-k5xnb","pid":21,"level":"ERROR","message":"unable to get local issuer certificate","request":{"headers":{"Accept":"application/json, text/plain, */*","User-Agent":"wikibase-termbox/0.1.0 (The Wikidata team) axios/^0.21.1","Host":"test.wikidata.org:4446"},"url":"api.php","params":{"action":"query","meta":"allmessages","ammessages":"wikibase-edit|wikibase-publish|wikibase-cancel|wikibase-aliases-separator|wikibase-entitytermsforlanguagelistview-more|wikibase-entitytermsforlanguagelistview-less|wikibase-entitytermsview-entitytermsforlanguagelistview-toggler|wikibase-label-empty|wikibase-description-empty|wikibase-label-edit-placeholder|wikibase-description-edit-placeholder|wikibase-alias-edit-placeholder|wikibase-anonymouseditwarning-heading|wikibase-anonymouseditwarning-message|wikibase-anonymouseditnotificationtempuser-message|wikibase-anonymouseditwarning-dismiss-button|wikibase-anonymouseditwarning-dismiss-persist|pt-login|pt-createaccount|wikibase-shortcopyrightwarning-accept-persist|wikibase-shortcopyrightwarning-heading|wikibase-entity-save-error-heading|wikibase-entity-save-error-message","amlang":"en","format":"json"}},"url":"/termbox?entity=Q235006&revision=676047&language=en&editLink=%2Fwiki%2FSpecial%3ASetLabelDescriptionAliases%2FQ235006&preferredLanguages=en%7Cmul","reqId":"34fc55cc-dbfc-4c12-bc43-b64ebe2e009f","levelPath":"error/service","msg":"unable to get local issuer certificate","time":"2024-07-10T12:28:17.803Z","v":0}

I have no idea if this could be related to the new envoy version (T368366) or not, though.

In T369011#9965709, @cmooney wrote:

In T369011#9948452, @JMeybohm wrote:

I've deleted the node from the k8s API as a required istio update would not finish successfully because it was waiting for the deamonset to be scheduled on the broken node. The node should auto-join the cluster (cordoned) when it comes back online.

Hi. I notice BGP for this host is still down on the switch side? If it's likely to continue let me know and I will set the Netbox flag for BGP to off and remove the config from the switch with Homer. Thanks.

In T341560#9967304, @RLazarus wrote:

We'll eventually send it to logstash too, but that hasn't happened yet.

Another datapoint to add: During the problematic time frame alerting did catch some of the orchestrator restarts (it just alerts on restarts that are caused by OOM kills): https://logstash.wikimedia.org/goto/aac4d9745b946d83bc486d3300221658
The memory increase is mostly not visible in grafana, so I would suspect it to have happened fast (we only scrape data every 30s). Maybe that helps with tracking down the root cause.

Again, limited understanding on what I'm looking at here, but https://logstash.wikimedia.org/goto/49bb255fe9a7c07a37389ad8e377b362 seems to assemble a portion of the problematic requests. Unfortunately not all because a 503 does not seem to always result in the zObject containing "Service Unavailable".

I've updated the variables as of T369607: High cardinality metrics break queries/dashboards (envoy, istio, ...) - dashboard is now usable again.

In T368892#9963273, @cmassaro wrote:

If I try running the function call above in the ApiSandbox, I get a "Service Unavailable."

I'm unable to reproduce that. But judging from the data I get back it seems to be a cached response (orchestrationCpuUsage\",\"K2\":\"688.86 ms is always the same for me)

If the issue is reproducible without calling any evaluator, could you check if curl'ing the orchestrator (e.g. bypassing the wiki) does also also result in an error? With bypassing the wikis you can also rule out timeout issues as you're (mostly, as there's also between the TLS terminator end the orchestrator) under control of those.
I would also suggest trying to figure out when the issues started and try to roll back to the release version that was active before to rule out code changes that happened in between.

@cmassaro thanks for the ping. Unfortunately I have trouble finding the right data to analyze this further. The orchestrator does call the evaluator(s) directly (not via the service mesh) and does not produce metrics regarding those requests like latency, error rates etc. (at least I'm unable to find those). Looking at the logs of the components I can't see failed requests as well.

In T365994#9949655, @JMeybohm wrote:

!log jayme@cumin1002 conftool action : set/pooled=no; selector: name=(wikikube-worker1007.eqiad.wmnet|wikikube-worker1021.eqiad.wmnet|kubernetes1060.eqiad.wmnet)

!log jayme@cumin1002 conftool action : set/pooled=no; selector: name=(wikikube-worker1007.eqiad.wmnet|wikikube-worker1021.eqiad.wmnet|kubernetes1060.eqiad.wmnet)

I've deleted the node from the k8s API as a required istio update would not finish successfully because it was waiting for the deamonset to be scheduled on the broken node. The node should auto-join the cluster (cordoned) when it comes back online.