User Details
- User Since
- Apr 2 2020, 9:01 AM (230 w, 4 d)
- Availability
- Available
- IRC Nick
- jayme
- LDAP User
- JMeybohm
- MediaWiki User
- JMeybohm (WMF) [ Global Accounts ]
Today
Fri, Aug 30
Deployed with:
helmfile -e staging-codfw -l name=cfssl-issuer-crds -l name=cfssl-issuer -i apply --context 5
Wed, Aug 28
@Ottomata could you please verify the above/patches to unblock the hardware refresh of kafka-main?
Tue, Aug 27
Mon, Aug 26
To unblock the kafka hardware replacements, I've added an alternative readiness probe to the chart which will be used when .Values.main_app.conf.test_events evaluates to false. It just does a GET /_info which is what we do for other servicerunner services as well IMHO. AIUI schema loading happens async anyways, so the service should be ready as soon as the routing is set up, right?
Fri, Aug 23
Thu, Aug 22
Wed, Aug 21
For the record: The CoreDNS Pods are, like all other Pods, reachable by their Pod IP. So going through a debug Pod to resolve the PTRs is not really necessary.
Regarding the IPs of Pods not listed in Endpoints: These could be added to CoreDNS via the https://github.com/coredns/kubepods plugin as well.
Fri, Aug 16
All of these have been reimaged with raid10-6dev now, thanks!
From T371423#10068548 I did:
All of these have been reimaged with raid10-6dev now, thanks!
Thu, Aug 15
Wed, Aug 14
They all failed because the installer tried to bring up some old mdadm arrays and failed doing so, Maybe they where on the new disks, or it is because the disk numbering changed with the new disks added, idk.
What worked is zapping GPT/MBR from a debian installer shell and re(PXE)booting into the installer:
dd if=/dev/zero of=/dev/sda bs=512 count=34 dd if=/dev/zero of=/dev/sda bs=512 count=34 seek=$((`blockdev --getsz /dev/sda` - 34)) dd if=/dev/zero of=/dev/sdb bs=512 count=34 dd if=/dev/zero of=/dev/sdb bs=512 count=34 seek=$((`blockdev --getsz /dev/sdb` - 34)) dd if=/dev/zero of=/dev/sdc bs=512 count=34 dd if=/dev/zero of=/dev/sdc bs=512 count=34 seek=$((`blockdev --getsz /dev/sdc` - 34)) dd if=/dev/zero of=/dev/sdd bs=512 count=34 dd if=/dev/zero of=/dev/sdd bs=512 count=34 seek=$((`blockdev --getsz /dev/sdd` - 34)) dd if=/dev/zero of=/dev/sde bs=512 count=34 dd if=/dev/zero of=/dev/sde bs=512 count=34 seek=$((`blockdev --getsz /dev/sde` - 34)) dd if=/dev/zero of=/dev/sdf bs=512 count=34 dd if=/dev/zero of=/dev/sdf bs=512 count=34 seek=$((`blockdev --getsz /dev/sdf` - 34))
The errors you get suggest that the command was not quoted properly, please double check. The examples linked in the wiki assume you have SSH tunneled port 9200 to localhost, so they use localhost:9200 everywhere. Replace that with https://logs-api.svc.eqiad.wmnet, check your quoting and you should be good.
Tue, Aug 13
Oh, wait. 0.9GB - I totally misread. That is obviously not okay :D Tried rescanning the drives without luck. I would assume it's broken.
@Jhancock.wm could you please check kafka-main2010 again? After trying to re-image I now only see 5 disks in iDRAC.
Correct. Anything that is at least as big as the ~900G of the four currently installed SSDs will be fine. Thanks!
Mon, Aug 12
Just to be extra sure, you want the following to be removed:
- stable/cloudnative-pg-operator-0.2.0.tgz
- stable/cloudnative-pg-operator-crds-0.1.0.tgz
- stable/cluster-0.1.0.tgz
- stable/cluster-0.1.1.tgz
- stable/cluster-0.1.2.tgz
- stable/cluster-0.1.3.tgz
- stable/cluster-0.1.4.tgz
- stable/cluster-0.1.5.tgz
- stable/cluster-0.1.6.tgz
One thing I've noticed is that kafka-main2010 seems to have a different disk then all the others (all others are 1.7T models):
Generally speaking throttling is not an issue (as long as availability/latency targets are still met) but more a measure against processes going rough (so it's very common and kind of desired to have containers throttled). AIUI the referenced task the throttling was not the issue in this case as well. So as long as there is no actual problem, I would refrain from trying to "fix" all throttling or alert on it - as it's also not clearly actionable.
With how the prometheus service discovery currently works (e.g scraping every container port by default) we do have a large number of "okay to be down" targets, so an alert like this will produce quite some alerts. It's also pretty common for pods to go away, which might produce a flurry of alerts as well.
Thu, Aug 8
For T337928: cfssl-issuer: Generate Kubernetes Events I tried to vendor our v1.6.1 which fails badly as etcd seemed to have renamed a bunch of their libraries at some time. To circumvent this I've imported upstream v1.6.5 via gbp into https://gitlab.wikimedia.org/repos/sre/cfssl/-/tree/upstream/1.6.5?ref_type=tags, rebased our patches onto that in https://gitlab.wikimedia.org/repos/sre/cfssl/-/tree/merge_1.6.5 and created a branch that includes the patches (https://gitlab.wikimedia.org/repos/sre/cfssl/-/tree/wmf-1.6.5?ref_type=heads) for cfssl-issuer to include
Wed, Aug 7
The nodes are not in service, so no need to schedule a maint-window from our side. Feel free to choose a time that suits you best.
The nodes are not in service, so no need to schedule a maint-window from our side. Feel free to choose a time that suits you best.
Regarding the throttling: Maybe it would help to set GOMAXPROCS to something sensible/related to the CPU limit (see https://wikitech.wikimedia.org/wiki/Kubernetes/Resource_requests_and_limits, https://github.com/uber-go/automaxprocs/tree/master)
Jul 19 2024
Implementation is (I think) completed. There is a README which hopefully explains what my intentions where and how the policies can be used to create something PSS like but with exceptions.
I have updated https://grafana-rw.wikimedia.org/d/b1jttnFMz/envoy-telemetry-k8s which is now loading the namespace variable via kube_namespace_created and I also flipped the variable order from (app, namespace) to (namespace, app) which greatly improves loading time of all variables after namespace (and therefore the dashboard as a whole)
Jul 18 2024
I think this would be nice to make and keep things a bit more tidy 👍
Jul 16 2024
The policy parser tool has moved to https://gitlab.wikimedia.org/repos/sre/kyverno-policy-parser
Jul 12 2024
Something like that, yes. Although you will need to use a mesh.public_port different from the one used for the "not test" release, as the nodePort would clash otherwise. See https://wikitech.wikimedia.org/wiki/Kubernetes/Service_ports
Jul 11 2024
I might be missing something obvious here (sorry for not mentioning earlier), but why does termbox test not use the service mesh to connect to mw-api-int-ro.discovery.wmnet like the other termbox releases do? That would leave the TLS part completely to envoy, removing the requirement to provide NODE_EXTRA_CA_CERTS etc.
Jul 10 2024
Jul 9 2024
Another datapoint to add: During the problematic time frame alerting did catch some of the orchestrator restarts (it just alerts on restarts that are caused by OOM kills): https://logstash.wikimedia.org/goto/aac4d9745b946d83bc486d3300221658
The memory increase is mostly not visible in grafana, so I would suspect it to have happened fast (we only scrape data every 30s). Maybe that helps with tracking down the root cause.
Again, limited understanding on what I'm looking at here, but https://logstash.wikimedia.org/goto/49bb255fe9a7c07a37389ad8e377b362 seems to assemble a portion of the problematic requests. Unfortunately not all because a 503 does not seem to always result in the zObject containing "Service Unavailable".
I've updated the variables as of T369607: High cardinality metrics break queries/dashboards (envoy, istio, ...) - dashboard is now usable again.
I'm unable to reproduce that. But judging from the data I get back it seems to be a cached response (orchestrationCpuUsage\",\"K2\":\"688.86 ms is always the same for me)
Jul 8 2024
If the issue is reproducible without calling any evaluator, could you check if curl'ing the orchestrator (e.g. bypassing the wiki) does also also result in an error? With bypassing the wikis you can also rule out timeout issues as you're (mostly, as there's also between the TLS terminator end the orchestrator) under control of those.
I would also suggest trying to figure out when the issues started and try to roll back to the release version that was active before to rule out code changes that happened in between.
Jul 4 2024
@cmassaro thanks for the ping. Unfortunately I have trouble finding the right data to analyze this further. The orchestrator does call the evaluator(s) directly (not via the service mesh) and does not produce metrics regarding those requests like latency, error rates etc. (at least I'm unable to find those). Looking at the logs of the components I can't see failed requests as well.
Jul 3 2024
!log jayme@cumin1002 conftool action : set/pooled=no; selector: name=(wikikube-worker1007.eqiad.wmnet|wikikube-worker1021.eqiad.wmnet|kubernetes1060.eqiad.wmnet)
I've deleted the node from the k8s API as a required istio update would not finish successfully because it was waiting for the deamonset to be scheduled on the broken node. The node should auto-join the cluster (cordoned) when it comes back online.