Hey @taavi 👋 One thing to note is that a decision was made to deprecate the processes that are running completely. WMDE analytics has no plan of maintaining quratorqcerevolver or quratorqcfrevolver - also known as Current Events (ce) and Curious Facts (cf).

wikidata-analytics-1.wmdeanalytics.eqiad1.wikimedia.cloud still has the broken Docker subnet blocking DNS access. I would assume this means the instance is mostly unreachable using the normal developer account login system as that relies on DNS working, and the containers running on that host are also restarting and logging errors like Could not resolve host: analytics.wikimedia.org.

As with T285291, it seems like this might be resolved, unless there are further subtasks or testing to perform.

Is this resolved now, since all subtasks are resolved, or is it lacking defined subtasks for some of the proposed features? The owning account is disabled.

No problem, I'm here to help :}, my time is limited though, so might take some extra time for me to help on things "out of the usual"

Thank you for the further information here, @dcaro! Sorry that we're a bit in the dark on this (inherited infrastructure where the lack of documentation keeps getting more and more apparent). I've noted this to the appropriate parties on our end and we'll get to it as quickly as possible!

In T354268#9432817, @AndrewTavis_WMDE wrote:

Thanks, @Michael 🙏 For further understanding, what's the frequency of the alerts?

You can probably change the default address pools in docker:

# /etc/docker/daemon.json

In T354268#9432766, @AndrewTavis_WMDE wrote:

Thanks from me as well, @dcaro! Following up on my point:

Yep, those are yes, the networks they are configured to use (ex. 172.20.0.0/16) overlaps with the one the cloudvps uses internally, specifically the one where the DNS is hosted (172.20.255.1)

For this we'll need to go into the deployment and change the targets within the code itself?

Quoting myself:

Thanks, @Michael 🙏 For further understanding, what's the frequency of the alerts?

In T354268#9432766, @AndrewTavis_WMDE wrote:

Can I ask what the severity is of this? As of now these processes are generating data that predominantly is populating outputs in the published datasets. These processes are now stalled until such a time as the deployment is updated?

Thanks from me as well, @dcaro! Following up on my point:

In T354268#9432687, @dcaro wrote:

In T354268#9432608, @Michael wrote:

Also, the alert doesn't say which three deleted VMs have left-over config. Maybe that is a step that we missed in some checklist for how to delete VMs?

Just updated the docs there, now it's pointing to an existing graph:
https://grafana-rw.wmcloud.org/d/SQM7MJZSz/cloud-vps-puppet-agents?orgId=1

In T354268#9432608, @Michael wrote:

In T354268#9432605, @dcaro wrote:

The last alert, is because your puppetmaster wdqspuppet was not cleaned up when the VMs where removed, that has to be manually done:

https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/PuppetStaleCertificates

Mh, is that something I/we should have the rights to do on our own?

Also, the alert doesn't say which three deleted VMs have left-over config. Maybe that is a step that we missed in some checklist for how to delete VMs?

In T354268#9432641, @AndrewTavis_WMDE wrote:

Hey @dcaro 👋 Really appreciate you checking all of this for us!

So for that I guess the solution is to change your docker configuration to use a different range.

Could I check my/our understanding of this with you? In the above output there are some IPs that have linkdown. Are those the problematic ones that we should avoid in the Docker setups?

Hey @dcaro 👋 Really appreciate you checking all of this for us!

In T354268#9432605, @dcaro wrote:

The last alert, is because your puppetmaster wdqspuppet was not cleaned up when the VMs where removed, that has to be manually done:

https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/PuppetStaleCertificates

In T354268#9432600, @dcaro wrote:

For wikidata-analytics-1, it seems that there's a docker installation there, that is reusing the same ranges that the cloud vps network uses, and the traffic to the DNS server is being sent through the docker interface:

root@wikidata-analytics-1:/etc# grep nameserver /etc/resolv.conf 
nameserver 172.20.255.1

root@wikidata-analytics-1:/etc# ip route get 172.20.255.1
172.20.255.1 dev br-f6fcafc9a433 src 172.20.0.1 uid 0 
    cache 

root@wikidata-analytics-1:/etc# ip route show
default via 172.16.0.1 dev ens3 
169.254.169.254 via 172.16.0.1 dev ens3 
172.16.0.0/21 dev ens3 proto kernel scope link src 172.16.5.182 
172.17.0.0/16 dev docker0 proto kernel scope link src 172.17.0.1 linkdown 
172.18.0.0/16 dev br-6ae54cf5ea1c proto kernel scope link src 172.18.0.1 linkdown 
172.19.0.0/16 dev br-4d14d798abd1 proto kernel scope link src 172.19.0.1 linkdown 
172.20.0.0/16 dev br-f6fcafc9a433 proto kernel scope link src 172.20.0.1 linkdown

So for that I guess the solution is to change your docker configuration to use a different range.

The last alert, is because your puppetmaster wdqspuppet was not cleaned up when the VMs where removed, that has to be manually done:

For wikidata-analytics-1, it seems that there's a docker installation there, that is reusing the same ranges that the cloud vps network uses, and the traffic to the DNS server is being sent through the docker interface:

root@wikidata-analytics-1:/etc# grep nameserver /etc/resolv.conf 
nameserver 172.20.255.1

For the wikidata-federated-properties and wb-reconcile, the issue is that there's an old cron for the prometheus user:

"/tmp/crontab.XOf4Il/crontab" 6L, 469C                                                                                                                                                                                                                                                                                                                                      1,1           All
# HEADER: This file was autogenerated at 2021-03-28 15:15:32 +0000 by puppet.
# HEADER: While it can still be managed manually, it is definitely not recommended.
# HEADER: Note particularly that the comments starting with 'Puppet Name' should
# HEADER: not be deleted, as doing so could cause duplicate cron jobs.
# Puppet Name: prometheus_puppet_agent_stats
* * * * * /usr/local/bin/prometheus-puppet-agent-stats --outfile /var/lib/prometheus/node.d/puppet_agent.prom

Change 961223 merged by SBassett:

[mediawiki/extensions/Wikibase@REL1_40] SECURITY: Escape label in FederatedPropertiesError

https://gerrit.wikimedia.org/r/961223

Change 961222 merged by SBassett:

[mediawiki/extensions/Wikibase@REL1_39] SECURITY: Escape label in FederatedPropertiesError

https://gerrit.wikimedia.org/r/961222

Change 961220 merged by SBassett:

[mediawiki/extensions/Wikibase@REL1_35] SECURITY: Escape label in FederatedPropertiesError

https://gerrit.wikimedia.org/r/961220

Change 961223 had a related patch set uploaded (by Mmartorana; author: Lucas Werkmeister (WMDE)):

[mediawiki/extensions/Wikibase@REL1_40] SECURITY: Escape label in FederatedPropertiesError

https://gerrit.wikimedia.org/r/961223

Change 961222 had a related patch set uploaded (by Mmartorana; author: Lucas Werkmeister (WMDE)):

[mediawiki/extensions/Wikibase@REL1_39] SECURITY: Escape label in FederatedPropertiesError

https://gerrit.wikimedia.org/r/961222

Change 961220 had a related patch set uploaded (by Mmartorana; author: Lucas Werkmeister (WMDE)):

[mediawiki/extensions/Wikibase@REL1_35] SECURITY: Escape label in FederatedPropertiesError

https://gerrit.wikimedia.org/r/961220

Change 935883 merged by jenkins-bot:

[mediawiki/extensions/Wikibase@master] SECURITY: Escape label in FederatedPropertiesError

https://gerrit.wikimedia.org/r/935883

Change 935883 had a related patch set uploaded (by SBassett; author: Lucas Werkmeister (WMDE)):

[mediawiki/extensions/Wikibase@master] SECURITY: Escape label in FederatedPropertiesError

https://gerrit.wikimedia.org/r/935883

As just discussed with @Lucas_Werkmeister_WMDE, the issue does not represent an exposed vulnerability and therefore does not need to be handled as a security issue. This task can be opened to the public now.

Suite devs: can you please confirm that you have this issue on your radar?

But that means third-party Wikibases with Federated Properties enabled would be vulnerable until the next Wikibase Suite Team release comes out. I think the timing of the change on Gerrit should be aligned with the next Wikibase Suite release (T340939, I guess? or T332786?), not with the WMF train.

In T339260#8949514, @Lucas_Werkmeister_WMDE wrote:

0001-SECURITY-Escape-label-in-FederatedPropertiesError.patch2 KBDownload

In T339260#8949468, @sbassett wrote:

+1 for the patch as well for this specific issue. I typically like using ENT_QUOTES with htmlentities when feasible, even if it isn't technically, contextually needed (it's not here), as long as it doesn't result in odd double-escapes or the unexpected display of html entity codes.

Sure, done: