Change #1069994 had a related patch set uploaded (by Arturo Borrero Gonzalez; author: Arturo Borrero Gonzalez):

[cloud/wmcs-cookbooks@main] wmcs.vps.create_project: replace logic with message about deprecation

https://gerrit.wikimedia.org/r/1069994

had this again today.

aborrero merged https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/28

In T348643#10104246, @wiki_willy wrote:

Hi @dcaro - just following up on this to see if you were ok with shipping these WMCS drives with data on them, back to Dell for identifying the root cause? From Dell's last email a couple weeks ago, they stated that they have a NDA with Hynix, along with the NDA with Wikimedia, which should cover any security concerns. To ensure we don't lose momentum, during my call with Dell today, I asked them to provide the number of drives they need and also a shipping label on where to send them to. Let us know though if you feel comfortable with sending the disks. Thanks, Willy

Hi @dcaro - just following up on this to see if you were ok with shipping these WMCS drives with data on them, back to Dell for identifying the root cause? From Dell's last email a couple weeks ago, they stated that they have a NDA with Hynix, along with the NDA with Wikimedia, which should cover any security concerns. To ensure we don't lose momentum, during my call with Dell today, I asked them to provide the number of drives they need and also a shipping label on where to send them to. Let us know though if you feel comfortable with sending the disks. Thanks, Willy

Change #1068839 merged by jenkins-bot:

[cloud/wmcs-cookbooks@main] ceph.osd.bootstrap_and_add: default to only adding 2 osds at once

https://gerrit.wikimedia.org/r/1068839

Change #1068839 had a related patch set uploaded (by Andrew Bogott; author: Andrew Bogott):

[cloud/wmcs-cookbooks@main] ceph.osd.bootstrap_and_add: default to only adding 2 osds at once

https://gerrit.wikimedia.org/r/1068839

Looks like some of the images for the 1.26 deploy of magnum k8s vanished from upstream, so tofu sees the deploy as good, but kube-system has a few image pull failures. 1.27 seems to work now. There is a little note in T373360 and a little discussion in T373093 on the issue. At any rate you should probably update. Quarry is updated to 1.27, so it's code is good to replicate from. PAWS has a pr open for updating but it hasn't gone out yet https://github.com/toolforge/paws/pull/451

In T373227#10089302, @Andrew wrote:

I didn't do a rabbitmq overhaul. This is silly, but I restarted all the magnum-conductor agents and the cert signing became reliable. And, as far as I can tell, that was it -- tofu can now create a k8s cluster.

So the conductor was wedged in a way that was invisible to monitoring, I guess.

In the meantime, bd808 can you retest?

I've applied the suggested (but messy) patch suggested as part of the upstream bug. It seems to be helping.

Change #1065988 merged by Andrew Bogott:

[operations/puppet@production] Magnum: hack in fixes to sqlalchemy use

https://gerrit.wikimedia.org/r/1065988

Change #1065934 abandoned by Andrew Bogott:

[operations/puppet@production] Openstack magnum: override default db settings to increase connection count

Reason:

This delays but does not resolve the problem.

https://gerrit.wikimedia.org/r/1065934

Change #1065988 had a related patch set uploaded (by Andrew Bogott; author: Andrew Bogott):

[operations/puppet@production] Magnum: hack in fixes to sqlalchemy use

https://gerrit.wikimedia.org/r/1065988

The attached bug delays, but does not resolve the problem. It's a connection leak of some sort, affecting many users: https://bugs.launchpad.net/magnum/+bug/2067345

Change #1065934 had a related patch set uploaded (by Andrew Bogott; author: Andrew Bogott):

[operations/puppet@production] Openstack magnum: override default db settings to increase connection count

https://gerrit.wikimedia.org/r/1065934

I didn't do a rabbitmq overhaul. This is silly, but I restarted all the magnum-conductor agents and the cert signing became reliable. And, as far as I can tell, that was it -- tofu can now create a k8s cluster.

On the server side, a failed curl looks like this:

Testing by hand, that curl works some of the time and fails some of the time. So maybe we're just seeing a partial service outage + a very touchy agent

Deep in the guts of the heat agent it is trying to get json containing a cert, and failing to parse what it gets:

https://docs.openstack.org/magnum/2024.1/admin/troubleshooting-guide.html#heat-stacks

$ openstack stack list | grep beta
| e93cb3ea-9a9e-47e8-a435-a475706bf948 | beta-v126-zzqihxfjnjjb                | CREATE_FAILED | 2024-08-23T18:03:44Z | None         |
$ openstack stack resource list beta-v126-zzqihxfjnjjb | grep FAILED
| kube_masters                            | dc43e1f5-2f7b-4e69-bc2b-537561edba46 | OS::Heat::ResourceGroup                                                            | CREATE_FAILED   | 2024-08-23T18:04:19Z |
$ openstack stack resource show beta-v126-zzqihxfjnjjb kube_masters
+------------------------+-----------------------------------------------------+
| Field                  | Value                                               |
+------------------------+-----------------------------------------------------+
| attributes             | {'removed_rsrc_list': [], 'refs_map': None, 'refs': |
|                        | None, 'attributes': None}                           |
| creation_time          | 2024-08-23T18:04:19Z                                |
| description            |                                                     |
| links                  | [{'href': 'https://openstack.eqiad1.wikimediacloud. |
|                        | org:28004/v1/deployment-prep/stacks/beta-v126-      |
|                        | zzqihxfjnjjb/e93cb3ea-9a9e-47e8-a435-               |
|                        | a475706bf948/resources/kube_masters', 'rel':        |
|                        | 'self'}, {'href': 'https://openstack.eqiad1.wikimed |
|                        | iacloud.org:28004/v1/deployment-prep/stacks/beta-   |
|                        | v126-zzqihxfjnjjb/e93cb3ea-9a9e-47e8-a435-          |
|                        | a475706bf948', 'rel': 'stack'}, {'href': 'https://o |
|                        | penstack.eqiad1.wikimediacloud.org:28004/v1/deploym |
|                        | ent-prep/stacks/beta-v126-zzqihxfjnjjb-kube_masters |
|                        | -3dcad4ye6nm6/dc43e1f5-2f7b-4e69-bc2b-              |
|                        | 537561edba46', 'rel': 'nested'}]                    |
| logical_resource_id    | kube_masters                                        |
| physical_resource_id   | dc43e1f5-2f7b-4e69-bc2b-537561edba46                |
| required_by            | ['kube_cluster_config', 'api_address_lb_switch',    |
|                        | 'kube_cluster_deploy', 'etcd_address_lb_switch']    |
| resource_name          | kube_masters                                        |
| resource_status        | CREATE_FAILED                                       |
| resource_status_reason | Error: resources.kube_masters.resources[0].resource |
|                        | s.master_config_deployment: Deployment to server    |
|                        | failed: deploy_status_code: Deployment exited with  |
|                        | non-zero status code: 1                             |
| resource_type          | OS::Heat::ResourceGroup                             |
| updated_time           | 2024-08-23T18:04:19Z                                |
+------------------------+-----------------------------------------------------+

@Andrew, any idea about where I should start looking for hints about what might be going wrong here?

Change #1065247 merged by Andrew Bogott:

[operations/puppet@production] Add openstack magnum and heat logs to logstash

https://gerrit.wikimedia.org/r/1065247

Change #1065247 had a related patch set uploaded (by Andrew Bogott; author: Andrew Bogott):

[operations/puppet@production] Add openstack magnum and heat logs to logstash

https://gerrit.wikimedia.org/r/1065247

Mentioned in SAL (#wikimedia-cloud-feed) [2024-08-23T14:48:42Z] <andrew@cloudcumin1001> END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1062.eqiad.wmnet' (T369044)

Mentioned in SAL (#wikimedia-cloud-feed) [2024-08-23T14:42:13Z] <andrew@cloudcumin1001> START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1062.eqiad.wmnet' (T369044)

Change #1056606 merged by Andrew Bogott:

[operations/puppet@production] cinderutils: add --allow-unattended-format when preparing volumes

https://gerrit.wikimedia.org/r/1056606

Mentioned in SAL (#wikimedia-cloud-feed) [2024-08-22T21:16:19Z] <andrew@cloudcumin1001> END (PASS) - Cookbook wmcs.openstack.cloudweb.set_maintenance (exit_code=0) (T369044)

Mentioned in SAL (#wikimedia-cloud-feed) [2024-08-22T21:14:17Z] <andrew@cloudcumin1001> START - Cookbook wmcs.openstack.cloudweb.set_maintenance (T369044)

Mentioned in SAL (#wikimedia-cloud-feed) [2024-08-22T20:46:08Z] <andrew@cloudcumin1001> END (ERROR) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=97) on host 'cloudvirt1047.eqiad.wmnet' (T369044)

Mentioned in SAL (#wikimedia-cloud-feed) [2024-08-22T20:43:52Z] <andrew@cloudcumin1001> START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1047.eqiad.wmnet' (T369044)

Mentioned in SAL (#wikimedia-cloud-feed) [2024-08-22T20:43:46Z] <andrew@cloudcumin1001> END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1038.eqiad.wmnet' (T369044)

Mentioned in SAL (#wikimedia-cloud-feed) [2024-08-22T20:36:30Z] <andrew@cloudcumin1001> START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1038.eqiad.wmnet' (T369044)