| • Marostegui | |
| Jul 7 2024, 6:04 AM |
| F56466620: Screenshot 2024-07-16 at 22.13.12.png | |
| Jul 16 2024, 7:16 PM |
| F56441968: Screenshot 2024-07-15 at 19.52.21.png | |
| Jul 15 2024, 4:54 PM |
We are getting some emails sent to root with the following errors
Can't stat /opt/otrs/var/tmp/CacheFileStorable/User: No such file or directory at /opt/znuny-6.5.8/Kernel/System/Cache/FileStorable.pm line 241.
I've not found anything related in Phabricator.
| Subject | Repo | Branch | Lines +/- | |
|---|---|---|---|---|
| vrts: change root mail alias | operations/puppet | production | +1 -1 |
@Marostegui Thank you for opening this. This might be a bug but will confirm with Znuny.
It was initially an error that occurred more frequently as we had two jobs deleting the cache so we disabled ours based on information we got from them:
There's another cron task that deletes the expired cache (Daemon::SchedulerCronTaskManager::Task###CoreCacheCleanup), so you don't need to delete the cache regularly.
https://github.com/znuny/Znuny/blob/dev/Kernel/System/Cache/FileStorable.pm#L241
The mysterious thing is why or what's trying to stat for deleted cache files thus my suspicion is it's a bug; the job runs when the cache is already empty and maybe it has no way to check if it's already empty. It's not listed as an issue in their repository so I'll just write to them.
There are 2 parts to this. The actual issue on the VRTS side and then the part that this arrives at root@wikimedia.org now.
The assumption, confirmed by Jesse, is that the latter part is related to recent work regarding the exim -> postfix migration.
Thanks @Dzahn I have filed a ticket with Znuny to get a better understanding of what might be causing this. I'll update once I get a response.
Just confirmed that this is indeed a bug but with no negative effect (other than the noise of course) and can safely be ignored. There are just two isolated processes that are not aware of what the other is up to (in a nutshell).
Can we suppress it somehow? While that's great to know, I think the goal here was a bit more than to determine if it can be ignored.
I would say it's technically resolved if that mail does not arrive at root@wikimedia.org anymore.
And it's only secondary if that error is generated at all. As long as it's not sent to all roots.
Probably not total silence as you could miss things, but silencing those "known things".
@Arnoldokoth For this ticket I think it's mostly the "why does root@ suddenly get those mails that they didn't get all this time". That is probably a question for Jesse.
The part that VRTS has this issue is a separate question to me since that doesnt seem to be new.
I think the ideal outcome would be that root@wikimedia.org has silence but you or our subteam still gets them.
Change #1054369 had a related patch set uploaded (by AOkoth; author: AOkoth):
[operations/puppet@production] vrts: change root mail alias
Change #1054369 merged by AOkoth:
[operations/puppet@production] vrts: change root mail alias
We got this to root:
VRTS Notifications <vrts@ticket.wikimedia.org> 5:31 AM (1 hour ago) to root Can't opendir(/opt/otrs/var/tmp/CacheFileStorable/User/6/d): No such file or directory at /opt/znuny-6.5.8/Kernel/System/Cache/FileStorable.pm line 241.
Though I noticed that none of the services on the host were restarted i.e. vrts-daemon and exim4 after the alias change. I've manually restarted them so let's see where the next one goes.
I think I found the setting that affects this. It's configured on the VRTS dashboard and was initially set to root@localhost. Updated it to our team's mailing list. Let's see where the next one goes.
I think I found the setting that affects this. It's configured on the VRTS dashboard and was initially set to root@localhost. Updated it to our team's mailing list. Let's see where the next one goes.
I think this option solved the mail issue. As far as I can tell the mails no longer go to root@. The latest mails from VRTS Notifications (with the same error) were sent to our team mail address sre-service-ops-collab@ instead of root@.
I think I found the setting that affects this. It's configured on the VRTS dashboard and was initially set to root@localhost. Updated it to our team's mailing list. Let's see where the next one goes.
I think this option solved the mail issue. As far as I can tell the mails no longer go to root@. The latest mails from VRTS Notifications (with the same error) were sent to our team mail address sre-service-ops-collab@ instead of root@.
root@ got a bunch of these emails e.g. just now
I think I found the setting that affects this. It's configured on the VRTS dashboard and was initially set to root@localhost. Updated it to our team's mailing list. Let's see where the next one goes.
I think this option solved the mail issue. As far as I can tell the mails no longer go to root@. The latest mails from VRTS Notifications (with the same error) were sent to our team mail address sre-service-ops-collab@ instead of root@.
root@ got a bunch of these emails e.g. just now
my apologies I misspoke, last email was about 13h ago
np! Yes yesterday 18:31 UTC was the last mail to root@. The more recent ones today at 00:01 UTC just arrived at our team address.
@Jelto @fgiunchedi Yeah, the last ones I got were also addressed to our team. :)
Going to resolve this after the upgrade (T368646) just to make sure the setting is persisted after an update.