[go: nahoru, domu]
Page MenuHomePhabricator
Create Task
Maniphest T248602

Lots of "EventBus: Unable to deliver all events: 504: Gateway Timeout"
Closed, DuplicatePublic
Actions

Description

2020-03-26 16:09:09 [XnzTmwpAAEQAAJUGnSwAAADI] mw1273 cswiki 1.35.0-wmf.24 exception ERROR: [XnzTmwpAAEQAAJUGnSwAAADI] /w/index.php?title=Wikipedistka:Zorka_Sojka/D%C3%ADlna/Jan_Kotek&action=delete   JobQueueError from line 106 of /srv/mediawiki/php-1.35.0-wmf.24/extensions/EventBus/includes/JobQueueEventBus.php: Could not enqueue jobs: Unable to deliver all events: 504: Gateway Timeout {"exception_id":"XnzTmwpAAEQAAJUGnSwAAADI","exception_url":"/w/index.php?title=Wikipedistka:Zorka_Sojka/D%C3%ADlna/Jan_Kotek&action=delete","caught_by":"mwe_handler"}

This happened when trying to delete a page at Czech Wikipedia. It first hanged out for ~10 seconds, and then threw this fatal error. Refreshing fixed it.

Details

SubjectRepoBranchLines +/-
services_proxy: higher timeout for eventgate-main, more retriesoperations/puppetproduction+2 -2
Customize query in gerrit

Related Objects

Event Timeline

Urbanecm created this task.Mar 26 2020, 4:55 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMar 26 2020, 4:55 PM
Joe subscribed.Mar 26 2020, 4:58 PM

So what happened there is that eventgate-main was struggling to respond, and we have a rather aggressive timeout now (at 10 seconds). I'll relax that and add some more retry logic.

Joe triaged this task as Medium priority.Mar 26 2020, 4:59 PM
gerritbot added a comment.Mar 26 2020, 5:18 PM

Change 583688 had a related patch set uploaded (by Giuseppe Lavagetto; owner: Giuseppe Lavagetto):
[operations/puppet@production] services_proxy: higher timeout for eventgate-main, more retries

https://gerrit.wikimedia.org/r/583688

gerritbot added a project: Patch-For-Review.Mar 26 2020, 5:18 PM
gerritbot added a comment.Mar 26 2020, 5:24 PM

Change 583688 merged by Giuseppe Lavagetto:
[operations/puppet@production] services_proxy: higher timeout for eventgate-main, more retries

https://gerrit.wikimedia.org/r/583688

Maintenance_bot removed a project: Patch-For-Review.Mar 26 2020, 6:10 PM
Ottomata subscribed.Mar 26 2020, 7:05 PM
Krinkle mentioned this in T249745: Could not enqueue jobs: "Unable to deliver all events: 503: Service Unavailable".Apr 29 2020, 6:51 PM
Krinkle added projects: WMF-JobQueue, Platform Engineering.Apr 29 2020, 6:53 PM
• Pchelolo subscribed.Apr 29 2020, 9:10 PM

I am no longer seeing 504s, instead we're seeing 503s now. I will merge it into T249745 since most likely the solution will be the same.

• Pchelolo closed this task as a duplicate of T249745: Could not enqueue jobs: "Unable to deliver all events: 503: Service Unavailable".Apr 29 2020, 9:11 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits