Difference between revisions of "Sheriffing/Sheriff Duty"

Revision as of 00:46, 20 June 2008

What's a sheriff, anyways?

Every day a person from the mozilla community is selected to watch over the build tree. Make sure red gets traction, call people on the phone, etc. Send mail to the sheriffs alias to reach the set of eligible sheriffs.

How should I get ahold of people?

Try irc.mozilla.org, #developers and #build.

How do I change the mozilla tinderbox message of the day?

• Firefox tree
• SeaMonkey tree

Click on Administrate Tinderbox Trees at the bottom of the tinderbox page. Edit the html and set the message of day at the top of the page using the mozilla tinderbox password.

What kind of comments might the sheriff add to Tinderbox?

The sheriff has broad discretion to add tinderbox comments related to the Mozilla builds, tree opening, blockers etc.

Developers wishing changes to the Mozilla tinderbox messages should communicate with the sheriff so that the changes can be coordinated.

How do I remove a cvs lock?

Ask someone with access to the repository (#bmo, #builds) to remove the lock:

cvs is bailing out with "signal 11 errors", what do I do?

If cvs is crashing with an error like this:

Terminated with fatal signal 11
make[1]: *** [real_checkout] Error 1
make[1]: Leaving directory `/builds/mozilla/Linux_2.4.17-rc1_Clobber'
make: *** [checkout] Error 2

This may be due to a case conflict, e.g. Makefile.in and makefile.in are both checked in, and makefile.in is hiding in the Attic directory. Figure out which file is the right one, and (safely) delete the presumed bogus file from the repository.

When should I close the mozilla tree?

The tree should be closed due to:

• tagging and branching associated with releases
• system outages that prevent the tinderbox machines from running the necessary tests (compilation, function, performance, memory) on changes that are checked in
• bustage (compilation errors, test failures, or significant performance/memory test regressions) that prevents tinderbox machines from determining if new changes cause further regressions
• bugs that are bad enough to prevent the builds from being used as dogfood

Note that a closed tree is different from a restricted tree, where checkins require additional approval or additional criteria due to an impending release.

If you close the tree, please record the time and the reasons why on the Tree Closures page.

When should I reopen the tree after closing it?

When the tree has been closed due to build bustage, performance regressions, or test regressions, it should be reopened once:

• the sheriff has confidence that those regressions are diagnosed and being fixed (preferably fixed already), and
• tinderbox is clear enough (or expected to be in the cycle that current checkins would go into) that any new regressions that should be caught will be.

This does not mean that all boxes need to cycle green again before reopening. However, if there were a large number of checkins since things started going bad, it is often good to wait for test boxes to cycle green again to make sure there aren't regressions caused by what has already landed.

If there has been a lot of tree closure recently, continuing to hold the tree closed increases the chances that there will be a mess when it reopens and everybody lands at once. Tree closures should therefore be kept to the minimum necessary, especially when there have been many recent closures or there is an upcoming deadline.

Please make sure to update the Tree Closures page with the time of the opening, as well as any notes that would be useful for a postmortem about that particular closure.

How do I open/close the mozilla tree?

• If the tree is being closed for anything other than red or short-lived orange, file a bug about the problem for which it is being closed:
  • If it's a performance or unit test regression, the bug should say which tinderboxes, when, and has a bonsai URL to the list of checkins in the window that may have caused the regression.
  • If it's a functional regression, the bug should have steps to reproduce, the dates of the nightly builds immediately surrounding the regression (or narrower window if known), and a bonsai URL to the list of checkins that could have caused the regression.
• Update tinderbox status message:
  • When closing, say that the tree is closed, and include the reason for the closure (if for something other than red/orange, this should include a hyperlink to the bug). Do this using the "Administer" link at the bottom of the page, and the sheriff password.
  • When opening the tree, remove the message added when closing.
• Open/Close the tree in bonsai:
  • Go to Bonsai Administration ['SeaMonkey' Tree] The top section is used to close the tree. Note that closing SeaMonkey automatically closes Firefox.
  • If opening the tree, uncheck "Clear the list of checkins". If you don't do this, you will clear the hook for the day.
  • Type in the mozilla sheriff password.
  • Click on Close the tree button. The button will rename to Open the tree if the tree is currently closed.
• Update the tree closure log

What do I do about regressions?

See the official regression policy document: Regression Policy

Who is the sheriff today?

See the Google Calendar that tracks the duty roster:

HTML ICS

Handy Sheriff tinderbox sidebar

A tinderbox summary sidebar can be handy for keeping an eye on the tree while you're looking at other pages in the browser.

Create a bookmark from the data: uri below and then edit the bookmark to open as a sidebar. The link currently updates every three minutes which is more useful for sheriffing than the tinderbox default 15, you could shorten it to two or one if you really wanted to be on top of any bustage.

data:text/html,<meta%20http-equiv="refresh"%20content="180"><frameset%20rows=*,20%,20%><frame%20src=%22http://tinderbox.mozilla.org/showbuilds.cgi?tree=Mozilla2&panel=1%22%20frameborder=0><frame%20src=%22http://tinderbox.mozilla.org/showbuilds.cgi?tree=Firefox&panel=1%22%20frameborder=0><frame%20src=%22http://tinderbox.mozilla.org/showbuilds.cgi?tree=Mozilla1.8&panel=1%22%20frameborder=0></frameset>

Is there an easy way to monitor all performance graphs?

Yes! Thanks to Johnathan Nightingale, we now have a performance dashboard.

How do I kick a box that won't build?

If the problem isn't caused by a checkin, or it remains after the checkin was reverted, file a blocker IT ticket in the mozilla.org product, Server Operations: Tinderbox Maintenance component:

File a Tinderbox Maintenance Blocker Ticket

How do I back out changes from Mercurial?

See Mercurial FAQ: Backing out changes.