Difference between revisions of "Bugdays/Bug-verification"
(→Q4) |
(→Mentorship: update mentors) |
||
Line 75: | Line 75: | ||
In case you have never verified a bug or just need someone to walk you through our process, please contact one of our QA Mentors. These are their names and IRC nicks: | In case you have never verified a bug or just need someone to walk you through our process, please contact one of our QA Mentors. These are their names and IRC nicks: | ||
* Mihai Boldan: mboldan | * Mihai Boldan: mboldan | ||
− | * | + | * Hossain Al Ikram: ikram |
If you can't find them on IRC, or don't want to contact them this way, you can just cc one of them in the bug you want to work on and ask for help. | If you can't find them on IRC, or don't want to contact them this way, you can just cc one of them in the bug you want to work on and ask for help. | ||
Revision as of 19:57, 7 December 2018
Contents
Bug Verification Day
Bug verification is done to verify that bugs are properly fixed and to check that the fixes don’t yield any new issues. This is a very important part of the QA work and one of the more challenging areas of community involvement.
This event is open to everyone wanting to help make Firefox better. We usually advertise it on QMO and IRC every week, so you don't miss any updates.
Details
- Where: IRC #qa channel
- When: Every Wednesday, all day long, but you can contribute any other time you wish.
Pre-requisites
- Bugzilla
Bugzilla is Mozilla's bug tracking system. This is where bugs are reported and where every change for a bug is stored and tracked.
To mark a bug as fixed or do any other Bugzilla work, you will need to sign in to Bugzilla using a Bugzilla account or Mozilla Persona. If you don't already have an account, you can sign up here.
- Mozillians
Mozillians is a network of our contributors. You might want to join it in order to connect with other contributors regarding community events, contribution opportunities, knowledge sharing and so on.
- Builds
These are the builds you should verify the fixes on:
- Latest Firefox Beta 64: Windows 32bit / 64bit, Linux 32bit / 64bit, Mac OS X
- Latest Firefox Nightly 65: Windows 32bit / 64bit, Linux 32bit / 64bit, Mac OS X
You can pick one of these builds, download it, and launch it with a new, clean profile. Then test the bugs from the appropriate list below!
If you have any questions regarding the above or this event, please address them on the IRC #qa channel.
How to Verify bugs
- To begin, choose a bug from one of these lists
- Firefox 64 Beta
- Firefox 65 Nightly
- Follow the detailed steps below to complete the verification
- First of all, we need to verify if the bug is marked as Resolved Fixed.
- Take note of the platforms / operating systems affected by the bug.
- Read the description and comments in the bug to understand the problem.
- If you don’t understand the bug, comment it asking for whatever additional details you need to understand and verify it.
- Try to reproduce the bug on a build that is known to be broken (usually any Nightly starting from the day the bug was filed to the day the bug was fixed). You can download the builds from here
- If you are able to reproduce the bug on a known broken build, download the latest build of the version that is thought to be fixed (usually indicated by the status-firefoxN flags) and try to reproduce the bug. You can find links to these builds in the Pre-requisites section.
- Take into consideration that the Nightly branch receive updates daily, while the Beta version (which Developer Edition is now based on) only updates twice a week. This means you might have to wait a few days to verify a very recent fix on Beta or Developer Edition.
- If you are still able to reproduce the bug, provide the details of your testing in the bug report.
- If the bug does NOT reproduce, then you can mark the bug as verified. Set the “status-firefoxN” (N = Firefox version) flag to “verified” for the version you tested and add a comment indicating the details of your testing. If you verified the fix against the latest Nightly, set the STATUS field to VERIFIED. Ask someone on IRC for help if you don’t have the rights to change the flags or status.
- When possible, also do same exploratory testing around the fix to ensure nothing got broken by it.
- Example of a comment after verifying a bug
Managed to reproduce the issue on an affected build (Firefox 52.0a1, build ID), using the steps to reproduce from Comment 0, on Windows 10x64.
The issue is no longer reproducible on Firefox 62.0a1 (build ID), Firefox 61.0b3 (build ID). Tests were performed under Windows 10x64, Mac OS X 10.11, Ubuntu 16.04x64.
- Notes
- Please verify the bugs on the OS/Platforms they were filed against.
- Make sure to add [bugday-yyyymmdd] in the QA Whiteboard (or simple Whiteboard, if the QA one is not there) of all the bugs you work on today. That way you will mark the fact that you contributed to this event.
- If you don’t have the rights to edit a whiteboard, you can simply add [bugday-yyyymmdd] in a comment.
- If you need any guidance or more information, please address your questions on the channel chat and we will try to help you. You can also take a look at our FAQ page.
- Scenarios you might encounter
- If the issue can’t be reproduced by the tester, a comment must be added asking for more information or just notifying that the issue could not be reproduced. In this case is not relevant verifying the bug on latest builds.
- If the issue is reproducible only on a specific platform at which the tester don’t have access too, then verifying the issue is also not relevant.
- If better STR are provided in the comments below Comment 0, then those STR must be used in order to verify the bug.
- If the build with the fix is not released, we use tinderbox builds, or wait until the build is available.
- If another issue is found while verifying a bug, we check to see if the issue is not already logged and if not, log a new bug.
Mentorship
In case you have never verified a bug or just need someone to walk you through our process, please contact one of our QA Mentors. These are their names and IRC nicks:
- Mihai Boldan: mboldan
- Hossain Al Ikram: ikram
If you can't find them on IRC, or don't want to contact them this way, you can just cc one of them in the bug you want to work on and ask for help.
Hosting a test day
Anyone is welcome to host a bugday or test day focused in an area of interest.
If you're hosting a test day or bug day, please report on its results!
- Email about the results on the dev-quality mailing list.
- Follow up in QA or community meetings to thank all the testers. :)
Results
Many thanks to everyone that contributed to these events!
2018
Q4
- 12/05 - x bugs
- 11/28 - 4 bugs
- 11/21 - 3 bugs
- 11/14 - 2 bugs
- 11/07 - 3 bugs
- 10/31 - 0 bugs
- 10/24 - 1 bugs
- 10/17 - 1 bugs
- 10/10 - 5 bugs
- 10/03 - 6 bugs
Q3
- 09/26 - 3 bugs
- 09/19 - 3 bugs
- 09/12 - 0 bugs
- 09/05 - 3 bugs
- 08/29 - 3 bugs
- 08/22 - 3 bugs
- 08/14 - 3 bugs
- 08/08 - 0 bugs
- 08/01 - 3 bugs
- 07/25 - 0 bugs
- 07/18 - 3 bugs
- 07/11 - 3 bugs
- 07/04 - 3 bugs
Q2
- 06/27 - 3 bugs
- 06/20 - 6 bugs
- 06/13 - 3 bugs
- 06/06 - 1 bugs
- 05/30 - 3 bugs
- 05/23 - 3 bugs
- 05/16 - 6 bugs
- 05/09 - 1 bugs
- 05/02 - 12 bugs
- 04/25 - 6 bugs
- 04/18 - 6 bugs
- 04/11 - 3 bugs
- 04/04 - 5 bugs
Q1
- 03/28 - 4 bugs
- 03/21 - 6 bugs
- 03/14 - 3 bugs
- 03/07 - 2 bugs
- 02/28 - 3 bugs
- 02/21 - 6 bugs
- 02/14 - 5 bugs
- 02/07 - 7 bugs
- 01/31 - 4 bugs
- 01/24 - 0 bugs
- 01/17 - 3 bugs
- 01/10 - 8 bugs
2017
Q4
Q3
Q2
Q1
2016
Q4
Q3
- 09/28 - 1 bugs
- 09/21 - 5 bugs
- 09/14 - 6 bugs
- 09/07 - 7 bugs
- 08/31 - 19 bugs
- 08/24 - 22 bugs
- 08/17 - 27 bugs
- 08/10 - 15 bugs
- 08/03 - 30 bugs
- 07/27 - 33 bugs
- 07/20 - 18 bugs
- 07/13 - 32 bugs
- 07/06 - 4 bugs
Q2
- 06/29 - 15 bugs
- 06/22 - 4 bugs
- 06/15 - 0 bugs
- 06/08 - 0 bugs
- 06/01 - 1 bugs
- 05/25 - 0 bugs
- 05/18 - 2 bugs
- 05/11 - 6 bugs
- 05/04 - 1 bugs
- 04/20 - 4 bugs
- 04/20 - 5 bugs
- 04/13 - 3 bugs
- 04/06 - 6 bugs
Q1
- 03/30 - 2 bugs
- 03/23 - 7 bugs
- 03/16 - 7 bugs
- 03/09 - 2 bugs
- 03/02 - 2 bugs
- 02/24 - 4 bugs
- 02/17 - 4 bugs
- 02/10 - 1 bugs
- 02/03 - 3 bugs
- 01/27 - 1 bugs
- 01/20 - 3 bugs
- 01/13 - 9 bugs
- 01/06 - 2 bugs
2015
Q4
Q3
- 09/30 - 0 bugs
- 09/23 - 0 bugs
- 09/16 - 0 bugs
- 09/09 - 0 bugs
- 09/02 - 6 bugs
- 08/26 - 4 bugs
- 08/19 - 0 bugs
- 08/12 - 3 bugs
- 08/05 - 8 bugs
- 07/29 - 12 bugs (including the verification performed via Firefox 41 Aurora Testday)
- 07/22 - 14 bugs (including the verification performed via Firefox 40 beta 7 Testday)
- 07/15 - 3 bugs
- 07/08 - 10 bugs
- 07/01 - 9 bugs
Q2
- 06/24 - 8 bugs
- 06/17 - 13 bugs (including the verification performed via Firefox 39 beta 7 Testday)
- 06/10 - 5 bugs
- 06/03 - 4 bugs
- 05/27 - 4 bugs newcomers: Towkir, Amir, NaSb, Nuzhat
- 05/20 - 0 bugs
- 05/13 - 0 bugs
- 05/06 - 0 bugs
- 04/29 - 0 bugs
- 04/22 - 0 bugs
- 04/15 - 0 bugs
- 04/08 - 0 bugs
- 04/01 - 0 bugs