core: expose error stack on errored audits #14491

connorjclark · 2022-11-02T21:27:32Z

This adds an optional errorStack to an error'd audit product.

Also included is some better handling for our .evaluate function for when there is a syntax error.

example:

"hreflang": {
  "id": "hreflang",
  "title": "Document has a valid `hreflang`",
  "description": "hreflang links tell search engines what version of a page they should list in search results for a given language or region. [Learn more about `hreflang`](https://web.dev/hreflang/).",
  "score": null,
  "scoreDisplayMode": "error",
  "errorMessage": "Required LinkElements gatherer encountered an error: browserElementsHAHA is not defined",
  "errorStack": "ReferenceError: browserElementsHAHA is not defined\n    at getLinkElementsInDOM (_lighthouse-eval.js:286:22)\n    at _lighthouse-eval.js:308:3\n    at _lighthouse-eval.js:309:7"
}

core/lib/lh-error.js

adamraine · 2022-11-02T22:18:45Z

core/gather/driver/execution-context.js

+      const evaluationError = new Error(`Runtime.evaluate exception: ${message}`);
+      if (ex.exception?.description && ex.stackTrace) {
+        // The description contains the stack trace formatted as expected, if present.
+        evaluationError.stack = ex.exception.description;


Could we combine the error stacks so the error stack also tells us where in LH code it occured?

Ex: https://github.com/GoogleChrome/lighthouse/blob/main/core/scripts/pptr-run-devtools.js#L87

If we want that, we should just recurse the error.cause chain and join all the stacks for errorStack.

EDIT: read the code, don't see what that has to do with stacks but ya adding maybe the first few characters of the expression could be good for context here. We don't have the name/Function at this layer, and the entire thing would be way too long (axe...)

I think that's what @adamraine meant, otherwise you lose the evaluate() call stack on the LH side. In theory not a huge deal: if you're getting an error this early, you were probably just editing the js and know where the call was coming from, but it can be an issue for evaluate calls originating in many places

I decided that modifying stack is probably a bad idea. Just added the extra details to the error message instead, and then was pleasantly surprised to see that v8 will parse the message for additional stack frames and add it to stacks itself!

Here's what printing evaluationError looks like now:

Error: Runtime.evaluate exception Expression: (() => { return (function computeBenchmarkIndex() { /** * The GC-heavy benchmark that creates a stri ---- (elided) ReferenceError: Prommmise is not defined at wrapInNativePromise (_lighthouse-eval.js:8:9) at _lighthouse-eval.js:83:8 at ExecutionContext._evaluateInContext (file:///Users/cjamcl/src/lighthouse/core/gather/driver/execution-context.js:139:31) at processTicksAndRejections (node:internal/process/task_queues:96:5) at async ExecutionContext.evaluateAsync (file:///Users/cjamcl/src/lighthouse/core/gather/driver/execution-context.js:173:14) at async getBenchmarkIndex (file:///Users/cjamcl/src/lighthouse/core/gather/driver/environment.js:54:20) at async getBaseArtifacts (file:///Users/cjamcl/src/lighthouse/core/gather/base-artifacts.js:21:26) at async _setup (file:///Users/cjamcl/src/lighthouse/core/gather/navigation-runner.js:56:25) at async file:///Users/cjamcl/src/lighthouse/core/gather/navigation-runner.js:336:31 at async Function.gather (file:///Users/cjamcl/src/lighthouse/core/runner.js:172:21) at async navigationGather (file:///Users/cjamcl/src/lighthouse/core/gather/navigation-runner.js:318:21) at async navigation (file:///Users/cjamcl/src/lighthouse/core/index.js:94:24)

did something change? Here's the error stack I get on hreflang if I make an error in LinkElements:

errorMessage: Required LinkElements gatherer encountered an error: getElementsInDocumentz is not defined errorStack: Error: LighthouseError: ERRORED_REQUIRED_ARTIFACT at Runner._runAudit (~/lighthouse/core/runner.js:431:25) at Runner._runAudits (~/lighthouse/core/runner.js:370:40) at process.processTicksAndRejections (node:internal/process/task_queues:95:5) at async Runner.audit (~/lighthouse/core/runner.js:62:32) at async runLighthouse (~/lighthouse/cli/run.js:250:8) at async ~/lighthouse/cli/index.js:10:1 at <anonymous>:1:1

ah, to get a populated response.exceptionDetails you actually need invalid JS or the like, not just an error. Adding some random characters, I now get

errorMessage: Required LinkElements gatherer encountered an error: Runtime.evaluate exception Expression: (() => { function getNodeDetails(element) { function truncate(string, characterLimit) { const Util = ---- (elided) Parse error at: 294:6 SyntaxError: Unexpected token '*' at <anonymous>:1:1 errorStack: LighthouseError: ERRORED_REQUIRED_ARTIFACT at Runner._runAudit (~/lighthouse/core/runner.js:431:25) at Runner._runAudits (~/lighthouse/core/runner.js:370:40) at process.processTicksAndRejections (node:internal/process/task_queues:95:5) at async Runner.audit (~/lighthouse/core/runner.js:62:32) at async runLighthouse (~/lighthouse/cli/run.js:250:8) at async ~/lighthouse/cli/index.js:10:1 at <anonymous>:1:1

errorMessage seems to have some of the useful new bits, but formatted poorly for an error message.

Breaking in _evaluateInContext I see the expanded evaluationError

Error: Runtime.evaluate exception Expression: (() => { function getNodeDetails(element) { function truncate(string, characterLimit) { const Util = ---- (elided) Parse error at: 294:6 SyntaxError: Unexpected token '*' at ExecutionContext._evaluateInContext (~/lighthouse/core/gather/driver/execution-context.js:140:31) at process.processTicksAndRejections (node:internal/process/task_queues:95:5) at async ExecutionContext.evaluateAsync (~/lighthouse/core/gather/driver/execution-context.js:173:14) at async LinkElements._getArtifact (~/lighthouse/core/gather/gatherers/link-elements.js:183:17) at async ~/lighthouse/core/gather/runner-helpers.js:101:24 at async collectPhaseArtifacts (~/lighthouse/core/gather/runner-helpers.js:118:5) at async _computeNavigationResult (~/lighthouse/core/gather/navigation-runner.js:200:5) at async _navigations (~/lighthouse/core/gather/navigation-runner.js:297:30) at async ~/lighthouse/core/gather/navigation-runner.js:370:27 at async Runner.gather (~/lighthouse/core/runner.js:218:21) {stack: 'Error: Runtime.evaluate exception Expression:…~/lighthouse/core/runner.js:218:21)', message: 'Runtime.evaluate exception Expression: (() =… at: 294:6 SyntaxError: Unexpected token '*''}

Looking at just evaluationError.stack:

'Error: Runtime.evaluate exception Expression: (() => { function getNodeDetails(element) { function truncate(string, characterLimit) { const Util = ---- (elided) Parse error at: 294:6 SyntaxError: Unexpected token '*' at ExecutionContext._evaluateInContext (~/lighthouse/core/gather/driver/execution-context.js:140:31) at process.processTicksAndRejections (node:internal/process/task_queues:95:5) at async ExecutionContext.evaluateAsync (~/lighthouse/core/gather/driver/execution-context.js:172:14) at async LinkElements._getArtifact (~/lighthouse/core/gather/gatherers/link-elements.js:183:17) at async ~/lighthouse/core/gather/runner-helpers.js:101:24 at async collectPhaseArtifacts (~/lighthouse/core/gather/runner-helpers.js:118:5) at async _computeNavigationResult (~/lighthouse/core/gather/navigation-runner.js:200:5) at async _navigations (~/lighthouse/core/gather/navigation-runner.js:297:30) at async ~/lighthouse/core/gather/navigation-runner.js:370:27 at async Runner.gather (~/lighthouse/core/runner.js:218:21)'

vs the old new Error(`Evaluation exception: ${errorMessage}`)'s stack:

Error: Evaluation exception: SyntaxError: Unexpected token '*' at ExecutionContext._evaluateInContext (~/lighthouse/core/gather/driver/execution-context.js:140:31) at process.processTicksAndRejections (node:internal/process/task_queues:95:5) at async ExecutionContext.evaluateAsync (~/lighthouse/core/gather/driver/execution-context.js:172:14) at async LinkElements._getArtifact (~/lighthouse/core/gather/gatherers/link-elements.js:183:17) at async ~/lighthouse/core/gather/runner-helpers.js:101:24 at async collectPhaseArtifacts (~/lighthouse/core/gather/runner-helpers.js:118:5) at async _computeNavigationResult (~/lighthouse/core/gather/navigation-runner.js:200:5) at async _navigations (~/lighthouse/core/gather/navigation-runner.js:297:30) at async ~/lighthouse/core/gather/navigation-runner.js:370:27 at async Runner.gather (~/lighthouse/core/runner.js:218:21)

looks exactly the same except the new version has more non-stack stuff at the top

brendankenny · 2022-11-03T17:53:03Z

core/gather/driver/execution-context.js

+      const evaluationError = new Error(`Runtime.evaluate exception: ${message}`);
+      if (ex.exception?.description && ex.stackTrace) {
+        // The description contains the stack trace formatted as expected, if present.
+        evaluationError.stack = ex.exception.description;


I think that's what @adamraine meant, otherwise you lose the evaluate() call stack on the LH side. In theory not a huge deal: if you're getting an error this early, you were probably just editing the js and know where the call was coming from, but it can be an issue for evaluate calls originating in many places

brendankenny · 2022-11-03T17:57:59Z

core/lib/lh-error.js

   */
-  constructor(errorDefinition, properties) {
-    super(errorDefinition.code);
+  constructor(errorDefinition, properties, cause) {


copying the Error options param might be better instead of just having a cause param, both to match the parent's style and so it's a named argument at call sites

done - but now need to bump tsc to get ErrorOptions to be unknown instead of Error.

core/lib/lh-error.js

core/test/lib/asset-saver-test.js

core/test/runner-test.js

core/scripts/update-flow-fixtures.js

brendankenny · 2022-11-03T19:04:38Z

core/scripts/cleanup-LHR-for-diff.js

      auditResult.description = '**Excluded from diff**';
    }
+    if (auditResult.errorStack) {
+      auditResult.errorStack = auditResult.errorStack.replaceAll(baseCallFrameUrl.href, '');


It's still pretty of annoying to update sample LHRs for line number changes in any of the files in an error stack. Could do like errorStack.replaceAll(/$[^)]+$/g, '()') but maybe we should snip it completely

vercel · 2023-01-04T00:22:53Z

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Updated
lighthouse	🔄 Building (Inspect)		Jan 4, 2023 at 0:22AM (UTC)

core/lib/asset-saver.js

core/gather/driver/execution-context.js

brendankenny · 2023-05-04T20:02:38Z

core/gather/driver/execution-context.js

+      const evaluationError = new Error(`Runtime.evaluate exception: ${message}`);
+      if (ex.exception?.description && ex.stackTrace) {
+        // The description contains the stack trace formatted as expected, if present.
+        evaluationError.stack = ex.exception.description;


did something change? Here's the error stack I get on hreflang if I make an error in LinkElements:

errorMessage: Required LinkElements gatherer encountered an error: getElementsInDocumentz is not defined errorStack: Error: LighthouseError: ERRORED_REQUIRED_ARTIFACT at Runner._runAudit (~/lighthouse/core/runner.js:431:25) at Runner._runAudits (~/lighthouse/core/runner.js:370:40) at process.processTicksAndRejections (node:internal/process/task_queues:95:5) at async Runner.audit (~/lighthouse/core/runner.js:62:32) at async runLighthouse (~/lighthouse/cli/run.js:250:8) at async ~/lighthouse/cli/index.js:10:1 at <anonymous>:1:1

core/runner.js

connorjclark requested a review from a team as a code owner November 2, 2022 21:27

connorjclark requested review from adamraine and removed request for a team November 2, 2022 21:27

devtools-bot assigned adamraine Nov 2, 2022

devtools-bot added the waiting4reviewer label Nov 2, 2022

vercel bot deployed to Preview November 2, 2022 21:31 View deployment

brendankenny reviewed Nov 2, 2022

View reviewed changes

core/lib/lh-error.js Outdated Show resolved Hide resolved

adamraine reviewed Nov 2, 2022

View reviewed changes

vercel bot deployed to Preview November 3, 2022 00:35 View deployment

brendankenny reviewed Nov 3, 2022

View reviewed changes

connorjclark force-pushed the error-stack branch from 5a3adcf to 1210b54 Compare January 3, 2023 21:44

vercel bot deployed to Preview January 3, 2023 21:45 View deployment

connorjclark added 2 commits January 3, 2023 15:28

core: expose error stack on errored audits

54b731b

pr updates

42724b5

connorjclark force-pushed the error-stack branch from 1210b54 to aa0c628 Compare January 4, 2023 00:22

vercel bot deployed to Preview January 4, 2023 00:24 View deployment

add test

d041926

connorjclark force-pushed the error-stack branch from aa0c628 to d041926 Compare January 4, 2023 00:28

vercel bot deployed to Preview January 4, 2023 00:28 View deployment

connorjclark added 2 commits May 4, 2023 09:09

Merge remote-tracking branch 'origin/main' into error-stack

7d219a2

revert changes to cleanup lhr file

5e8293e

vercel bot deployed to Preview May 4, 2023 16:28 View deployment

fix test bitrot

046bae7

vercel bot deployed to Preview May 4, 2023 16:43 View deployment

fix unit test delta bc of node 18 vs 16

cf92c4c

vercel bot deployed to Preview May 4, 2023 17:35 View deployment

adamraine approved these changes May 4, 2023

View reviewed changes

brendankenny reviewed May 4, 2023

View reviewed changes

core/runner.js Outdated Show resolved Hide resolved

pr

5a077d8

vercel bot deployed to Preview May 4, 2023 21:24 View deployment

oops

3149532

vercel bot deployed to Preview May 4, 2023 21:59 View deployment

Merge branch 'main' into error-stack

2a678d4

connorjclark added the land-when-ci-is-green label May 31, 2023

vercel bot deployed to Preview May 31, 2023 16:54 View deployment

devtools-bot merged commit 186e180 into main May 31, 2023
33 checks passed

devtools-bot deleted the error-stack branch May 31, 2023 17:36

devtools-bot removed the land-when-ci-is-green label May 31, 2023

connorjclark mentioned this pull request Jun 20, 2023

core(proto): add errorStack to AuditResult #15187

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

core: expose error stack on errored audits #14491

core: expose error stack on errored audits #14491

core: expose error stack on errored audits #14491

core: expose error stack on errored audits #14491

Conversation

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment