[go: nahoru, domu]
Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Workqueue depth and duration metrics are not accurate #674

Open
Tracked by #4620
turkenh opened this issue Feb 23, 2024 · 2 comments
Open
Tracked by #4620

Workqueue depth and duration metrics are not accurate #674

turkenh opened this issue Feb 23, 2024 · 2 comments
Labels
bug Something isn't working metrics observability

Comments

@turkenh
Copy link
Member
turkenh commented Feb 23, 2024 •
edited
Loading

What happened?

During scalability testing efforts leading crossplane-contrib/provider-kubernetes#203, I noticed that the controller runtime metrics like workqueue_depth and workqueue_queue_duration_seconds are not accurate or not reflecting the state of the system as expected.

See the workqueue depth and duration graphs for "1m, 10" here.

How can we reproduce it?

Checkout https://github.com/turkenh/provider-kubernetes-scalability/tree/repro-xp-no-metric

just setup
just create_x_objects 1 1000

just help_launch_prometheus

just help_launch_grafana
# import dashboard json there

What environment did it happen in?

Crossplane version: v1.14.5
Provider Kubernetes: v0.11.4
@turkenh turkenh added the bug Something isn't working label Feb 23, 2024
@negz
Copy link
Member
negz commented Feb 26, 2024

@turkenh Do you have any theories on why depth is inaccurate? I think per crossplane/crossplane#5415 (comment) we have a good theory why duration would be inaccurate.

@negz
Copy link
Member
negz commented Feb 26, 2024

Ah - maybe rate limiters calling AddAfter (as in "add to queue after some duration") means reconciles are spending time in limbo. They're not yet being processed, but also not yet even in the work queue?

@jbw976 jbw976 mentioned this issue Mar 4, 2024
Open
7 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working metrics observability
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@negz @turkenh