Add per tenant bytes counter #331
Conversation
Signed-off-by: Daniel González Lopes <danielgonzalezlopes@gmail.com>
Signed-off-by: Daniel González Lopes <danielgonzalezlopes@gmail.com>
|
Hi thanks for this submission, it's really good to add metrics that would be useful. Based on the location of this metric, maybe the name "ingester_bytes_written_total" is good? Bytes "processed" might be better reserved for the input side of the ingester, instead of the output side. |
modules/ingester/instance.go
Outdated
| @@ -111,7 +118,7 @@ func (i *instance) CutCompleteTraces(cutoff time.Duration, immediate bool) error | |||
| if err != nil { | |||
| return err | |||
| } | |||
|
|
|||
| i.bytesProcessedTotal.Add(float64(len(out))) | |||
There was a problem hiding this comment.
Should this be after the err check? Depends on how we define the metric, as attempted vs succeeded.
There was a problem hiding this comment.
I went with the attempted approach. But probably succeded makes more sense here 🤔
btw, I agree with the metric name. Love your proposal! I'll change it.
There was a problem hiding this comment.
So I just want to make a note about this spot:
- This is occurring after a trace is considered complete, so it will be delayed a bit while the trace is aggregating spans.
- On rollout we'd expect to see this metric to spike while ingesters are flushing their in memory traces to disk
- This metric is post RF. So it will be ~1x, ~2x or ~3x the number of pushed bytes depending on that setting.
Also, I agree with @mdisibio. May as well put this after the Write to headblock has succeeded.
Signed-off-by: Daniel González Lopes <danielgonzalezlopes@gmail.com>
Signed-off-by: Daniel González Lopes <danielgonzalezlopes@gmail.com>
What this PR does:
This PR adds a
per tenant bytes countermetric. I went with the distributor approach!The metric is increased when we cut the complete traces.
Which issue(s) this PR fixes:
Fixes #223
Checklist
CHANGELOG.mdupdated - the order of entries should be[CHANGE],[FEATURE],[ENHANCEMENT],[BUGFIX]