[go: nahoru, domu]

Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add Prediction Smoothing #696

Merged
merged 29 commits into from
Jan 24, 2024
Merged

Add Prediction Smoothing #696

merged 29 commits into from
Jan 24, 2024

Conversation

yeldarby
Copy link
Contributor
@yeldarby yeldarby commented Dec 27, 2023

Description

Adds a Smoother class which averages the last several predictions to decrease noise (like wobbly or flickering boxes).

Before & after smoothing:

smoothed-grocery-example-480.mp4

Type of change

  • New feature (non-breaking change which adds functionality)

How has this change been tested, please provide a testcase or example of how you tested the change?

Installed locally & tested on several videos/models with InferencePipeline.

Any specific deployment considerations

No.

Docs

  • Docs updated? What were the changes: Added new docs page with video & example code.

@yeldarby yeldarby added the enhancement New feature or request label Dec 27, 2023
@SkalskiP SkalskiP mentioned this pull request Dec 28, 2023
4 tasks
supervision/detection/smoother.py Outdated Show resolved Hide resolved
supervision/detection/smoother.py Outdated Show resolved Hide resolved
> _On the left are the model's raw predictions,
> on the right is the output of Smoother._

## Example Usage:
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we show a more general example? I assume Smoother can work with any object detection model.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sure, what do you have in mind? I figured person detection with COCO using one of the videos from assets was about as general as it gets to demonstrate the functionality but we can change it to something else.

Good candidate videos to communicate the functionality would have

  • A small number of objects moving fairly subtly (otherwise it's really distracting trying to understand what you're supposed to be comparing side by side)
  • Prediction jitter coming from the model

For example, while it works with vehicles-2.mp4, it's not a good example video because there's too much going on and the things are moving too fast to be able to grok what's going on at the detection by detection level.

mkdocs.yml Outdated Show resolved Hide resolved
supervision/detection/smoother.py Outdated Show resolved Hide resolved
supervision/detection/smoother.py Outdated Show resolved Hide resolved
supervision/detection/smoother.py Outdated Show resolved Hide resolved
supervision/detection/smoother.py Outdated Show resolved Hide resolved
supervision/detection/smoother.py Outdated Show resolved Hide resolved
supervision/detection/smoother.py Outdated Show resolved Hide resolved
@SkalskiP
Copy link
Collaborator

@yeldarby, please let me know once the PR is ready for the next review round.

@yeldarby
Copy link
Contributor Author

PR is updated in response to your comments here & @capjamesg's comments on Slack.

@SkalskiP
Copy link
Collaborator

@yeldarby I started the second round of review, but I got confused and decided to ask here:

  • Is tracker_length still needed? Because it is not used, it is not documented and generally unrelated to the Smoother task (smoothing boxes).
  • If tracker_length is not needed, do we need self.track_starts and self.track_ends ? self.track_starts is only there to support tracker_length and self.track_ends is used in two places but the second place can be easily implemented differently.
  • If we don't need self.track_starts and self.track_ends we probably do not need self.current_frame.

All around, the whole logic can get a lot easier if we drop tracker_length, which, as I said, is not used; it is not documented and, in general, is unrelated to the Smoother task.

@yeldarby
Copy link
Contributor Author

Is tracker_length still needed?

I'm using it in some client code to create animated Annotators. The frame of the animation for a detection needs to be tracker_length%num_frames.

Also planning to use it to let people choose if we should wait to display a detection until it's been seen some number of times so that if something is detected in a frame or two but not again it doesn't flicker onto the screen (but haven't implemented that here just yet).

Because it is not used, it is not documented

Good call; if we keep it here I'll document it.

and generally unrelated to the Smoother task (smoothing boxes).

The entry delay will be related to this (though that's easier since you should be able to just take the length of the non-None detections from the tracker; it'd be a tiny bit slower to sum vs reference a count but that's negligible given length will typically be small).

But yeah I think this probably makes a bit more structural sense at the Tracker level (even in my primary use-case you should be able to have an animated Annotator w/o using Smoother); want me to move it there?

@yeldarby
Copy link
Contributor Author

Updated to remove tracker_length and track_starts.

for track_id in self.tracks:
self.tracks[track_id] = deque(self.tracks[track_id], maxlen=length)

def tracker_length(self, tracker_id: int) -> int:
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It looks like this method is unused. We don't expose it in docs. Can we just remove it?


return self.get_smoothed_detections()

def get_track(self, track_id: int) -> Optional[dict]:
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I believe we have an incorrect return type here. Shouldn't it be Optional[Detections]?

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Updated in my latest commit.

if len(track) == 0:
return None

ret = track[0]
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It looks like you mutate that object every time. Is that intended? Let's create a copy here?

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Updated in my latest commit.

self.tracks[tracker_id].append(detections[detection_idx])
self.track_ends[tracker_id] = self.current_frame

for track_id in self.tracks:
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We have no state-clearing mechanism. After some time, most of the tracks we store will be dead.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@SkalskiP How would you recommend we clear state?

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we have an explicit method DetectionsSmoother.clear(), or some logic here?

from supervision.detection.core import Detections


class Smoother:
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What do you think about changing the name to DetectionsSmoother?

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Updated in my latest commit.

@capjamesg capjamesg self-assigned this Jan 23, 2024
@capjamesg
Copy link
Collaborator

@SkalskiP In my latest change, I add logic to purge unused tracker IDs:

used_tracker_ids = {}

for detection_idx in range(len(detections)):
    tracker_id = detections.tracker_id[detection_idx]
    if tracker_id is None:
        # skip detections without a tracker id
        continue

    if self.tracks.get(tracker_id, None) is None:
        self.tracks[tracker_id] = deque(maxlen=self.length)

    self.tracks[tracker_id].append(detections[detection_idx])
    self.track_ends[tracker_id] = self.current_frame

    used_tracker_ids[tracker_id] = True

for track_id in list(self.tracks.keys()):
    if track_id not in used_tracker_ids:
        del self.tracks[track_id]
        del self.track_ends[track_id]

Here is an example of the smoothing running on a video:

container_output_1.1.mp4

@SkalskiP SkalskiP merged commit 80b7eec into develop Jan 24, 2024
8 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants