[go: nahoru, domu]
Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Always stuck in step5 [5/8 - Executing task CreateClustersTask (Create new persons or update existing persons)] #712

Closed
3038922 opened this issue Dec 2, 2023 · 7 comments
Closed

Always stuck in step5 [5/8 - Executing task CreateClustersTask (Create new persons or update existing persons)] #712

3038922 opened this issue Dec 2, 2023 · 7 comments

Comments

@3038922
Copy link
3038922 commented Dec 2, 2023

I use external decoding and then he always gets stuck.Why is that?And my nextcloud's memory keeps taking up until it's full.My decoding VM sets up model 3.

 ⚡ root@nextcloud  ~  sudo -u www-data php /var/www/nextcloud/occ face:background_job
1/8 - Executing task CheckRequirementsTask (Check all requirements)
2/8 - Executing task CheckCronTask (Check that service is started from either cron or from command)
3/8 - Executing task DisabledUserRemovalTask (Purge all the information of a user when disable the analysis.)
4/8 - Executing task StaleImagesRemovalTask (Crawl for stale images (either missing in filesystem or under .nomedia) and remove them from DB)
5/8 - Executing task CreateClustersTask (Create new persons or update existing persons)
        Skipping cluster creation, not enough data (yet) collected. For cluster creation, you need either one of the following:
        * have 1000 faces already processed
        * or you need to have 95% of you images processed
        Use stats command to track progress
        Skipping cluster creation, not enough data (yet) collected. For cluster creation, you need either one of the following:
        * have 1000 faces already processed
        * or you need to have 95% of you images processed
        Use stats command to track progress
        Skipping cluster creation, not enough data (yet) collected. For cluster creation, you need either one of the following:
        * have 1000 faces already processed
        * or you need to have 95% of you images processed
        Use stats command to track progress
        Skipping cluster creation, not enough data (yet) collected. For cluster creation, you need either one of the following:
        * have 1000 faces already processed
        * or you need to have 95% of you images processed
        Use stats command to track progress
        Skipping cluster creation, not enough data (yet) collected. For cluster creation, you need either one of the following:
        * have 1000 faces already processed
        * or you need to have 95% of you images processed
        Use stats command to track progress
        Face clustering will be created for the first time.
        0 faces found for clustering
        0 clusters found after clustering
        Skipping cluster creation, not enough data (yet) collected. For cluster creation, you need either one of the following:
        * have 1000 faces already processed
        * or you need to have 95% of you images processed
        Use stats command to track progress
        Face clustering will be created for the first time.
@matiasdelellis
Copy link
Owner
matiasdelellis commented Jan 9, 2024 •
edited
Loading

Hi @3038922
I understand that you have many images analyzed, and many faces to clustering.. It is strange that you run out of resources, but it is a very demanding task. Can you use the face:stats command to contextualize?

If you are really running out of ram, you can add SWAP, and increase the php_limits

p.s: sorry for delay. I'm very busy. 😞

@vwbusguy
Copy link
vwbusguy commented Jun 3, 2024

I ran into this recently and it just took a very long time, but eventually made it through step 5. I had to run it outside of cron without the timeout since it was timing out on this step within 20 minutes before working on faces.

matiasdelellis added a commit that referenced this issue Jun 4, 2024
@matiasdelellis
Adds batch option for face clustering. Issue #712 (Among many others)
58e3e0e 
This improves the performance of clustering with many faces. In my
tests with 43 thousand faces, it takes 11,91 minutes to run the full
'face:background_job --cluster-mode' command. Using a batch size of 20
thousand the time is reduced to 4.47 minutes. With 5000, 1.71 minute
and with 2000 (Which is the minimum cut it takes only 54 seconds.

Against all odds, memory consumption does not increase in any way,
but it still has a disadvantage. The clusters... although they do not
increase their quantity as much (only 5%), these will generally be
smaller (Presumably of higher quality), but there will be more
clusters to give them names.

Well, this is another advanced option that will not be available in
the administrator interface.

occ config:app:set  facerecognition clustering_batch_size --value='1000' --type=integer
@matiasdelellis
Copy link
Owner

Hi both,
You can try applying the latest commit to your installation. They should help you a lot.

58e3e0e

@vwbusguy vwbusguy mentioned this issue Aug 6, 2024
Open
@vwbusguy
Copy link
vwbusguy commented Aug 6, 2024

That release fixed it for me (sorry I didn't reply earlier). 20k was the sweet spot in my case - still takes a few minutes per user (with around 100k faces per user), but the cluster quality was still more similar to before batching.

@rarealphacat
Copy link
rarealphacat commented Oct 19, 2024 •
edited
Loading

I ran into a type error of dlib_vector_length():

5/8 - Executing task CreateClustersTask (Create new persons or update existing persons)
Skipping cluster creation, not enough data (yet) collected. For cluster creation, you need either one of the following:
* have 1000 faces already processed
* or you need to have 95% of you images processed
Use stats command to track progress
yielding
Face clustering will be created for the first time.
There are 3020 faces for clustering
We will cluster with 2 batch(es) of 1510 faces
An unhandled exception has been thrown:
TypeError: dlib_vector_length(): Argument https://github.com/matiasdelellis/facerecognition/issues/1 ($x_arg) must be of type array, string given in /var/www/html/nextcloud/apps/facerecognition/lib/BackgroundJob/Tasks/CreateClustersTask.php:311
Stack trace:
#0 /var/www/html/nextcloud/apps/facerecognition/lib/BackgroundJob/Tasks/CreateClustersTask.php(311): dlib_vector_length()
https://github.com/matiasdelellis/facerecognition/issues/1 /var/www/html/nextcloud/apps/facerecognition/lib/BackgroundJob/Tasks/CreateClustersTask.php(177): OCA\FaceRecognition\BackgroundJob\Tasks\CreateClustersTask->getNewClusters()
https://github.com/matiasdelellis/facerecognition/issues/2 /var/www/html/nextcloud/apps/facerecognition/lib/BackgroundJob/Tasks/CreateClustersTask.php(90): OCA\FaceRecognition\BackgroundJob\Tasks\CreateClustersTask->createClusterIfNeeded()
https://github.com/matiasdelellis/facerecognition/issues/3 /var/www/html/nextcloud/apps/facerecognition/lib/BackgroundJob/BackgroundService.php(150): OCA\FaceRecognition\BackgroundJob\Tasks\CreateClustersTask->execute()
https://github.com/matiasdelellis/facerecognition/pull/4 /var/www/html/nextcloud/apps/facerecognition/lib/Command/BackgroundCommand.php(192): OCA\FaceRecognition\BackgroundJob\BackgroundService->execute()
https://github.com/matiasdelellis/facerecognition/issues/5 /var/www/html/nextcloud/3rdparty/symfony/console/Command/Command.php(298): OCA\FaceRecognition\Command\BackgroundCommand->execute()
https://github.com/matiasdelellis/facerecognition/pull/6 /var/www/html/nextcloud/3rdparty/symfony/console/Application.php(1040): Symfony\Component\Console\Command\Command->run()
https://github.com/matiasdelellis/facerecognition/pull/7 /var/www/html/nextcloud/3rdparty/symfony/console/Application.php(301): Symfony\Component\Console\Application->doRunCommand()
https://github.com/matiasdelellis/facerecognition/issues/8 /var/www/html/nextcloud/3rdparty/symfony/console/Application.php(171): Symfony\Component\Console\Application->doRun()
https://github.com/matiasdelellis/facerecognition/issues/9 /var/www/html/nextcloud/lib/private/Console/Application.php(213): Symfony\Component\Console\Application->run()
https://github.com/matiasdelellis/facerecognition/issues/10 /var/www/html/nextcloud/console.php(102): OC\Console\Application->run()
https://github.com/matiasdelellis/facerecognition/issues/11 /var/www/html/nextcloud/occ(11): require_once('...')

@vwbusguy
Copy link
vwbusguy commented Oct 21, 2024 •
edited
Loading

Argument https://github.com/matiasdelellis/facerecognition/issues/1 ($x_arg) must be of type array

Why is a GitHub issue link being passed as an argument? This looks like an unrelated problem to the OP.

I think GitHub might be expanding #1 in your log and your issue might be #780 .

@matiasdelellis
Copy link
Owner

Hi @rarealphacat

I ran into a type error of dlib_vector_length():

As @vwbusguy comments, this error has its own report.. and as for the original report it seems fixed.

p.s: In the next release due this week there is a 40% improvement in memory usage using batches. 😉

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@matiasdelellis @vwbusguy @rarealphacat @3038922