[go: nahoru, domu]

Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Up to 16 threads? #211

Open
jianshu93 opened this issue Nov 17, 2019 · 6 comments
Open

Up to 16 threads? #211

jianshu93 opened this issue Nov 17, 2019 · 6 comments

Comments

@jianshu93
Copy link

Actually it uses only 4-6 cores in real datasets no matter how many threads I assigned. Why limiting it to 16 cores only? How can I use 128 threads on my linux server.

@sklages
Copy link
sklages commented Nov 20, 2019

I/O is and will be limiting. That's probably why you see only a handful cores actually working.
Our fastq files range from 2G to 50G per read .. it sometimes takes ages to read the files (indepedently from tool).

@MADscientist314
Copy link

same issue. I am using a cluster with client servers that each have 48 cores with hyperthreading up to 192 and want to trim PE fastq files that have ~ 400 million reads. I am reading from a dedicated raid 0 striped server which mitigates the I/O issue, but this tool ends up taking way longer than Trimmomatic, simply due to the time that it is taking to read the files and limiting its use to 16 threads.

@Benjamin-Lee
Copy link

I just hit this too using a 96 core machine on AWS. It would be nice if it were a warning but then attempted to use all the cores.

@slambrechts
Copy link

So is this limited to 16 threads or not? Because I also get this warning

@ravisaroch
Copy link

Same issue..Why limit it to 16 threads only?

@sfchen
Copy link
Member
sfchen commented Feb 23, 2022

Since using >16 threads cannot make fastp faster. The bottleneck is I/O when thread number > 8.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

7 participants