How to limit number of threads for python interpreter?
up vote
2
down vote
favorite
I want to know whether it is possible to limit the number of threads python interpreter can spawn when running a script.
I am not using threading myself, but my script uses modules like pandas, numpy, and others which I assume would be using threading in the background. The script works fine, but when I run it on an LSF cluster, it spawns too many threads crossing thread limit per job (for the cluster) and as a result, crashes.
I would like to restrict the python interpreter to not allow more than, say, 100, threads at a time. Is it possible to run a python script on an LSF server with such a restriction on thread count?
Thanks for your help.
P.S.: My colleague faces the same issue with java apps, so a solution for that would be cool as well.
java python multithreading hpc lsf
asked Nov 8 at 10:03
Manish Goel
341213
add a comment |
up vote
2
down vote
favorite
I want to know whether it is possible to limit the number of threads python interpreter can spawn when running a script.
I am not using threading myself, but my script uses modules like pandas, numpy, and others which I assume would be using threading in the background. The script works fine, but when I run it on an LSF cluster, it spawns too many threads crossing thread limit per job (for the cluster) and as a result, crashes.
I would like to restrict the python interpreter to not allow more than, say, 100, threads at a time. Is it possible to run a python script on an LSF server with such a restriction on thread count?
Thanks for your help.
P.S.: My colleague faces the same issue with java apps, so a solution for that would be cool as well.
java python multithreading hpc lsf
asked Nov 8 at 10:03
Manish Goel
341213
Are you using vanilla python to run scripts? Are you aware of the Python GIL ? Also, see stackoverflow.com/questions/30791550/…
– RottenCandy
Nov 8 at 10:30
add a comment |
up vote
2
down vote
favorite
up vote
2
down vote
favorite
I want to know whether it is possible to limit the number of threads python interpreter can spawn when running a script.
I am not using threading myself, but my script uses modules like pandas, numpy, and others which I assume would be using threading in the background. The script works fine, but when I run it on an LSF cluster, it spawns too many threads crossing thread limit per job (for the cluster) and as a result, crashes.
I would like to restrict the python interpreter to not allow more than, say, 100, threads at a time. Is it possible to run a python script on an LSF server with such a restriction on thread count?
Thanks for your help.
P.S.: My colleague faces the same issue with java apps, so a solution for that would be cool as well.
java python multithreading hpc lsf
asked Nov 8 at 10:03
Manish Goel
341213
I want to know whether it is possible to limit the number of threads python interpreter can spawn when running a script.
I am not using threading myself, but my script uses modules like pandas, numpy, and others which I assume would be using threading in the background. The script works fine, but when I run it on an LSF cluster, it spawns too many threads crossing thread limit per job (for the cluster) and as a result, crashes.
I would like to restrict the python interpreter to not allow more than, say, 100, threads at a time. Is it possible to run a python script on an LSF server with such a restriction on thread count?
Thanks for your help.
P.S.: My colleague faces the same issue with java apps, so a solution for that would be cool as well.
java python multithreading hpc lsf
java python multithreading hpc lsf
asked Nov 8 at 10:03
Manish Goel
341213
asked Nov 8 at 10:03
Manish Goel
341213
asked Nov 8 at 10:03
Manish Goel
341213
asked Nov 8 at 10:03
Manish Goel
341213
asked Nov 8 at 10:03
Manish Goel
341213
341213
Are you using vanilla python to run scripts? Are you aware of the Python GIL ? Also, see stackoverflow.com/questions/30791550/…
– RottenCandy
Nov 8 at 10:30
add a comment |
Are you using vanilla python to run scripts? Are you aware of the Python GIL ? Also, see stackoverflow.com/questions/30791550/…
– RottenCandy
Nov 8 at 10:30
Are you using vanilla python to run scripts? Are you aware of the Python GIL ? Also, see stackoverflow.com/questions/30791550/…
– RottenCandy
Nov 8 at 10:30
Are you using vanilla python to run scripts? Are you aware of the Python GIL ? Also, see stackoverflow.com/questions/30791550/…
– RottenCandy
Nov 8 at 10:30
add a comment |
active
oldest
votes
active
oldest
votes
active
oldest
votes
active
oldest
votes
active
oldest
votes
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53205405%2fhow-to-limit-number-of-threads-for-python-interpreter%23new-answer', 'question_page');
}
);
Post as a guest
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Are you using vanilla python to run scripts? Are you aware of the Python GIL ? Also, see stackoverflow.com/questions/30791550/…
– RottenCandy
Nov 8 at 10:30