How to limit number of threads for python interpreter?

Multi tool use
Multi tool use











up vote
2
down vote

favorite
1












I want to know whether it is possible to limit the number of threads python interpreter can spawn when running a script.



I am not using threading myself, but my script uses modules like pandas, numpy, and others which I assume would be using threading in the background. The script works fine, but when I run it on an LSF cluster, it spawns too many threads crossing thread limit per job (for the cluster) and as a result, crashes.



I would like to restrict the python interpreter to not allow more than, say, 100, threads at a time. Is it possible to run a python script on an LSF server with such a restriction on thread count?



Thanks for your help.



P.S.: My colleague faces the same issue with java apps, so a solution for that would be cool as well.






java python multithreading hpc lsf







share|improve this question






















  • Are you using vanilla python to run scripts? Are you aware of the Python GIL ? Also, see stackoverflow.com/questions/30791550/…
    – RottenCandy
    Nov 8 at 10:30

















up vote
2
down vote

favorite
1












I want to know whether it is possible to limit the number of threads python interpreter can spawn when running a script.



I am not using threading myself, but my script uses modules like pandas, numpy, and others which I assume would be using threading in the background. The script works fine, but when I run it on an LSF cluster, it spawns too many threads crossing thread limit per job (for the cluster) and as a result, crashes.



I would like to restrict the python interpreter to not allow more than, say, 100, threads at a time. Is it possible to run a python script on an LSF server with such a restriction on thread count?



Thanks for your help.



P.S.: My colleague faces the same issue with java apps, so a solution for that would be cool as well.






java python multithreading hpc lsf







share|improve this question






















  • Are you using vanilla python to run scripts? Are you aware of the Python GIL ? Also, see stackoverflow.com/questions/30791550/…
    – RottenCandy
    Nov 8 at 10:30















up vote
2
down vote

favorite
1









up vote
2
down vote

favorite
1






1





I want to know whether it is possible to limit the number of threads python interpreter can spawn when running a script.



I am not using threading myself, but my script uses modules like pandas, numpy, and others which I assume would be using threading in the background. The script works fine, but when I run it on an LSF cluster, it spawns too many threads crossing thread limit per job (for the cluster) and as a result, crashes.



I would like to restrict the python interpreter to not allow more than, say, 100, threads at a time. Is it possible to run a python script on an LSF server with such a restriction on thread count?



Thanks for your help.



P.S.: My colleague faces the same issue with java apps, so a solution for that would be cool as well.






java python multithreading hpc lsf







share|improve this question













I want to know whether it is possible to limit the number of threads python interpreter can spawn when running a script.



I am not using threading myself, but my script uses modules like pandas, numpy, and others which I assume would be using threading in the background. The script works fine, but when I run it on an LSF cluster, it spawns too many threads crossing thread limit per job (for the cluster) and as a result, crashes.



I would like to restrict the python interpreter to not allow more than, say, 100, threads at a time. Is it possible to run a python script on an LSF server with such a restriction on thread count?



Thanks for your help.



P.S.: My colleague faces the same issue with java apps, so a solution for that would be cool as well.






java python multithreading hpc lsf




java python multithreading hpc lsf






share|improve this question













share|improve this question











share|improve this question




share|improve this question










asked Nov 8 at 10:03









Manish Goel

341213




341213












  • Are you using vanilla python to run scripts? Are you aware of the Python GIL ? Also, see stackoverflow.com/questions/30791550/…
    – RottenCandy
    Nov 8 at 10:30




















  • Are you using vanilla python to run scripts? Are you aware of the Python GIL ? Also, see stackoverflow.com/questions/30791550/…
    – RottenCandy
    Nov 8 at 10:30


















Are you using vanilla python to run scripts? Are you aware of the Python GIL ? Also, see stackoverflow.com/questions/30791550/…
– RottenCandy
Nov 8 at 10:30






Are you using vanilla python to run scripts? Are you aware of the Python GIL ? Also, see stackoverflow.com/questions/30791550/…
– RottenCandy
Nov 8 at 10:30



















active

oldest

votes











Your Answer






StackExchange.ifUsing("editor", function () {
StackExchange.using("externalEditor", function () {
StackExchange.using("snippets", function () {
StackExchange.snippets.init();
});
});
}, "code-snippets");

StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "1"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);

StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});

function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});


}
});














 

draft saved


draft discarded


















StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53205405%2fhow-to-limit-number-of-threads-for-python-interpreter%23new-answer', 'question_page');
}
);

Post as a guest























active

oldest

votes













active

oldest

votes









active

oldest

votes






active

oldest

votes
















 

draft saved


draft discarded



















































 


draft saved


draft discarded














StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53205405%2fhow-to-limit-number-of-threads-for-python-interpreter%23new-answer', 'question_page');
}
);

Post as a guest






































































l6 nj,NfAWvKoew4i0nIZX,oxsx,M535R3C8KehYwYA4WPDq 4M5EEPD0xCht2edxQ
-
HoTi4oTIH8j2Tw5p,nOGG,4xkoADd6QPUsFUYbc1WCb Ymo 92oai2GrYc

Popular posts from this blog

Schultheiß

Image
Der Titel dieses Artikels ist mehrdeutig. Weitere Bedeutungen sind unter Schultheiß (Begriffsklärung) aufgeführt. Ein Schultheiß. Holzschnitt von Peter Flötner (16. Jahrhundert) Der Schultheiß oder Schuldheiß (von althochdeutsch  sculdheizo ‚Leistung Befehlender‘, vgl. mittelniederdeutsch  schult(h)ēte , latinisiert (mittellat.) sculte(t)us , schwäbisch heute noch Schultes für „Bürgermeister“) bezeichnet einen in vielen westgermanischen Rechtsordnungen vorgesehenen Beamten, der Schuld heischt : Er hatte im Auftrag seines Herren (Landesherrn, Stadtherrn, Grundherrn) die Mitglieder einer Gemeinde zur Leistung ihrer Schuldigkeit anzuhalten, also Abgaben einzuziehen oder für das Beachten anderer Verpflichtungen Sorge zu tragen. Sprachliche Varianten des Schultheißen sind Schulte , Schultes oder Schulze . Früher wurde zwischen dem Stadtschulzen und dem Dorfschulzen unterschieden. In der städtischen Gerichts- und Gemeindeverfassung war er ein vom städtischen Rat oder vom L...
Read more

how to define a CAPL function taking a sysvar argument

Image
up vote 0 down vote favorite In Vector CANoe, is it possible to define a function that takes a system variable argument like the system function TestWaitForSignalMatch() ? For my use case it is not sufficient to supply the current value of the system variable because I want to pass the system variable to TestWaitForSignalMatch() or similar system functions. The CANoe help seems to show examples: long TestWaitForSignalMatch (Signal aSignal, float aCompareValue, dword aTimeout); // form 1 long TestWaitForSignalMatch (sysvar aSysVar, float aCompareValue, dword aTimeout); // form 3 I tried like this void foo(sysvar aSysvar) {} ^ or this void foo(sysvar *aSysvar) {} ^ but I get a parse error at the marked position of the sysvar keyword in both cases. I successfully created functions that take a...
Read more

Krone von Aragonien

Image
Wappenschild der Krone von Aragonien Die Länder der Krone Aragon (hellblaue Gebiete: nur kurzzeitig oder indirekt unter aragonesischer Herrschaft) Herrschaftsgebiet zur Zeit Ramon Berenguers IV. Krone von Aragonien im 13. Jahrhundert Krone von Aragonien im 15. Jahrhundert Unter dem Begriff Krone von Aragonien ( spanisch Corona de Aragón , aragonesisch Corona d’Aragón , katalanisch Corona d’Aragó ) werden Herrschaftsgebiete unterschiedlicher Verfasstheit zusammengefasst, die zwischen 1137 und 1516 bzw. 1714 in Personalunion von den Königen von Aragonien regiert wurden. Dazu gehörten Königreiche wie Aragonien, Mallorca, Valencia, Sizilien, Sardinien, Korsika und Neapel, das Herzogtum Athen und Neopatria, die Markgrafschaft Provence, die Grafschaften Barcelona, Roussillon und Cerdanya und die Herrschaft Montpellier. [1] Die Herrscher der Krone von Aragonien und von Spanien zählten und zählen in ihrer Titulatur eine große Anzahl von Herrschaftsgebieten auf. Dies...
Read more