How to limit number of threads for python interpreter?











up vote
2
down vote

favorite
1












I want to know whether it is possible to limit the number of threads python interpreter can spawn when running a script.



I am not using threading myself, but my script uses modules like pandas, numpy, and others which I assume would be using threading in the background. The script works fine, but when I run it on an LSF cluster, it spawns too many threads crossing thread limit per job (for the cluster) and as a result, crashes.



I would like to restrict the python interpreter to not allow more than, say, 100, threads at a time. Is it possible to run a python script on an LSF server with such a restriction on thread count?



Thanks for your help.



P.S.: My colleague faces the same issue with java apps, so a solution for that would be cool as well.






java python multithreading hpc lsf







share|improve this question






















  • Are you using vanilla python to run scripts? Are you aware of the Python GIL ? Also, see stackoverflow.com/questions/30791550/…
    – RottenCandy
    Nov 8 at 10:30

















up vote
2
down vote

favorite
1












I want to know whether it is possible to limit the number of threads python interpreter can spawn when running a script.



I am not using threading myself, but my script uses modules like pandas, numpy, and others which I assume would be using threading in the background. The script works fine, but when I run it on an LSF cluster, it spawns too many threads crossing thread limit per job (for the cluster) and as a result, crashes.



I would like to restrict the python interpreter to not allow more than, say, 100, threads at a time. Is it possible to run a python script on an LSF server with such a restriction on thread count?



Thanks for your help.



P.S.: My colleague faces the same issue with java apps, so a solution for that would be cool as well.






java python multithreading hpc lsf







share|improve this question






















  • Are you using vanilla python to run scripts? Are you aware of the Python GIL ? Also, see stackoverflow.com/questions/30791550/…
    – RottenCandy
    Nov 8 at 10:30















up vote
2
down vote

favorite
1









up vote
2
down vote

favorite
1






1





I want to know whether it is possible to limit the number of threads python interpreter can spawn when running a script.



I am not using threading myself, but my script uses modules like pandas, numpy, and others which I assume would be using threading in the background. The script works fine, but when I run it on an LSF cluster, it spawns too many threads crossing thread limit per job (for the cluster) and as a result, crashes.



I would like to restrict the python interpreter to not allow more than, say, 100, threads at a time. Is it possible to run a python script on an LSF server with such a restriction on thread count?



Thanks for your help.



P.S.: My colleague faces the same issue with java apps, so a solution for that would be cool as well.






java python multithreading hpc lsf







share|improve this question













I want to know whether it is possible to limit the number of threads python interpreter can spawn when running a script.



I am not using threading myself, but my script uses modules like pandas, numpy, and others which I assume would be using threading in the background. The script works fine, but when I run it on an LSF cluster, it spawns too many threads crossing thread limit per job (for the cluster) and as a result, crashes.



I would like to restrict the python interpreter to not allow more than, say, 100, threads at a time. Is it possible to run a python script on an LSF server with such a restriction on thread count?



Thanks for your help.



P.S.: My colleague faces the same issue with java apps, so a solution for that would be cool as well.






java python multithreading hpc lsf




java python multithreading hpc lsf






share|improve this question













share|improve this question











share|improve this question




share|improve this question










asked Nov 8 at 10:03









Manish Goel

341213




341213












  • Are you using vanilla python to run scripts? Are you aware of the Python GIL ? Also, see stackoverflow.com/questions/30791550/…
    – RottenCandy
    Nov 8 at 10:30




















  • Are you using vanilla python to run scripts? Are you aware of the Python GIL ? Also, see stackoverflow.com/questions/30791550/…
    – RottenCandy
    Nov 8 at 10:30


















Are you using vanilla python to run scripts? Are you aware of the Python GIL ? Also, see stackoverflow.com/questions/30791550/…
– RottenCandy
Nov 8 at 10:30






Are you using vanilla python to run scripts? Are you aware of the Python GIL ? Also, see stackoverflow.com/questions/30791550/…
– RottenCandy
Nov 8 at 10:30



















active

oldest

votes











Your Answer






StackExchange.ifUsing("editor", function () {
StackExchange.using("externalEditor", function () {
StackExchange.using("snippets", function () {
StackExchange.snippets.init();
});
});
}, "code-snippets");

StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "1"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);

StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});

function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});


}
});














 

draft saved


draft discarded


















StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53205405%2fhow-to-limit-number-of-threads-for-python-interpreter%23new-answer', 'question_page');
}
);

Post as a guest























active

oldest

votes













active

oldest

votes









active

oldest

votes






active

oldest

votes
















 

draft saved


draft discarded



















































 


draft saved


draft discarded














StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53205405%2fhow-to-limit-number-of-threads-for-python-interpreter%23new-answer', 'question_page');
}
);

Post as a guest






































































-

Popular posts from this blog

Schultheiß

Image
Der Titel dieses Artikels ist mehrdeutig. Weitere Bedeutungen sind unter Schultheiß (Begriffsklärung) aufgeführt. Ein Schultheiß. Holzschnitt von Peter Flötner (16. Jahrhundert) Der Schultheiß oder Schuldheiß (von althochdeutsch  sculdheizo ‚Leistung Befehlender‘, vgl. mittelniederdeutsch  schult(h)ēte , latinisiert (mittellat.) sculte(t)us , schwäbisch heute noch Schultes für „Bürgermeister“) bezeichnet einen in vielen westgermanischen Rechtsordnungen vorgesehenen Beamten, der Schuld heischt : Er hatte im Auftrag seines Herren (Landesherrn, Stadtherrn, Grundherrn) die Mitglieder einer Gemeinde zur Leistung ihrer Schuldigkeit anzuhalten, also Abgaben einzuziehen oder für das Beachten anderer Verpflichtungen Sorge zu tragen. Sprachliche Varianten des Schultheißen sind Schulte , Schultes oder Schulze . Früher wurde zwischen dem Stadtschulzen und dem Dorfschulzen unterschieden. In der städtischen Gerichts- und Gemeindeverfassung war er ein vom städtischen Rat oder vom L
Read more

Liste der Kulturdenkmale in Wilsdruff

Image
Wappen von Wilsdruff Die Liste der Kulturdenkmale in Wilsdruff enthält die Kulturdenkmale in Wilsdruff. Die Anmerkungen sind zu beachten. Diese Liste ist eine Teilliste der Liste der Kulturdenkmale im Landkreis Sächsische Schweiz-Osterzgebirge. Diese Liste ist eine Teilliste der Liste der Kulturdenkmale in Sachsen. Inhaltsverzeichnis 1 Legende 2 Wilsdruff 3 Birkenhain 4 Blankenstein 5 Braunsdorf 6 Grumbach 7 Grund 8 Helbigsdorf 9 Herzogswalde 10 Kaufbach 11 Kesselsdorf 12 Kleinopitz 13 Limbach 14 Mohorn 15 Oberhermsdorf 16 Anmerkungen 17 Ausführliche Denkmaltexte 18 Quellen 19 Weblinks Legende | Bild: zeigt ein Bild des Kulturdenkmals und gegebenenfalls einen Link zu weiteren Fotos des Kulturdenkmals Bezeichnung: Name, Bezeichnung oder die Art des Kulturdenkmals Lage: Straßenname und wenn vorhanden Hausnummer des Kulturdenkmals; Grundsortierung der Liste erfolgt nach dieser A
Read more

Android Play Services Check

Image
up vote 0 down vote favorite Im facing a problem currently im making my app to run on t.v its working fine on mobiles. but the issue is if i run my app on t.v emulator splash screen appears then the app crashes log shows this error. No acceptable module found. Local version is 0 and remote version is 0 i googled this error and came with a solution that my emulator should have latest google play services installed in it.. but in my t.v emulator there is no play store so i cant update my play services to latest version. so is there any way to make a check that if play services version is less then this so run the app instead of crashing, and if the play services version is latest also run the aap... here is my code where im getting the play services version.. int playServicesAvailable = GoogleApiAvailability.g
Read more