Spring ThreadPoolTaskExecutor with fixed delay?
up vote
0
down vote
favorite
I have a Job which is scheduled to run every hour which uses Spring ThreadPoolTaskExecutor bean to fire simultaneous calls (close to 100 calls every hour) to external API.
@Bean
public TaskExecutor getExecutor() {
ThreadPoolTaskExecutor threadPoolTaskExecutor = new ThreadPoolTaskExecutor();
threadPoolTaskExecutor.setCorePoolSize(10);
threadPoolTaskExecutor.setMaxPoolSize(20);
return threadPoolTaskExecutor;
}
Now, the external API has throttled the number of requests and allows one request per 30 secs. I will have to wait 30 secs before making each call.
In this case, I see use of ThreadPoolTaskExecutor is no longer helpful. Will ThreadPoolTaskScheduler with Fixed Delay configuration work?
What is the best way to handle such type of API throttling? Please help
I'm on Java 8, Spring Boot 2.0.1 if that helps
java spring multithreading spring-boot threadpoolexecutor
asked Nov 8 at 9:53
jusermar10
733721
|
show 1 more comment
up vote
0
down vote
favorite
I have a Job which is scheduled to run every hour which uses Spring ThreadPoolTaskExecutor bean to fire simultaneous calls (close to 100 calls every hour) to external API.
@Bean
public TaskExecutor getExecutor() {
ThreadPoolTaskExecutor threadPoolTaskExecutor = new ThreadPoolTaskExecutor();
threadPoolTaskExecutor.setCorePoolSize(10);
threadPoolTaskExecutor.setMaxPoolSize(20);
return threadPoolTaskExecutor;
}
Now, the external API has throttled the number of requests and allows one request per 30 secs. I will have to wait 30 secs before making each call.
In this case, I see use of ThreadPoolTaskExecutor is no longer helpful. Will ThreadPoolTaskScheduler with Fixed Delay configuration work?
What is the best way to handle such type of API throttling? Please help
I'm on Java 8, Spring Boot 2.0.1 if that helps
java spring multithreading spring-boot threadpoolexecutor
asked Nov 8 at 9:53
jusermar10
733721
You are actually aware of the solution, aThreadPoolTaskSchedulerscheduler with a 30 seconds delayed trigger would work. Why wouldn't you just implement that? Are there any reasons?
– tmarwen
Nov 8 at 10:13
@tmarwen I have used@Scheduledwith a cron expression in the actual Job which runs every hour. Can I have a concurrent fixed delay schedule in ThreadPoolTaskScheduler ?
– jusermar10
Nov 8 at 10:19
@jusemar10 You might like to control the schedule on your @Scheduled annotation and leave theThreadPoolTaskScheduleralone.
– Arun Patra
Nov 8 at 12:54
1
I assume what you actually need is some kind of way to spread out the calls so it knows how much time has passed since the last one, and puts them on the queue if tasks need throttling. Although none of this helps if you don't change your app in a meaningfull way: you have to change it so it goes from 100 calls a minute to at most 2 calls. This isn't achievable with just throttling the executor.
– M. Prokhorov
Nov 8 at 13:26
... because if you just change the executor, and still load it with 100 tasks while it can process 2 at most, the tasks queue would get polluted very fast (98 a minute at least, to be precise).
– M. Prokhorov
Nov 8 at 13:28
|
show 1 more comment
up vote
0
down vote
favorite
up vote
0
down vote
favorite
I have a Job which is scheduled to run every hour which uses Spring ThreadPoolTaskExecutor bean to fire simultaneous calls (close to 100 calls every hour) to external API.
@Bean
public TaskExecutor getExecutor() {
ThreadPoolTaskExecutor threadPoolTaskExecutor = new ThreadPoolTaskExecutor();
threadPoolTaskExecutor.setCorePoolSize(10);
threadPoolTaskExecutor.setMaxPoolSize(20);
return threadPoolTaskExecutor;
}
Now, the external API has throttled the number of requests and allows one request per 30 secs. I will have to wait 30 secs before making each call.
In this case, I see use of ThreadPoolTaskExecutor is no longer helpful. Will ThreadPoolTaskScheduler with Fixed Delay configuration work?
What is the best way to handle such type of API throttling? Please help
I'm on Java 8, Spring Boot 2.0.1 if that helps
java spring multithreading spring-boot threadpoolexecutor
asked Nov 8 at 9:53
jusermar10
733721
I have a Job which is scheduled to run every hour which uses Spring ThreadPoolTaskExecutor bean to fire simultaneous calls (close to 100 calls every hour) to external API.
@Bean
public TaskExecutor getExecutor() {
ThreadPoolTaskExecutor threadPoolTaskExecutor = new ThreadPoolTaskExecutor();
threadPoolTaskExecutor.setCorePoolSize(10);
threadPoolTaskExecutor.setMaxPoolSize(20);
return threadPoolTaskExecutor;
}
Now, the external API has throttled the number of requests and allows one request per 30 secs. I will have to wait 30 secs before making each call.
In this case, I see use of ThreadPoolTaskExecutor is no longer helpful. Will ThreadPoolTaskScheduler with Fixed Delay configuration work?
What is the best way to handle such type of API throttling? Please help
I'm on Java 8, Spring Boot 2.0.1 if that helps
java spring multithreading spring-boot threadpoolexecutor
java spring multithreading spring-boot threadpoolexecutor
asked Nov 8 at 9:53
jusermar10
733721
asked Nov 8 at 9:53
jusermar10
733721
asked Nov 8 at 9:53
jusermar10
733721
asked Nov 8 at 9:53
jusermar10
733721
asked Nov 8 at 9:53
jusermar10
733721
733721
You are actually aware of the solution, aThreadPoolTaskSchedulerscheduler with a 30 seconds delayed trigger would work. Why wouldn't you just implement that? Are there any reasons?
– tmarwen
Nov 8 at 10:13
@tmarwen I have used@Scheduledwith a cron expression in the actual Job which runs every hour. Can I have a concurrent fixed delay schedule in ThreadPoolTaskScheduler ?
– jusermar10
Nov 8 at 10:19
@jusemar10 You might like to control the schedule on your @Scheduled annotation and leave theThreadPoolTaskScheduleralone.
– Arun Patra
Nov 8 at 12:54
1
I assume what you actually need is some kind of way to spread out the calls so it knows how much time has passed since the last one, and puts them on the queue if tasks need throttling. Although none of this helps if you don't change your app in a meaningfull way: you have to change it so it goes from 100 calls a minute to at most 2 calls. This isn't achievable with just throttling the executor.
– M. Prokhorov
Nov 8 at 13:26
... because if you just change the executor, and still load it with 100 tasks while it can process 2 at most, the tasks queue would get polluted very fast (98 a minute at least, to be precise).
– M. Prokhorov
Nov 8 at 13:28
|
show 1 more comment
You are actually aware of the solution, aThreadPoolTaskSchedulerscheduler with a 30 seconds delayed trigger would work. Why wouldn't you just implement that? Are there any reasons?
– tmarwen
Nov 8 at 10:13
@tmarwen I have used@Scheduledwith a cron expression in the actual Job which runs every hour. Can I have a concurrent fixed delay schedule in ThreadPoolTaskScheduler ?
– jusermar10
Nov 8 at 10:19
@jusemar10 You might like to control the schedule on your @Scheduled annotation and leave theThreadPoolTaskScheduleralone.
– Arun Patra
Nov 8 at 12:54
1
I assume what you actually need is some kind of way to spread out the calls so it knows how much time has passed since the last one, and puts them on the queue if tasks need throttling. Although none of this helps if you don't change your app in a meaningfull way: you have to change it so it goes from 100 calls a minute to at most 2 calls. This isn't achievable with just throttling the executor.
– M. Prokhorov
Nov 8 at 13:26
... because if you just change the executor, and still load it with 100 tasks while it can process 2 at most, the tasks queue would get polluted very fast (98 a minute at least, to be precise).
– M. Prokhorov
Nov 8 at 13:28
You are actually aware of the solution, a
ThreadPoolTaskScheduler scheduler with a 30 seconds delayed trigger would work. Why wouldn't you just implement that? Are there any reasons?– tmarwen
Nov 8 at 10:13
You are actually aware of the solution, a
ThreadPoolTaskScheduler scheduler with a 30 seconds delayed trigger would work. Why wouldn't you just implement that? Are there any reasons?– tmarwen
Nov 8 at 10:13
@tmarwen I have used
@Scheduled with a cron expression in the actual Job which runs every hour. Can I have a concurrent fixed delay schedule in ThreadPoolTaskScheduler ?– jusermar10
Nov 8 at 10:19
@tmarwen I have used
@Scheduled with a cron expression in the actual Job which runs every hour. Can I have a concurrent fixed delay schedule in ThreadPoolTaskScheduler ?– jusermar10
Nov 8 at 10:19
@jusemar10 You might like to control the schedule on your @Scheduled annotation and leave the
ThreadPoolTaskScheduler alone.– Arun Patra
Nov 8 at 12:54
@jusemar10 You might like to control the schedule on your @Scheduled annotation and leave the
ThreadPoolTaskScheduler alone.– Arun Patra
Nov 8 at 12:54
1
1
I assume what you actually need is some kind of way to spread out the calls so it knows how much time has passed since the last one, and puts them on the queue if tasks need throttling. Although none of this helps if you don't change your app in a meaningfull way: you have to change it so it goes from 100 calls a minute to at most 2 calls. This isn't achievable with just throttling the executor.
– M. Prokhorov
Nov 8 at 13:26
I assume what you actually need is some kind of way to spread out the calls so it knows how much time has passed since the last one, and puts them on the queue if tasks need throttling. Although none of this helps if you don't change your app in a meaningfull way: you have to change it so it goes from 100 calls a minute to at most 2 calls. This isn't achievable with just throttling the executor.
– M. Prokhorov
Nov 8 at 13:26
... because if you just change the executor, and still load it with 100 tasks while it can process 2 at most, the tasks queue would get polluted very fast (98 a minute at least, to be precise).
– M. Prokhorov
Nov 8 at 13:28
... because if you just change the executor, and still load it with 100 tasks while it can process 2 at most, the tasks queue would get polluted very fast (98 a minute at least, to be precise).
– M. Prokhorov
Nov 8 at 13:28
|
show 1 more comment
1 Answer
1
active
oldest
votes
up vote
0
down vote
You could do the following.
- Configure the
TaskExecutor. You have already done this.
Write a method that you want to invoke in a scheduled fashion and annotate with
@Scheduled.
@Scheduled(fixedDelay = 3600000)
void doSomething() {
}
The code above will trigger every 1 hour.
answered Nov 8 at 12:53
Arun Patra
43328
I have this setup already. My question is how to limit the API call which is currently fired about 100 times in parallel every time the job runs.
– jusermar10
Nov 8 at 13:13
This will always throttle the task, even if there were no tasks run for the past 10 minutes.
– M. Prokhorov
Nov 8 at 13:24
Agree with @M.Prokhorov . Your problem is not in the domain of the Scheduling infrastructure that Spring provides. You would probably have to build your own logic. Maybe your own custom logic (or reuse lower level of task management infrastructure from Java itself or Spring) to submit tasks to an intermediate processor. That intermediate processor would keep track of timestamps, and would coordinate a proper delay between consecutive submissions.
– Arun Patra
Nov 9 at 6:04
add a comment |
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
up vote
0
down vote
You could do the following.
- Configure the
TaskExecutor. You have already done this.
Write a method that you want to invoke in a scheduled fashion and annotate with
@Scheduled.
@Scheduled(fixedDelay = 3600000)
void doSomething() {
}
The code above will trigger every 1 hour.
answered Nov 8 at 12:53
Arun Patra
43328
I have this setup already. My question is how to limit the API call which is currently fired about 100 times in parallel every time the job runs.
– jusermar10
Nov 8 at 13:13
This will always throttle the task, even if there were no tasks run for the past 10 minutes.
– M. Prokhorov
Nov 8 at 13:24
Agree with @M.Prokhorov . Your problem is not in the domain of the Scheduling infrastructure that Spring provides. You would probably have to build your own logic. Maybe your own custom logic (or reuse lower level of task management infrastructure from Java itself or Spring) to submit tasks to an intermediate processor. That intermediate processor would keep track of timestamps, and would coordinate a proper delay between consecutive submissions.
– Arun Patra
Nov 9 at 6:04
add a comment |
up vote
0
down vote
You could do the following.
- Configure the
TaskExecutor. You have already done this.
Write a method that you want to invoke in a scheduled fashion and annotate with
@Scheduled.
@Scheduled(fixedDelay = 3600000)
void doSomething() {
}
The code above will trigger every 1 hour.
answered Nov 8 at 12:53
Arun Patra
43328
I have this setup already. My question is how to limit the API call which is currently fired about 100 times in parallel every time the job runs.
– jusermar10
Nov 8 at 13:13
This will always throttle the task, even if there were no tasks run for the past 10 minutes.
– M. Prokhorov
Nov 8 at 13:24
Agree with @M.Prokhorov . Your problem is not in the domain of the Scheduling infrastructure that Spring provides. You would probably have to build your own logic. Maybe your own custom logic (or reuse lower level of task management infrastructure from Java itself or Spring) to submit tasks to an intermediate processor. That intermediate processor would keep track of timestamps, and would coordinate a proper delay between consecutive submissions.
– Arun Patra
Nov 9 at 6:04
add a comment |
up vote
0
down vote
up vote
0
down vote
You could do the following.
- Configure the
TaskExecutor. You have already done this.
Write a method that you want to invoke in a scheduled fashion and annotate with
@Scheduled.
@Scheduled(fixedDelay = 3600000)
void doSomething() {
}
The code above will trigger every 1 hour.
answered Nov 8 at 12:53
Arun Patra
43328
You could do the following.
- Configure the
TaskExecutor. You have already done this.
Write a method that you want to invoke in a scheduled fashion and annotate with
@Scheduled.
@Scheduled(fixedDelay = 3600000)
void doSomething() {
}
The code above will trigger every 1 hour.
answered Nov 8 at 12:53
Arun Patra
43328
answered Nov 8 at 12:53
Arun Patra
43328
answered Nov 8 at 12:53
Arun Patra
43328
answered Nov 8 at 12:53
Arun Patra
43328
43328
I have this setup already. My question is how to limit the API call which is currently fired about 100 times in parallel every time the job runs.
– jusermar10
Nov 8 at 13:13
This will always throttle the task, even if there were no tasks run for the past 10 minutes.
– M. Prokhorov
Nov 8 at 13:24
Agree with @M.Prokhorov . Your problem is not in the domain of the Scheduling infrastructure that Spring provides. You would probably have to build your own logic. Maybe your own custom logic (or reuse lower level of task management infrastructure from Java itself or Spring) to submit tasks to an intermediate processor. That intermediate processor would keep track of timestamps, and would coordinate a proper delay between consecutive submissions.
– Arun Patra
Nov 9 at 6:04
add a comment |
I have this setup already. My question is how to limit the API call which is currently fired about 100 times in parallel every time the job runs.
– jusermar10
Nov 8 at 13:13
This will always throttle the task, even if there were no tasks run for the past 10 minutes.
– M. Prokhorov
Nov 8 at 13:24
Agree with @M.Prokhorov . Your problem is not in the domain of the Scheduling infrastructure that Spring provides. You would probably have to build your own logic. Maybe your own custom logic (or reuse lower level of task management infrastructure from Java itself or Spring) to submit tasks to an intermediate processor. That intermediate processor would keep track of timestamps, and would coordinate a proper delay between consecutive submissions.
– Arun Patra
Nov 9 at 6:04
I have this setup already. My question is how to limit the API call which is currently fired about 100 times in parallel every time the job runs.
– jusermar10
Nov 8 at 13:13
I have this setup already. My question is how to limit the API call which is currently fired about 100 times in parallel every time the job runs.
– jusermar10
Nov 8 at 13:13
This will always throttle the task, even if there were no tasks run for the past 10 minutes.
– M. Prokhorov
Nov 8 at 13:24
This will always throttle the task, even if there were no tasks run for the past 10 minutes.
– M. Prokhorov
Nov 8 at 13:24
Agree with @M.Prokhorov . Your problem is not in the domain of the Scheduling infrastructure that Spring provides. You would probably have to build your own logic. Maybe your own custom logic (or reuse lower level of task management infrastructure from Java itself or Spring) to submit tasks to an intermediate processor. That intermediate processor would keep track of timestamps, and would coordinate a proper delay between consecutive submissions.
– Arun Patra
Nov 9 at 6:04
Agree with @M.Prokhorov . Your problem is not in the domain of the Scheduling infrastructure that Spring provides. You would probably have to build your own logic. Maybe your own custom logic (or reuse lower level of task management infrastructure from Java itself or Spring) to submit tasks to an intermediate processor. That intermediate processor would keep track of timestamps, and would coordinate a proper delay between consecutive submissions.
– Arun Patra
Nov 9 at 6:04
add a comment |
draft saved
draft discarded
draft saved
draft discarded
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53205239%2fspring-threadpooltaskexecutor-with-fixed-delay%23new-answer', 'question_page');
}
);
Post as a guest
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
You are actually aware of the solution, a
ThreadPoolTaskSchedulerscheduler with a 30 seconds delayed trigger would work. Why wouldn't you just implement that? Are there any reasons?– tmarwen
Nov 8 at 10:13
@tmarwen I have used
@Scheduledwith a cron expression in the actual Job which runs every hour. Can I have a concurrent fixed delay schedule in ThreadPoolTaskScheduler ?– jusermar10
Nov 8 at 10:19
@jusemar10 You might like to control the schedule on your @Scheduled annotation and leave the
ThreadPoolTaskScheduleralone.– Arun Patra
Nov 8 at 12:54
1
I assume what you actually need is some kind of way to spread out the calls so it knows how much time has passed since the last one, and puts them on the queue if tasks need throttling. Although none of this helps if you don't change your app in a meaningfull way: you have to change it so it goes from 100 calls a minute to at most 2 calls. This isn't achievable with just throttling the executor.
– M. Prokhorov
Nov 8 at 13:26
... because if you just change the executor, and still load it with 100 tasks while it can process 2 at most, the tasks queue would get polluted very fast (98 a minute at least, to be precise).
– M. Prokhorov
Nov 8 at 13:28