Is there a AWS service, to chunk and perform as concurrent, a bulk request to a Lambda?
up vote
1
down vote
favorite
This is the use case:
the client is requesting a lambda to compute a bulk payload.
The idea is to chunk this bulk payload and call multiple times the same lambda.
I've already done it on client side by chunking and recollecting answers, merging as if it was unique.
Now I'm wondering if I may leverage on some AWS service to perform this chunking, parallelisation to lambda and have it back as if a unique request.
On the other hand (if there's nothing out of the AWS box) I might think to develop a new lambda myself that will perform the above
amazon-web-services request aws-lambda
edited Nov 8 at 15:49
John Rotenstein
64k766110
asked Nov 8 at 9:20
koalaok
1,08621334
add a comment |
up vote
1
down vote
favorite
This is the use case:
the client is requesting a lambda to compute a bulk payload.
The idea is to chunk this bulk payload and call multiple times the same lambda.
I've already done it on client side by chunking and recollecting answers, merging as if it was unique.
Now I'm wondering if I may leverage on some AWS service to perform this chunking, parallelisation to lambda and have it back as if a unique request.
On the other hand (if there's nothing out of the AWS box) I might think to develop a new lambda myself that will perform the above
amazon-web-services request aws-lambda
edited Nov 8 at 15:49
John Rotenstein
64k766110
asked Nov 8 at 9:20
koalaok
1,08621334
Questions asking for off-site resources (such books/tutorials, tools/libraries) are off-topic here. Especially here: the services offered by a 3rd party provider ... can change all the time. The answers you are looking for will be some links, and as man people have said here before: answers that only constitute of a link are a clear indication that the question answering to ... is off topic here. But still, I can understand that you are asking :-)
– GhostCat
Nov 8 at 9:59
I see your points, And my question might be OT. But everything changes not only 3rd parties providers. On the second end... if this is OT on SO, what could be another option to ask about this within StackExchange?
– koalaok
Nov 8 at 10:16
There is softwarerecs.stackexchange.com ... but probably not a good fit either. You could turn to meta.stackexchange.com and ask there where to ask. Or on "our" meta: meta.stackoverflow.com
– GhostCat
Nov 8 at 10:18
add a comment |
up vote
1
down vote
favorite
up vote
1
down vote
favorite
This is the use case:
the client is requesting a lambda to compute a bulk payload.
The idea is to chunk this bulk payload and call multiple times the same lambda.
I've already done it on client side by chunking and recollecting answers, merging as if it was unique.
Now I'm wondering if I may leverage on some AWS service to perform this chunking, parallelisation to lambda and have it back as if a unique request.
On the other hand (if there's nothing out of the AWS box) I might think to develop a new lambda myself that will perform the above
amazon-web-services request aws-lambda
edited Nov 8 at 15:49
John Rotenstein
64k766110
asked Nov 8 at 9:20
koalaok
1,08621334
This is the use case:
the client is requesting a lambda to compute a bulk payload.
The idea is to chunk this bulk payload and call multiple times the same lambda.
I've already done it on client side by chunking and recollecting answers, merging as if it was unique.
Now I'm wondering if I may leverage on some AWS service to perform this chunking, parallelisation to lambda and have it back as if a unique request.
On the other hand (if there's nothing out of the AWS box) I might think to develop a new lambda myself that will perform the above
amazon-web-services request aws-lambda
amazon-web-services request aws-lambda
edited Nov 8 at 15:49
John Rotenstein
64k766110
asked Nov 8 at 9:20
koalaok
1,08621334
edited Nov 8 at 15:49
John Rotenstein
64k766110
asked Nov 8 at 9:20
koalaok
1,08621334
edited Nov 8 at 15:49
John Rotenstein
64k766110
edited Nov 8 at 15:49
John Rotenstein
64k766110
edited Nov 8 at 15:49
John Rotenstein
64k766110
64k766110
asked Nov 8 at 9:20
koalaok
1,08621334
asked Nov 8 at 9:20
koalaok
1,08621334
asked Nov 8 at 9:20
koalaok
1,08621334
1,08621334
Questions asking for off-site resources (such books/tutorials, tools/libraries) are off-topic here. Especially here: the services offered by a 3rd party provider ... can change all the time. The answers you are looking for will be some links, and as man people have said here before: answers that only constitute of a link are a clear indication that the question answering to ... is off topic here. But still, I can understand that you are asking :-)
– GhostCat
Nov 8 at 9:59
I see your points, And my question might be OT. But everything changes not only 3rd parties providers. On the second end... if this is OT on SO, what could be another option to ask about this within StackExchange?
– koalaok
Nov 8 at 10:16
There is softwarerecs.stackexchange.com ... but probably not a good fit either. You could turn to meta.stackexchange.com and ask there where to ask. Or on "our" meta: meta.stackoverflow.com
– GhostCat
Nov 8 at 10:18
add a comment |
Questions asking for off-site resources (such books/tutorials, tools/libraries) are off-topic here. Especially here: the services offered by a 3rd party provider ... can change all the time. The answers you are looking for will be some links, and as man people have said here before: answers that only constitute of a link are a clear indication that the question answering to ... is off topic here. But still, I can understand that you are asking :-)
– GhostCat
Nov 8 at 9:59
I see your points, And my question might be OT. But everything changes not only 3rd parties providers. On the second end... if this is OT on SO, what could be another option to ask about this within StackExchange?
– koalaok
Nov 8 at 10:16
There is softwarerecs.stackexchange.com ... but probably not a good fit either. You could turn to meta.stackexchange.com and ask there where to ask. Or on "our" meta: meta.stackoverflow.com
– GhostCat
Nov 8 at 10:18
Questions asking for off-site resources (such books/tutorials, tools/libraries) are off-topic here. Especially here: the services offered by a 3rd party provider ... can change all the time. The answers you are looking for will be some links, and as man people have said here before: answers that only constitute of a link are a clear indication that the question answering to ... is off topic here. But still, I can understand that you are asking :-)
– GhostCat
Nov 8 at 9:59
Questions asking for off-site resources (such books/tutorials, tools/libraries) are off-topic here. Especially here: the services offered by a 3rd party provider ... can change all the time. The answers you are looking for will be some links, and as man people have said here before: answers that only constitute of a link are a clear indication that the question answering to ... is off topic here. But still, I can understand that you are asking :-)
– GhostCat
Nov 8 at 9:59
I see your points, And my question might be OT. But everything changes not only 3rd parties providers. On the second end... if this is OT on SO, what could be another option to ask about this within StackExchange?
– koalaok
Nov 8 at 10:16
I see your points, And my question might be OT. But everything changes not only 3rd parties providers. On the second end... if this is OT on SO, what could be another option to ask about this within StackExchange?
– koalaok
Nov 8 at 10:16
There is softwarerecs.stackexchange.com ... but probably not a good fit either. You could turn to meta.stackexchange.com and ask there where to ask. Or on "our" meta: meta.stackoverflow.com
– GhostCat
Nov 8 at 10:18
There is softwarerecs.stackexchange.com ... but probably not a good fit either. You could turn to meta.stackexchange.com and ask there where to ask. Or on "our" meta: meta.stackoverflow.com
– GhostCat
Nov 8 at 10:18
add a comment |
1 Answer
1
active
oldest
votes
up vote
1
down vote
AWS Step functions is close to what you are looking for. You can write your own workflow of executing Lambda functions (sequentially or parallel). In your case you need to process each chunk in parallel. It will go as follows:
Note: I'm not aware of a way to dynamically run multiple Lambda functions in parallel using AWS Step Functions. If splitting the chunks into n parallel functions is not an option for you. I'd suggest you use SNS to trigger functions all in parallel and after each one finishes check if the others have saved their output to a store (S3/Dynamodb/Mysql ...etc). If yes, group all outputs and save their aggregation else do nothing. The only draw back to this is that you can't do this inside an HTTP lifecycle. The user will need to PULL for the status of the batch processing.
answered Nov 8 at 9:56
mostafazh
1,506718
Let's say I have a json payload array containing many childern. I'd like to chunk them and give each chunk to a lambda instance (parallel execution), I'm still not sure in your design where I can put the json chunking logic.... and then the logic of merging back all lambda executions response to provide the client a seemlessy execution. Does AWS Step functions provides all of this?
– koalaok
Nov 8 at 10:34
you sadly have to specify the number of parallel executions in the Step function and not at runtime during the execution of the first function. If you used, for example 10, parallel functions than your first function breaks the input on you will get 10x the performance of single function. Then at the end combine the output of all 10 functions. See the "Parallel State Output" section in the AWS Docs link I've provided above. It will also be seamless to the client. Would this still work?
– mostafazh
Nov 8 at 10:42
1
Not sure... I need to RTM about Step function and figure out. Anyway thanks for the hint by now.
– koalaok
Nov 8 at 10:44
add a comment |
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
up vote
1
down vote
AWS Step functions is close to what you are looking for. You can write your own workflow of executing Lambda functions (sequentially or parallel). In your case you need to process each chunk in parallel. It will go as follows:
Note: I'm not aware of a way to dynamically run multiple Lambda functions in parallel using AWS Step Functions. If splitting the chunks into n parallel functions is not an option for you. I'd suggest you use SNS to trigger functions all in parallel and after each one finishes check if the others have saved their output to a store (S3/Dynamodb/Mysql ...etc). If yes, group all outputs and save their aggregation else do nothing. The only draw back to this is that you can't do this inside an HTTP lifecycle. The user will need to PULL for the status of the batch processing.
answered Nov 8 at 9:56
mostafazh
1,506718
Let's say I have a json payload array containing many childern. I'd like to chunk them and give each chunk to a lambda instance (parallel execution), I'm still not sure in your design where I can put the json chunking logic.... and then the logic of merging back all lambda executions response to provide the client a seemlessy execution. Does AWS Step functions provides all of this?
– koalaok
Nov 8 at 10:34
you sadly have to specify the number of parallel executions in the Step function and not at runtime during the execution of the first function. If you used, for example 10, parallel functions than your first function breaks the input on you will get 10x the performance of single function. Then at the end combine the output of all 10 functions. See the "Parallel State Output" section in the AWS Docs link I've provided above. It will also be seamless to the client. Would this still work?
– mostafazh
Nov 8 at 10:42
1
Not sure... I need to RTM about Step function and figure out. Anyway thanks for the hint by now.
– koalaok
Nov 8 at 10:44
add a comment |
up vote
1
down vote
AWS Step functions is close to what you are looking for. You can write your own workflow of executing Lambda functions (sequentially or parallel). In your case you need to process each chunk in parallel. It will go as follows:
Note: I'm not aware of a way to dynamically run multiple Lambda functions in parallel using AWS Step Functions. If splitting the chunks into n parallel functions is not an option for you. I'd suggest you use SNS to trigger functions all in parallel and after each one finishes check if the others have saved their output to a store (S3/Dynamodb/Mysql ...etc). If yes, group all outputs and save their aggregation else do nothing. The only draw back to this is that you can't do this inside an HTTP lifecycle. The user will need to PULL for the status of the batch processing.
answered Nov 8 at 9:56
mostafazh
1,506718
Let's say I have a json payload array containing many childern. I'd like to chunk them and give each chunk to a lambda instance (parallel execution), I'm still not sure in your design where I can put the json chunking logic.... and then the logic of merging back all lambda executions response to provide the client a seemlessy execution. Does AWS Step functions provides all of this?
– koalaok
Nov 8 at 10:34
you sadly have to specify the number of parallel executions in the Step function and not at runtime during the execution of the first function. If you used, for example 10, parallel functions than your first function breaks the input on you will get 10x the performance of single function. Then at the end combine the output of all 10 functions. See the "Parallel State Output" section in the AWS Docs link I've provided above. It will also be seamless to the client. Would this still work?
– mostafazh
Nov 8 at 10:42
1
Not sure... I need to RTM about Step function and figure out. Anyway thanks for the hint by now.
– koalaok
Nov 8 at 10:44
add a comment |
up vote
1
down vote
up vote
1
down vote
AWS Step functions is close to what you are looking for. You can write your own workflow of executing Lambda functions (sequentially or parallel). In your case you need to process each chunk in parallel. It will go as follows:
Note: I'm not aware of a way to dynamically run multiple Lambda functions in parallel using AWS Step Functions. If splitting the chunks into n parallel functions is not an option for you. I'd suggest you use SNS to trigger functions all in parallel and after each one finishes check if the others have saved their output to a store (S3/Dynamodb/Mysql ...etc). If yes, group all outputs and save their aggregation else do nothing. The only draw back to this is that you can't do this inside an HTTP lifecycle. The user will need to PULL for the status of the batch processing.
answered Nov 8 at 9:56
mostafazh
1,506718
AWS Step functions is close to what you are looking for. You can write your own workflow of executing Lambda functions (sequentially or parallel). In your case you need to process each chunk in parallel. It will go as follows:
Note: I'm not aware of a way to dynamically run multiple Lambda functions in parallel using AWS Step Functions. If splitting the chunks into n parallel functions is not an option for you. I'd suggest you use SNS to trigger functions all in parallel and after each one finishes check if the others have saved their output to a store (S3/Dynamodb/Mysql ...etc). If yes, group all outputs and save their aggregation else do nothing. The only draw back to this is that you can't do this inside an HTTP lifecycle. The user will need to PULL for the status of the batch processing.
answered Nov 8 at 9:56
mostafazh
1,506718
edited Nov 8 at 10:37
answered Nov 8 at 9:56
mostafazh
1,506718
answered Nov 8 at 9:56
mostafazh
1,506718
answered Nov 8 at 9:56
mostafazh
1,506718
1,506718
Let's say I have a json payload array containing many childern. I'd like to chunk them and give each chunk to a lambda instance (parallel execution), I'm still not sure in your design where I can put the json chunking logic.... and then the logic of merging back all lambda executions response to provide the client a seemlessy execution. Does AWS Step functions provides all of this?
– koalaok
Nov 8 at 10:34
you sadly have to specify the number of parallel executions in the Step function and not at runtime during the execution of the first function. If you used, for example 10, parallel functions than your first function breaks the input on you will get 10x the performance of single function. Then at the end combine the output of all 10 functions. See the "Parallel State Output" section in the AWS Docs link I've provided above. It will also be seamless to the client. Would this still work?
– mostafazh
Nov 8 at 10:42
1
Not sure... I need to RTM about Step function and figure out. Anyway thanks for the hint by now.
– koalaok
Nov 8 at 10:44
add a comment |
Let's say I have a json payload array containing many childern. I'd like to chunk them and give each chunk to a lambda instance (parallel execution), I'm still not sure in your design where I can put the json chunking logic.... and then the logic of merging back all lambda executions response to provide the client a seemlessy execution. Does AWS Step functions provides all of this?
– koalaok
Nov 8 at 10:34
you sadly have to specify the number of parallel executions in the Step function and not at runtime during the execution of the first function. If you used, for example 10, parallel functions than your first function breaks the input on you will get 10x the performance of single function. Then at the end combine the output of all 10 functions. See the "Parallel State Output" section in the AWS Docs link I've provided above. It will also be seamless to the client. Would this still work?
– mostafazh
Nov 8 at 10:42
1
Not sure... I need to RTM about Step function and figure out. Anyway thanks for the hint by now.
– koalaok
Nov 8 at 10:44
Let's say I have a json payload array containing many childern. I'd like to chunk them and give each chunk to a lambda instance (parallel execution), I'm still not sure in your design where I can put the json chunking logic.... and then the logic of merging back all lambda executions response to provide the client a seemlessy execution. Does AWS Step functions provides all of this?
– koalaok
Nov 8 at 10:34
Let's say I have a json payload array containing many childern. I'd like to chunk them and give each chunk to a lambda instance (parallel execution), I'm still not sure in your design where I can put the json chunking logic.... and then the logic of merging back all lambda executions response to provide the client a seemlessy execution. Does AWS Step functions provides all of this?
– koalaok
Nov 8 at 10:34
you sadly have to specify the number of parallel executions in the Step function and not at runtime during the execution of the first function. If you used, for example 10, parallel functions than your first function breaks the input on you will get 10x the performance of single function. Then at the end combine the output of all 10 functions. See the "Parallel State Output" section in the AWS Docs link I've provided above. It will also be seamless to the client. Would this still work?
– mostafazh
Nov 8 at 10:42
you sadly have to specify the number of parallel executions in the Step function and not at runtime during the execution of the first function. If you used, for example 10, parallel functions than your first function breaks the input on you will get 10x the performance of single function. Then at the end combine the output of all 10 functions. See the "Parallel State Output" section in the AWS Docs link I've provided above. It will also be seamless to the client. Would this still work?
– mostafazh
Nov 8 at 10:42
1
1
Not sure... I need to RTM about Step function and figure out. Anyway thanks for the hint by now.
– koalaok
Nov 8 at 10:44
Not sure... I need to RTM about Step function and figure out. Anyway thanks for the hint by now.
– koalaok
Nov 8 at 10:44
add a comment |
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53204692%2fis-there-a-aws-service-to-chunk-and-perform-as-concurrent-a-bulk-request-to-a%23new-answer', 'question_page');
}
);
Post as a guest
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Questions asking for off-site resources (such books/tutorials, tools/libraries) are off-topic here. Especially here: the services offered by a 3rd party provider ... can change all the time. The answers you are looking for will be some links, and as man people have said here before: answers that only constitute of a link are a clear indication that the question answering to ... is off topic here. But still, I can understand that you are asking :-)
– GhostCat
Nov 8 at 9:59
I see your points, And my question might be OT. But everything changes not only 3rd parties providers. On the second end... if this is OT on SO, what could be another option to ask about this within StackExchange?
– koalaok
Nov 8 at 10:16
There is softwarerecs.stackexchange.com ... but probably not a good fit either. You could turn to meta.stackexchange.com and ask there where to ask. Or on "our" meta: meta.stackoverflow.com
– GhostCat
Nov 8 at 10:18