Batch load from S3 - fatal error: An error occurred (403) when calling the HeadObject operation: Forbidden
up vote
0
down vote
favorite
Apologies if I am not explaining this correctly as I am new to AWS. I have a file in an S3 bucket that has brackets in the name. We are using S3, Lambda and Batch to transfer the data over to a SQL database.
If I manually set Batch to load the file, it works okay. If I set up Lambda to see the file when it drops into S3 then it seems to see the file with % signs instead of the brackets. If I amend the upload handler to accept % in the name, then the batch file fails with:
fatal error: An error occurred (403) when calling the HeadObject operation: Forbidden
As I know we can connect okay to the S3 bucket, I am assuming this is not permissions but may be the fact that the name that Lambda sees does not actually exist (i.e. TestLoad_(file1).csv comes across as TestLoad_%28file1%29.csv).
Has anyone come across this and how would I fix it (other than manually rename the file every time it drops into S3).
amazon-web-services amazon-s3
asked Nov 8 at 13:34
Caz
11
add a comment |
up vote
0
down vote
favorite
Apologies if I am not explaining this correctly as I am new to AWS. I have a file in an S3 bucket that has brackets in the name. We are using S3, Lambda and Batch to transfer the data over to a SQL database.
If I manually set Batch to load the file, it works okay. If I set up Lambda to see the file when it drops into S3 then it seems to see the file with % signs instead of the brackets. If I amend the upload handler to accept % in the name, then the batch file fails with:
fatal error: An error occurred (403) when calling the HeadObject operation: Forbidden
As I know we can connect okay to the S3 bucket, I am assuming this is not permissions but may be the fact that the name that Lambda sees does not actually exist (i.e. TestLoad_(file1).csv comes across as TestLoad_%28file1%29.csv).
Has anyone come across this and how would I fix it (other than manually rename the file every time it drops into S3).
amazon-web-services amazon-s3
asked Nov 8 at 13:34
Caz
11
What language is your Lambda function written in?
– Michael - sqlbot
Nov 8 at 17:29
Hi Michael - it's in Python
– Caz
Nov 9 at 7:30
S3 event notifications contain the object key in S3's somewhat strange internal representation -- essentially query-string encoded, presumably an artifact of a very old and unfortunate design decision that it's too late to fix without breaking different things. You have to decode it before using it, including using it to do other things with S3. Check stackoverflow.com/a/39465221/1695906
– Michael - sqlbot
Nov 9 at 12:32
add a comment |
up vote
0
down vote
favorite
up vote
0
down vote
favorite
Apologies if I am not explaining this correctly as I am new to AWS. I have a file in an S3 bucket that has brackets in the name. We are using S3, Lambda and Batch to transfer the data over to a SQL database.
If I manually set Batch to load the file, it works okay. If I set up Lambda to see the file when it drops into S3 then it seems to see the file with % signs instead of the brackets. If I amend the upload handler to accept % in the name, then the batch file fails with:
fatal error: An error occurred (403) when calling the HeadObject operation: Forbidden
As I know we can connect okay to the S3 bucket, I am assuming this is not permissions but may be the fact that the name that Lambda sees does not actually exist (i.e. TestLoad_(file1).csv comes across as TestLoad_%28file1%29.csv).
Has anyone come across this and how would I fix it (other than manually rename the file every time it drops into S3).
amazon-web-services amazon-s3
asked Nov 8 at 13:34
Caz
11
Apologies if I am not explaining this correctly as I am new to AWS. I have a file in an S3 bucket that has brackets in the name. We are using S3, Lambda and Batch to transfer the data over to a SQL database.
If I manually set Batch to load the file, it works okay. If I set up Lambda to see the file when it drops into S3 then it seems to see the file with % signs instead of the brackets. If I amend the upload handler to accept % in the name, then the batch file fails with:
fatal error: An error occurred (403) when calling the HeadObject operation: Forbidden
As I know we can connect okay to the S3 bucket, I am assuming this is not permissions but may be the fact that the name that Lambda sees does not actually exist (i.e. TestLoad_(file1).csv comes across as TestLoad_%28file1%29.csv).
Has anyone come across this and how would I fix it (other than manually rename the file every time it drops into S3).
amazon-web-services amazon-s3
amazon-web-services amazon-s3
asked Nov 8 at 13:34
Caz
11
asked Nov 8 at 13:34
Caz
11
asked Nov 8 at 13:34
Caz
11
asked Nov 8 at 13:34
Caz
11
asked Nov 8 at 13:34
Caz
11
11
What language is your Lambda function written in?
– Michael - sqlbot
Nov 8 at 17:29
Hi Michael - it's in Python
– Caz
Nov 9 at 7:30
S3 event notifications contain the object key in S3's somewhat strange internal representation -- essentially query-string encoded, presumably an artifact of a very old and unfortunate design decision that it's too late to fix without breaking different things. You have to decode it before using it, including using it to do other things with S3. Check stackoverflow.com/a/39465221/1695906
– Michael - sqlbot
Nov 9 at 12:32
add a comment |
What language is your Lambda function written in?
– Michael - sqlbot
Nov 8 at 17:29
Hi Michael - it's in Python
– Caz
Nov 9 at 7:30
S3 event notifications contain the object key in S3's somewhat strange internal representation -- essentially query-string encoded, presumably an artifact of a very old and unfortunate design decision that it's too late to fix without breaking different things. You have to decode it before using it, including using it to do other things with S3. Check stackoverflow.com/a/39465221/1695906
– Michael - sqlbot
Nov 9 at 12:32
What language is your Lambda function written in?
– Michael - sqlbot
Nov 8 at 17:29
What language is your Lambda function written in?
– Michael - sqlbot
Nov 8 at 17:29
Hi Michael - it's in Python
– Caz
Nov 9 at 7:30
Hi Michael - it's in Python
– Caz
Nov 9 at 7:30
S3 event notifications contain the object key in S3's somewhat strange internal representation -- essentially query-string encoded, presumably an artifact of a very old and unfortunate design decision that it's too late to fix without breaking different things. You have to decode it before using it, including using it to do other things with S3. Check stackoverflow.com/a/39465221/1695906
– Michael - sqlbot
Nov 9 at 12:32
S3 event notifications contain the object key in S3's somewhat strange internal representation -- essentially query-string encoded, presumably an artifact of a very old and unfortunate design decision that it's too late to fix without breaking different things. You have to decode it before using it, including using it to do other things with S3. Check stackoverflow.com/a/39465221/1695906
– Michael - sqlbot
Nov 9 at 12:32
add a comment |
active
oldest
votes
active
oldest
votes
active
oldest
votes
active
oldest
votes
active
oldest
votes
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53208837%2fbatch-load-from-s3-fatal-error-an-error-occurred-403-when-calling-the-heado%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
What language is your Lambda function written in?
– Michael - sqlbot
Nov 8 at 17:29
Hi Michael - it's in Python
– Caz
Nov 9 at 7:30
S3 event notifications contain the object key in S3's somewhat strange internal representation -- essentially query-string encoded, presumably an artifact of a very old and unfortunate design decision that it's too late to fix without breaking different things. You have to decode it before using it, including using it to do other things with S3. Check stackoverflow.com/a/39465221/1695906
– Michael - sqlbot
Nov 9 at 12:32