Mixing
AWS S3 charging for 4TB of storage when only <1GB
Clash Royale CLAN TAG#URR8PPP
up vote
2
down vote
favorite
I'm having trouble understanding my large S3 bill, figured I'd ask here before dropping $30 on AWS monethly support. Basically, I have an ec2 instance that makes API to different cryptocurrency exchanges and saves responses to the instance HD. Calls are made about every 5mins, response objects are about 100kb, is read by an R script and added to a .csv file every ~8mins. That csv file is synced to an S3 bucket about every 15mins.
The csv files are usually 10mb or so, for about 15 cryptocurrencies, every 15mins. So looking in the S3 bucket, there might be 0.5 GB of space used at the most.
However, the 'TimedStorage-ByteHours' reads at about 4 TB!
Amazon Simple Storage Service TimedStorage-ByteHrs $89.55
$0.000 per GB - storage under the monthly global free tier5 GB-Mo$0.00
$0.023 per GB - first 50 TB / month of storage used3,893.399 GB-Mo$89.55
Any ideas?
amazon-web-services amazon-s3
asked 3 hours ago
Garglesoap
162
add a comment |Â
up vote
2
down vote
favorite
I'm having trouble understanding my large S3 bill, figured I'd ask here before dropping $30 on AWS monethly support. Basically, I have an ec2 instance that makes API to different cryptocurrency exchanges and saves responses to the instance HD. Calls are made about every 5mins, response objects are about 100kb, is read by an R script and added to a .csv file every ~8mins. That csv file is synced to an S3 bucket about every 15mins.
The csv files are usually 10mb or so, for about 15 cryptocurrencies, every 15mins. So looking in the S3 bucket, there might be 0.5 GB of space used at the most.
However, the 'TimedStorage-ByteHours' reads at about 4 TB!
Amazon Simple Storage Service TimedStorage-ByteHrs $89.55
$0.000 per GB - storage under the monthly global free tier5 GB-Mo$0.00
$0.023 per GB - first 50 TB / month of storage used3,893.399 GB-Mo$89.55
Any ideas?
amazon-web-services amazon-s3
asked 3 hours ago
Garglesoap
162
Do you have versioning turned on? If so, when you overwrite a file you may be leaving the old version behind.
– ceejayoz
3 hours ago
add a comment |Â
up vote
2
down vote
favorite
up vote
2
down vote
favorite
I'm having trouble understanding my large S3 bill, figured I'd ask here before dropping $30 on AWS monethly support. Basically, I have an ec2 instance that makes API to different cryptocurrency exchanges and saves responses to the instance HD. Calls are made about every 5mins, response objects are about 100kb, is read by an R script and added to a .csv file every ~8mins. That csv file is synced to an S3 bucket about every 15mins.
The csv files are usually 10mb or so, for about 15 cryptocurrencies, every 15mins. So looking in the S3 bucket, there might be 0.5 GB of space used at the most.
However, the 'TimedStorage-ByteHours' reads at about 4 TB!
Amazon Simple Storage Service TimedStorage-ByteHrs $89.55
$0.000 per GB - storage under the monthly global free tier5 GB-Mo$0.00
$0.023 per GB - first 50 TB / month of storage used3,893.399 GB-Mo$89.55
Any ideas?
amazon-web-services amazon-s3
asked 3 hours ago
Garglesoap
162
I'm having trouble understanding my large S3 bill, figured I'd ask here before dropping $30 on AWS monethly support. Basically, I have an ec2 instance that makes API to different cryptocurrency exchanges and saves responses to the instance HD. Calls are made about every 5mins, response objects are about 100kb, is read by an R script and added to a .csv file every ~8mins. That csv file is synced to an S3 bucket about every 15mins.
The csv files are usually 10mb or so, for about 15 cryptocurrencies, every 15mins. So looking in the S3 bucket, there might be 0.5 GB of space used at the most.
However, the 'TimedStorage-ByteHours' reads at about 4 TB!
Amazon Simple Storage Service TimedStorage-ByteHrs $89.55
$0.000 per GB - storage under the monthly global free tier5 GB-Mo$0.00
$0.023 per GB - first 50 TB / month of storage used3,893.399 GB-Mo$89.55
Any ideas?
amazon-web-services amazon-s3
amazon-web-services amazon-s3
asked 3 hours ago
Garglesoap
162
asked 3 hours ago
Garglesoap
162
asked 3 hours ago
Garglesoap
162
asked 3 hours ago
Garglesoap
162
asked 3 hours ago
Garglesoap
162
162
Do you have versioning turned on? If so, when you overwrite a file you may be leaving the old version behind.
– ceejayoz
3 hours ago
add a comment |Â
Do you have versioning turned on? If so, when you overwrite a file you may be leaving the old version behind.
– ceejayoz
3 hours ago
Do you have versioning turned on? If so, when you overwrite a file you may be leaving the old version behind.
– ceejayoz
3 hours ago
Do you have versioning turned on? If so, when you overwrite a file you may be leaving the old version behind.
– ceejayoz
3 hours ago
add a comment |Â
1 Answer
1
active
oldest
votes
up vote
2
down vote
Most likely you've got S3 Versioning enabled - that means old objects when overwritten with a newer version don't get deleted but are instead hidden in a history. To verify go to the Bucket -> Properties -> Versioning.
You can also view the old versions in the browser, like on this screenshot I've got several versions of the 108c05...json file:
If you've got versioning enabled but don't want to you can Suspend versioning but be aware that it won't delete the old versions, you'll have to either:
- use AWS-CLI and some scripting (start with
aws s3api list-object-versions) - configure Bucket Lifecycle Policy to expire the old versions. That's done through S3 -> bucket -> Management -> Lifecycle -> Add lifecycle rule and then on the Expiration screen fill these details:
Hope that helps :)
answered 2 hours ago
MLu
3,1031329
add a comment |Â
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
up vote
2
down vote
Most likely you've got S3 Versioning enabled - that means old objects when overwritten with a newer version don't get deleted but are instead hidden in a history. To verify go to the Bucket -> Properties -> Versioning.
You can also view the old versions in the browser, like on this screenshot I've got several versions of the 108c05...json file:
If you've got versioning enabled but don't want to you can Suspend versioning but be aware that it won't delete the old versions, you'll have to either:
- use AWS-CLI and some scripting (start with
aws s3api list-object-versions) - configure Bucket Lifecycle Policy to expire the old versions. That's done through S3 -> bucket -> Management -> Lifecycle -> Add lifecycle rule and then on the Expiration screen fill these details:
Hope that helps :)
answered 2 hours ago
MLu
3,1031329
add a comment |Â
up vote
2
down vote
Most likely you've got S3 Versioning enabled - that means old objects when overwritten with a newer version don't get deleted but are instead hidden in a history. To verify go to the Bucket -> Properties -> Versioning.
You can also view the old versions in the browser, like on this screenshot I've got several versions of the 108c05...json file:
If you've got versioning enabled but don't want to you can Suspend versioning but be aware that it won't delete the old versions, you'll have to either:
- use AWS-CLI and some scripting (start with
aws s3api list-object-versions) - configure Bucket Lifecycle Policy to expire the old versions. That's done through S3 -> bucket -> Management -> Lifecycle -> Add lifecycle rule and then on the Expiration screen fill these details:
Hope that helps :)
answered 2 hours ago
MLu
3,1031329
add a comment |Â
up vote
2
down vote
up vote
2
down vote
Most likely you've got S3 Versioning enabled - that means old objects when overwritten with a newer version don't get deleted but are instead hidden in a history. To verify go to the Bucket -> Properties -> Versioning.
You can also view the old versions in the browser, like on this screenshot I've got several versions of the 108c05...json file:
If you've got versioning enabled but don't want to you can Suspend versioning but be aware that it won't delete the old versions, you'll have to either:
- use AWS-CLI and some scripting (start with
aws s3api list-object-versions) - configure Bucket Lifecycle Policy to expire the old versions. That's done through S3 -> bucket -> Management -> Lifecycle -> Add lifecycle rule and then on the Expiration screen fill these details:
Hope that helps :)
answered 2 hours ago
MLu
3,1031329
Most likely you've got S3 Versioning enabled - that means old objects when overwritten with a newer version don't get deleted but are instead hidden in a history. To verify go to the Bucket -> Properties -> Versioning.
You can also view the old versions in the browser, like on this screenshot I've got several versions of the 108c05...json file:
If you've got versioning enabled but don't want to you can Suspend versioning but be aware that it won't delete the old versions, you'll have to either:
- use AWS-CLI and some scripting (start with
aws s3api list-object-versions) - configure Bucket Lifecycle Policy to expire the old versions. That's done through S3 -> bucket -> Management -> Lifecycle -> Add lifecycle rule and then on the Expiration screen fill these details:
Hope that helps :)
answered 2 hours ago
MLu
3,1031329
answered 2 hours ago
MLu
3,1031329
answered 2 hours ago
MLu
3,1031329
answered 2 hours ago
MLu
3,1031329
3,1031329
add a comment |Â
add a comment |Â
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fserverfault.com%2fquestions%2f937468%2faws-s3-charging-for-4tb-of-storage-when-only-1gb%23new-answer', 'question_page');
);
Post as a guest
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Do you have versioning turned on? If so, when you overwrite a file you may be leaving the old version behind.
– ceejayoz
3 hours ago