Mixing
AWS S3 charging for 4 TB of storage when only using less than 1 GB
Clash Royale CLAN TAG#URR8PPP
up vote
10
down vote
favorite
I'm having trouble understanding my large S3 bill, and figured I'd ask here before dropping $30 on AWS monthly support.
Basically, I have an Amazon EC2 instance that makes an API to different cryptocurrency exchanges and saves the responses to the instance HD. Calls are made about every 5 minutes, response objects are about 100Â kb, is read by an R script, and added to a CSV file every ~8Â minutes. That CSV file is synchronised to an Amazon S3 bucket about every 15Â minutes.
The CSV files are usually 10Â MB or so, for about 15 cryptocurrencies, every 15Â minutes. So looking in the Amazon S3 bucket, there might be 0.5 GB of space used at the most.
However, the 'TimedStorage-ByteHours' reads at about 4Â TB!
Amazon Simple Storage Service TimedStorage-ByteHrs $89.55
$0.000 per GB - storage under the monthly global free tier5 GB - Mo $0.00
$0.023 per GB - first 50 TB / month of storage used 3,893.399 GB - Mo $89.55
Any ideas?
amazon-web-services amazon-s3
edited 18 mins ago
Peter Mortensen
2,09742124
asked 12 hours ago
Garglesoap
563
add a comment |Â
up vote
10
down vote
favorite
I'm having trouble understanding my large S3 bill, and figured I'd ask here before dropping $30 on AWS monthly support.
Basically, I have an Amazon EC2 instance that makes an API to different cryptocurrency exchanges and saves the responses to the instance HD. Calls are made about every 5 minutes, response objects are about 100Â kb, is read by an R script, and added to a CSV file every ~8Â minutes. That CSV file is synchronised to an Amazon S3 bucket about every 15Â minutes.
The CSV files are usually 10Â MB or so, for about 15 cryptocurrencies, every 15Â minutes. So looking in the Amazon S3 bucket, there might be 0.5 GB of space used at the most.
However, the 'TimedStorage-ByteHours' reads at about 4Â TB!
Amazon Simple Storage Service TimedStorage-ByteHrs $89.55
$0.000 per GB - storage under the monthly global free tier5 GB - Mo $0.00
$0.023 per GB - first 50 TB / month of storage used 3,893.399 GB - Mo $89.55
Any ideas?
amazon-web-services amazon-s3
edited 18 mins ago
Peter Mortensen
2,09742124
asked 12 hours ago
Garglesoap
563
1
Do you have versioning turned on? If so, when you overwrite a file you may be leaving the old version behind.
– ceejayoz
12 hours ago
Ah the joys of AWS billing. Next time someone asks what it means to “nickel and dime†someone, I will send them here ;)
– jonatan
3 hours ago
add a comment |Â
up vote
10
down vote
favorite
up vote
10
down vote
favorite
I'm having trouble understanding my large S3 bill, and figured I'd ask here before dropping $30 on AWS monthly support.
Basically, I have an Amazon EC2 instance that makes an API to different cryptocurrency exchanges and saves the responses to the instance HD. Calls are made about every 5 minutes, response objects are about 100Â kb, is read by an R script, and added to a CSV file every ~8Â minutes. That CSV file is synchronised to an Amazon S3 bucket about every 15Â minutes.
The CSV files are usually 10Â MB or so, for about 15 cryptocurrencies, every 15Â minutes. So looking in the Amazon S3 bucket, there might be 0.5 GB of space used at the most.
However, the 'TimedStorage-ByteHours' reads at about 4Â TB!
Amazon Simple Storage Service TimedStorage-ByteHrs $89.55
$0.000 per GB - storage under the monthly global free tier5 GB - Mo $0.00
$0.023 per GB - first 50 TB / month of storage used 3,893.399 GB - Mo $89.55
Any ideas?
amazon-web-services amazon-s3
edited 18 mins ago
Peter Mortensen
2,09742124
asked 12 hours ago
Garglesoap
563
I'm having trouble understanding my large S3 bill, and figured I'd ask here before dropping $30 on AWS monthly support.
Basically, I have an Amazon EC2 instance that makes an API to different cryptocurrency exchanges and saves the responses to the instance HD. Calls are made about every 5 minutes, response objects are about 100Â kb, is read by an R script, and added to a CSV file every ~8Â minutes. That CSV file is synchronised to an Amazon S3 bucket about every 15Â minutes.
The CSV files are usually 10Â MB or so, for about 15 cryptocurrencies, every 15Â minutes. So looking in the Amazon S3 bucket, there might be 0.5 GB of space used at the most.
However, the 'TimedStorage-ByteHours' reads at about 4Â TB!
Amazon Simple Storage Service TimedStorage-ByteHrs $89.55
$0.000 per GB - storage under the monthly global free tier5 GB - Mo $0.00
$0.023 per GB - first 50 TB / month of storage used 3,893.399 GB - Mo $89.55
Any ideas?
amazon-web-services amazon-s3
amazon-web-services amazon-s3
edited 18 mins ago
Peter Mortensen
2,09742124
asked 12 hours ago
Garglesoap
563
edited 18 mins ago
Peter Mortensen
2,09742124
asked 12 hours ago
Garglesoap
563
edited 18 mins ago
Peter Mortensen
2,09742124
edited 18 mins ago
Peter Mortensen
2,09742124
edited 18 mins ago
Peter Mortensen
2,09742124
2,09742124
asked 12 hours ago
Garglesoap
563
asked 12 hours ago
Garglesoap
563
asked 12 hours ago
Garglesoap
563
563
1
Do you have versioning turned on? If so, when you overwrite a file you may be leaving the old version behind.
– ceejayoz
12 hours ago
Ah the joys of AWS billing. Next time someone asks what it means to “nickel and dime†someone, I will send them here ;)
– jonatan
3 hours ago
add a comment |Â
1
Do you have versioning turned on? If so, when you overwrite a file you may be leaving the old version behind.
– ceejayoz
12 hours ago
Ah the joys of AWS billing. Next time someone asks what it means to “nickel and dime†someone, I will send them here ;)
– jonatan
3 hours ago
1
1
Do you have versioning turned on? If so, when you overwrite a file you may be leaving the old version behind.
– ceejayoz
12 hours ago
Do you have versioning turned on? If so, when you overwrite a file you may be leaving the old version behind.
– ceejayoz
12 hours ago
Ah the joys of AWS billing. Next time someone asks what it means to “nickel and dime†someone, I will send them here ;)
– jonatan
3 hours ago
Ah the joys of AWS billing. Next time someone asks what it means to “nickel and dime†someone, I will send them here ;)
– jonatan
3 hours ago
add a comment |Â
1 Answer
1
active
oldest
votes
up vote
17
down vote
Most likely you've got S3 Versioning enabled - that means old objects when overwritten with a newer version don't get deleted but are instead hidden in a history. To verify go to the Bucket -> Properties -> Versioning.
You can also view the old versions in the browser, like on this screenshot I've got several versions of the 108c05...json file:
If you've got versioning enabled but don't want to you can Suspend versioning but be aware that it won't delete the old versions, you'll have to either:
- use AWS-CLI and some scripting (start with
aws s3api list-object-versions) - configure Bucket Lifecycle Policy to expire the old versions. That's done through S3 -> bucket -> Management -> Lifecycle -> Add lifecycle rule and then on the Expiration screen fill these details:
Hope that helps :)
answered 10 hours ago
MLu
3,2531331
add a comment |Â
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
up vote
17
down vote
Most likely you've got S3 Versioning enabled - that means old objects when overwritten with a newer version don't get deleted but are instead hidden in a history. To verify go to the Bucket -> Properties -> Versioning.
You can also view the old versions in the browser, like on this screenshot I've got several versions of the 108c05...json file:
If you've got versioning enabled but don't want to you can Suspend versioning but be aware that it won't delete the old versions, you'll have to either:
- use AWS-CLI and some scripting (start with
aws s3api list-object-versions) - configure Bucket Lifecycle Policy to expire the old versions. That's done through S3 -> bucket -> Management -> Lifecycle -> Add lifecycle rule and then on the Expiration screen fill these details:
Hope that helps :)
answered 10 hours ago
MLu
3,2531331
add a comment |Â
up vote
17
down vote
Most likely you've got S3 Versioning enabled - that means old objects when overwritten with a newer version don't get deleted but are instead hidden in a history. To verify go to the Bucket -> Properties -> Versioning.
You can also view the old versions in the browser, like on this screenshot I've got several versions of the 108c05...json file:
If you've got versioning enabled but don't want to you can Suspend versioning but be aware that it won't delete the old versions, you'll have to either:
- use AWS-CLI and some scripting (start with
aws s3api list-object-versions) - configure Bucket Lifecycle Policy to expire the old versions. That's done through S3 -> bucket -> Management -> Lifecycle -> Add lifecycle rule and then on the Expiration screen fill these details:
Hope that helps :)
answered 10 hours ago
MLu
3,2531331
add a comment |Â
up vote
17
down vote
up vote
17
down vote
Most likely you've got S3 Versioning enabled - that means old objects when overwritten with a newer version don't get deleted but are instead hidden in a history. To verify go to the Bucket -> Properties -> Versioning.
You can also view the old versions in the browser, like on this screenshot I've got several versions of the 108c05...json file:
If you've got versioning enabled but don't want to you can Suspend versioning but be aware that it won't delete the old versions, you'll have to either:
- use AWS-CLI and some scripting (start with
aws s3api list-object-versions) - configure Bucket Lifecycle Policy to expire the old versions. That's done through S3 -> bucket -> Management -> Lifecycle -> Add lifecycle rule and then on the Expiration screen fill these details:
Hope that helps :)
answered 10 hours ago
MLu
3,2531331
Most likely you've got S3 Versioning enabled - that means old objects when overwritten with a newer version don't get deleted but are instead hidden in a history. To verify go to the Bucket -> Properties -> Versioning.
You can also view the old versions in the browser, like on this screenshot I've got several versions of the 108c05...json file:
If you've got versioning enabled but don't want to you can Suspend versioning but be aware that it won't delete the old versions, you'll have to either:
- use AWS-CLI and some scripting (start with
aws s3api list-object-versions) - configure Bucket Lifecycle Policy to expire the old versions. That's done through S3 -> bucket -> Management -> Lifecycle -> Add lifecycle rule and then on the Expiration screen fill these details:
Hope that helps :)
answered 10 hours ago
MLu
3,2531331
answered 10 hours ago
MLu
3,2531331
answered 10 hours ago
MLu
3,2531331
answered 10 hours ago
MLu
3,2531331
3,2531331
add a comment |Â
add a comment |Â
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fserverfault.com%2fquestions%2f937468%2faws-s3-charging-for-4-tb-of-storage-when-only-using-less-than-1-gb%23new-answer', 'question_page');
);
Post as a guest
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
1
Do you have versioning turned on? If so, when you overwrite a file you may be leaving the old version behind.
– ceejayoz
12 hours ago
Ah the joys of AWS billing. Next time someone asks what it means to “nickel and dime†someone, I will send them here ;)
– jonatan
3 hours ago