Why does piping `tar` into `dd` not stop until the disk is full?

up vote
2
down vote

favorite

I have a tar archive of a single disk image. The image inside this tar file is about 4GB in size. I pipe the output of tar xf into dd to write the disk image to an SD card. The diskdump never stops until the card is full. Here is my shell session:

$ ls -l disk.img.tgz
-rw-r--r-- 1 confus confus 192M Okt 5 00:53

$ tar -tvf disk.img.tgz
-rw-r--r-- root/root 4294968320 2018-10-05 00:52 disk.img

$ lsblk -lb /dev/sdc
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sdc 8:32 1 16022241280 0 disk

$ tar zxf disk.img.tgz -O | sudo dd status=progress conv=sync bs=1M of=/dev/sdc
[sudo] password for user: 
15992881152 bytes (16 GB, 15 GiB) copied, 212 s, 75,4 MB/s 
dd: error writing '/dev/sdc': No space left on device
0+15281 records in
15280+0 records out
16022241280 bytes (16 GB, 15 GiB) copied, 217,67 s, 73,6 MB/s

Why? It should stop after hit has written the 4GB image to the 16GB cart and never run out of space!

edited 54 mins ago

asked 58 mins ago

con-f-use

18318

Do you have the disk space to try running this through dd and writing it to another file? tar zxf disk.img.tgz -O | dd status=progress conv=sync bs=1M of=/path/to/some/file/on/disk ? If so, does that get you an exact copy of the original file?
â€“Â Andy Dalton
48 mins ago

Why do you have conv=sync? Did you mean to use conv=fsync perhaps?
â€“Â Ralph RÃ¶nnquist
38 mins ago

add a commentÂ |Â

up vote
2
down vote

favorite

$ ls -l disk.img.tgz
-rw-r--r-- 1 confus confus 192M Okt 5 00:53

$ tar -tvf disk.img.tgz
-rw-r--r-- root/root 4294968320 2018-10-05 00:52 disk.img

$ lsblk -lb /dev/sdc
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sdc 8:32 1 16022241280 0 disk

$ tar zxf disk.img.tgz -O | sudo dd status=progress conv=sync bs=1M of=/dev/sdc
[sudo] password for user: 
15992881152 bytes (16 GB, 15 GiB) copied, 212 s, 75,4 MB/s 
dd: error writing '/dev/sdc': No space left on device
0+15281 records in
15280+0 records out
16022241280 bytes (16 GB, 15 GiB) copied, 217,67 s, 73,6 MB/s

Why? It should stop after hit has written the 4GB image to the 16GB cart and never run out of space!

edited 54 mins ago

asked 58 mins ago

con-f-use

18318

Do you have the disk space to try running this through dd and writing it to another file? tar zxf disk.img.tgz -O | dd status=progress conv=sync bs=1M of=/path/to/some/file/on/disk ? If so, does that get you an exact copy of the original file?
â€“Â Andy Dalton
48 mins ago

Why do you have conv=sync? Did you mean to use conv=fsync perhaps?
â€“Â Ralph RÃ¶nnquist
38 mins ago

add a commentÂ |Â

up vote
2
down vote

favorite

$ ls -l disk.img.tgz
-rw-r--r-- 1 confus confus 192M Okt 5 00:53

$ tar -tvf disk.img.tgz
-rw-r--r-- root/root 4294968320 2018-10-05 00:52 disk.img

$ lsblk -lb /dev/sdc
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sdc 8:32 1 16022241280 0 disk

$ tar zxf disk.img.tgz -O | sudo dd status=progress conv=sync bs=1M of=/dev/sdc
[sudo] password for user: 
15992881152 bytes (16 GB, 15 GiB) copied, 212 s, 75,4 MB/s 
dd: error writing '/dev/sdc': No space left on device
0+15281 records in
15280+0 records out
16022241280 bytes (16 GB, 15 GiB) copied, 217,67 s, 73,6 MB/s

Why? It should stop after hit has written the 4GB image to the 16GB cart and never run out of space!

edited 54 mins ago

asked 58 mins ago

con-f-use

18318

$ ls -l disk.img.tgz
-rw-r--r-- 1 confus confus 192M Okt 5 00:53

$ tar -tvf disk.img.tgz
-rw-r--r-- root/root 4294968320 2018-10-05 00:52 disk.img

$ lsblk -lb /dev/sdc
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sdc 8:32 1 16022241280 0 disk

$ tar zxf disk.img.tgz -O | sudo dd status=progress conv=sync bs=1M of=/dev/sdc
[sudo] password for user: 
15992881152 bytes (16 GB, 15 GiB) copied, 212 s, 75,4 MB/s 
dd: error writing '/dev/sdc': No space left on device
0+15281 records in
15280+0 records out
16022241280 bytes (16 GB, 15 GiB) copied, 217,67 s, 73,6 MB/s

Why? It should stop after hit has written the 4GB image to the 16GB cart and never run out of space!

pipe tar dd disk-image

edited 54 mins ago

asked 58 mins ago

con-f-use

18318

edited 54 mins ago

asked 58 mins ago

con-f-use

18318

edited 54 mins ago

asked 58 mins ago

con-f-use

18318

asked 58 mins ago

con-f-use

18318

asked 58 mins ago

con-f-use

18318

Do you have the disk space to try running this through dd and writing it to another file? tar zxf disk.img.tgz -O | dd status=progress conv=sync bs=1M of=/path/to/some/file/on/disk ? If so, does that get you an exact copy of the original file?
â€“Â Andy Dalton
48 mins ago

Why do you have conv=sync? Did you mean to use conv=fsync perhaps?
â€“Â Ralph RÃ¶nnquist
38 mins ago

add a commentÂ |Â

Do you have the disk space to try running this through dd and writing it to another file? tar zxf disk.img.tgz -O | dd status=progress conv=sync bs=1M of=/path/to/some/file/on/disk ? If so, does that get you an exact copy of the original file?
â€“Â Andy Dalton
48 mins ago

Why do you have conv=sync? Did you mean to use conv=fsync perhaps?
â€“Â Ralph RÃ¶nnquist
38 mins ago

Do you have the disk space to try running this through dd and writing it to another file? tar zxf disk.img.tgz -O | dd status=progress conv=sync bs=1M of=/path/to/some/file/on/disk ? If so, does that get you an exact copy of the original file?
â€“Â Andy Dalton
48 mins ago

Why do you have conv=sync? Did you mean to use conv=fsync perhaps?
â€“Â Ralph RÃ¶nnquist
38 mins ago

add a commentÂ |Â

1 Answer
1

active

oldest

votes

up vote
11
down vote

It's because you're doing it wrong.

You're using bs=1M but reading from stdin, pipe, will have smaller reads. In fact, according to dd, you didn't get a single full read.

And then you have conv=sync which complements incomplete reads with zeroes.

0+15281 records in
15280+0 records out

dd received 0 full and 15281 incomplete reads, and wrote 15280 full blocks (conv=sync zero filled). So the output is much much larger than the input, until you get no space left.

 sync pad every input block with NULs to ibs-size; when used with
 block or unblock, pad with spaces rather than NULs

To solve this, you could remove conv=sync and add iflag=fullblock.

answered 36 mins ago

frostschutz

24.9k14878

add a commentÂ |Â

Your Answer

StackExchange.ready(function()
var channelOptions =
tags: "".split(" "),
id: "106"
;
initTagRenderer("".split(" "), "".split(" "), channelOptions);

StackExchange.using("externalEditor", function()
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled)
StackExchange.using("snippets", function()
createEditor();
);

else
createEditor();

);

function createEditor()
StackExchange.prepareEditor(
heartbeatType: 'answer',
convertImagesToLinks: false,
noModals: false,
showLowRepImageUploadWarning: true,
reputationToPostImages: null,
bindNavPrevention: true,
postfix: "",
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
);

);

draft saved

draft discarded

StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2funix.stackexchange.com%2fquestions%2f473464%2fwhy-does-piping-tar-into-dd-not-stop-until-the-disk-is-full%23new-answer', 'question_page');

);

Post as a guest

Name

1 Answer
1

active

oldest

votes

1 Answer
1

active

oldest

votes

up vote
11
down vote

It's because you're doing it wrong.

You're using bs=1M but reading from stdin, pipe, will have smaller reads. In fact, according to dd, you didn't get a single full read.

And then you have conv=sync which complements incomplete reads with zeroes.

0+15281 records in
15280+0 records out

dd received 0 full and 15281 incomplete reads, and wrote 15280 full blocks (conv=sync zero filled). So the output is much much larger than the input, until you get no space left.

 sync pad every input block with NULs to ibs-size; when used with
 block or unblock, pad with spaces rather than NULs

To solve this, you could remove conv=sync and add iflag=fullblock.

answered 36 mins ago

frostschutz

24.9k14878

add a commentÂ |Â

up vote
11
down vote

It's because you're doing it wrong.

You're using bs=1M but reading from stdin, pipe, will have smaller reads. In fact, according to dd, you didn't get a single full read.

And then you have conv=sync which complements incomplete reads with zeroes.

0+15281 records in
15280+0 records out

dd received 0 full and 15281 incomplete reads, and wrote 15280 full blocks (conv=sync zero filled). So the output is much much larger than the input, until you get no space left.

 sync pad every input block with NULs to ibs-size; when used with
 block or unblock, pad with spaces rather than NULs

To solve this, you could remove conv=sync and add iflag=fullblock.

answered 36 mins ago

frostschutz

24.9k14878

add a commentÂ |Â

up vote
11
down vote

It's because you're doing it wrong.

You're using bs=1M but reading from stdin, pipe, will have smaller reads. In fact, according to dd, you didn't get a single full read.

And then you have conv=sync which complements incomplete reads with zeroes.

0+15281 records in
15280+0 records out

dd received 0 full and 15281 incomplete reads, and wrote 15280 full blocks (conv=sync zero filled). So the output is much much larger than the input, until you get no space left.

 sync pad every input block with NULs to ibs-size; when used with
 block or unblock, pad with spaces rather than NULs

To solve this, you could remove conv=sync and add iflag=fullblock.

answered 36 mins ago

frostschutz

24.9k14878

It's because you're doing it wrong.

You're using bs=1M but reading from stdin, pipe, will have smaller reads. In fact, according to dd, you didn't get a single full read.

And then you have conv=sync which complements incomplete reads with zeroes.

0+15281 records in
15280+0 records out

dd received 0 full and 15281 incomplete reads, and wrote 15280 full blocks (conv=sync zero filled). So the output is much much larger than the input, until you get no space left.

 sync pad every input block with NULs to ibs-size; when used with
 block or unblock, pad with spaces rather than NULs

To solve this, you could remove conv=sync and add iflag=fullblock.

answered 36 mins ago

frostschutz

24.9k14878

answered 36 mins ago

frostschutz

24.9k14878

answered 36 mins ago

frostschutz

24.9k14878

answered 36 mins ago

frostschutz

24.9k14878

add a commentÂ |Â

draft saved

draft discarded

draft saved

draft discarded

Post as a guest

Name

Search This Blog

Iyfjky