Mixing
Why does piping `tar` into `dd` not stop until the disk is full?
Clash Royale CLAN TAG#URR8PPP
up vote
2
down vote
favorite
I have a tar archive of a single disk image. The image inside this tar file is about 4GB in size. I pipe the output of tar xf into dd to write the disk image to an SD card. The diskdump never stops until the card is full. Here is my shell session:
$ ls -l disk.img.tgz
-rw-r--r-- 1 confus confus 192M Okt 5 00:53
$ tar -tvf disk.img.tgz
-rw-r--r-- root/root 4294968320 2018-10-05 00:52 disk.img
$ lsblk -lb /dev/sdc
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sdc 8:32 1 16022241280 0 disk
$ tar zxf disk.img.tgz -O | sudo dd status=progress conv=sync bs=1M of=/dev/sdc
[sudo] password for user:
15992881152 bytes (16 GB, 15 GiB) copied, 212 s, 75,4 MB/s
dd: error writing '/dev/sdc': No space left on device
0+15281 records in
15280+0 records out
16022241280 bytes (16 GB, 15 GiB) copied, 217,67 s, 73,6 MB/s
Why? It should stop after hit has written the 4GB image to the 16GB cart and never run out of space!
pipe tar dd disk-image
asked 58 mins ago
con-f-use
18318
add a comment |Â
up vote
2
down vote
favorite
I have a tar archive of a single disk image. The image inside this tar file is about 4GB in size. I pipe the output of tar xf into dd to write the disk image to an SD card. The diskdump never stops until the card is full. Here is my shell session:
$ ls -l disk.img.tgz
-rw-r--r-- 1 confus confus 192M Okt 5 00:53
$ tar -tvf disk.img.tgz
-rw-r--r-- root/root 4294968320 2018-10-05 00:52 disk.img
$ lsblk -lb /dev/sdc
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sdc 8:32 1 16022241280 0 disk
$ tar zxf disk.img.tgz -O | sudo dd status=progress conv=sync bs=1M of=/dev/sdc
[sudo] password for user:
15992881152 bytes (16 GB, 15 GiB) copied, 212 s, 75,4 MB/s
dd: error writing '/dev/sdc': No space left on device
0+15281 records in
15280+0 records out
16022241280 bytes (16 GB, 15 GiB) copied, 217,67 s, 73,6 MB/s
Why? It should stop after hit has written the 4GB image to the 16GB cart and never run out of space!
pipe tar dd disk-image
asked 58 mins ago
con-f-use
18318
Do you have the disk space to try running this throughddand writing it to another file?tar zxf disk.img.tgz -O | dd status=progress conv=sync bs=1M of=/path/to/some/file/on/disk? If so, does that get you an exact copy of the original file?
– Andy Dalton
48 mins ago
Why do you haveconv=sync? Did you mean to useconv=fsyncperhaps?
– Ralph Rönnquist
38 mins ago
add a comment |Â
up vote
2
down vote
favorite
up vote
2
down vote
favorite
I have a tar archive of a single disk image. The image inside this tar file is about 4GB in size. I pipe the output of tar xf into dd to write the disk image to an SD card. The diskdump never stops until the card is full. Here is my shell session:
$ ls -l disk.img.tgz
-rw-r--r-- 1 confus confus 192M Okt 5 00:53
$ tar -tvf disk.img.tgz
-rw-r--r-- root/root 4294968320 2018-10-05 00:52 disk.img
$ lsblk -lb /dev/sdc
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sdc 8:32 1 16022241280 0 disk
$ tar zxf disk.img.tgz -O | sudo dd status=progress conv=sync bs=1M of=/dev/sdc
[sudo] password for user:
15992881152 bytes (16 GB, 15 GiB) copied, 212 s, 75,4 MB/s
dd: error writing '/dev/sdc': No space left on device
0+15281 records in
15280+0 records out
16022241280 bytes (16 GB, 15 GiB) copied, 217,67 s, 73,6 MB/s
Why? It should stop after hit has written the 4GB image to the 16GB cart and never run out of space!
pipe tar dd disk-image
asked 58 mins ago
con-f-use
18318
I have a tar archive of a single disk image. The image inside this tar file is about 4GB in size. I pipe the output of tar xf into dd to write the disk image to an SD card. The diskdump never stops until the card is full. Here is my shell session:
$ ls -l disk.img.tgz
-rw-r--r-- 1 confus confus 192M Okt 5 00:53
$ tar -tvf disk.img.tgz
-rw-r--r-- root/root 4294968320 2018-10-05 00:52 disk.img
$ lsblk -lb /dev/sdc
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sdc 8:32 1 16022241280 0 disk
$ tar zxf disk.img.tgz -O | sudo dd status=progress conv=sync bs=1M of=/dev/sdc
[sudo] password for user:
15992881152 bytes (16 GB, 15 GiB) copied, 212 s, 75,4 MB/s
dd: error writing '/dev/sdc': No space left on device
0+15281 records in
15280+0 records out
16022241280 bytes (16 GB, 15 GiB) copied, 217,67 s, 73,6 MB/s
Why? It should stop after hit has written the 4GB image to the 16GB cart and never run out of space!
pipe tar dd disk-image
pipe tar dd disk-image
asked 58 mins ago
con-f-use
18318
asked 58 mins ago
con-f-use
18318
edited 54 mins ago
asked 58 mins ago
con-f-use
18318
asked 58 mins ago
con-f-use
18318
asked 58 mins ago
con-f-use
18318
18318
Do you have the disk space to try running this throughddand writing it to another file?tar zxf disk.img.tgz -O | dd status=progress conv=sync bs=1M of=/path/to/some/file/on/disk? If so, does that get you an exact copy of the original file?
– Andy Dalton
48 mins ago
Why do you haveconv=sync? Did you mean to useconv=fsyncperhaps?
– Ralph Rönnquist
38 mins ago
add a comment |Â
Do you have the disk space to try running this throughddand writing it to another file?tar zxf disk.img.tgz -O | dd status=progress conv=sync bs=1M of=/path/to/some/file/on/disk? If so, does that get you an exact copy of the original file?
– Andy Dalton
48 mins ago
Why do you haveconv=sync? Did you mean to useconv=fsyncperhaps?
– Ralph Rönnquist
38 mins ago
Do you have the disk space to try running this through
dd and writing it to another file? tar zxf disk.img.tgz -O | dd status=progress conv=sync bs=1M of=/path/to/some/file/on/disk ? If so, does that get you an exact copy of the original file?– Andy Dalton
48 mins ago
Do you have the disk space to try running this through
dd and writing it to another file? tar zxf disk.img.tgz -O | dd status=progress conv=sync bs=1M of=/path/to/some/file/on/disk ? If so, does that get you an exact copy of the original file?– Andy Dalton
48 mins ago
Why do you have
conv=sync? Did you mean to use conv=fsync perhaps?– Ralph Rönnquist
38 mins ago
Why do you have
conv=sync? Did you mean to use conv=fsync perhaps?– Ralph Rönnquist
38 mins ago
add a comment |Â
1 Answer
1
active
oldest
votes
up vote
11
down vote
It's because you're doing it wrong.
You're using bs=1M but reading from stdin, pipe, will have smaller reads. In fact, according to dd, you didn't get a single full read.
And then you have conv=sync which complements incomplete reads with zeroes.
0+15281 records in
15280+0 records out
dd received 0 full and 15281 incomplete reads, and wrote 15280 full blocks (conv=sync zero filled). So the output is much much larger than the input, until you get no space left.
sync pad every input block with NULs to ibs-size; when used with
block or unblock, pad with spaces rather than NULs
To solve this, you could remove conv=sync and add iflag=fullblock.
answered 36 mins ago
frostschutz
24.9k14878
add a comment |Â
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
up vote
11
down vote
It's because you're doing it wrong.
You're using bs=1M but reading from stdin, pipe, will have smaller reads. In fact, according to dd, you didn't get a single full read.
And then you have conv=sync which complements incomplete reads with zeroes.
0+15281 records in
15280+0 records out
dd received 0 full and 15281 incomplete reads, and wrote 15280 full blocks (conv=sync zero filled). So the output is much much larger than the input, until you get no space left.
sync pad every input block with NULs to ibs-size; when used with
block or unblock, pad with spaces rather than NULs
To solve this, you could remove conv=sync and add iflag=fullblock.
answered 36 mins ago
frostschutz
24.9k14878
add a comment |Â
up vote
11
down vote
It's because you're doing it wrong.
You're using bs=1M but reading from stdin, pipe, will have smaller reads. In fact, according to dd, you didn't get a single full read.
And then you have conv=sync which complements incomplete reads with zeroes.
0+15281 records in
15280+0 records out
dd received 0 full and 15281 incomplete reads, and wrote 15280 full blocks (conv=sync zero filled). So the output is much much larger than the input, until you get no space left.
sync pad every input block with NULs to ibs-size; when used with
block or unblock, pad with spaces rather than NULs
To solve this, you could remove conv=sync and add iflag=fullblock.
answered 36 mins ago
frostschutz
24.9k14878
add a comment |Â
up vote
11
down vote
up vote
11
down vote
It's because you're doing it wrong.
You're using bs=1M but reading from stdin, pipe, will have smaller reads. In fact, according to dd, you didn't get a single full read.
And then you have conv=sync which complements incomplete reads with zeroes.
0+15281 records in
15280+0 records out
dd received 0 full and 15281 incomplete reads, and wrote 15280 full blocks (conv=sync zero filled). So the output is much much larger than the input, until you get no space left.
sync pad every input block with NULs to ibs-size; when used with
block or unblock, pad with spaces rather than NULs
To solve this, you could remove conv=sync and add iflag=fullblock.
answered 36 mins ago
frostschutz
24.9k14878
It's because you're doing it wrong.
You're using bs=1M but reading from stdin, pipe, will have smaller reads. In fact, according to dd, you didn't get a single full read.
And then you have conv=sync which complements incomplete reads with zeroes.
0+15281 records in
15280+0 records out
dd received 0 full and 15281 incomplete reads, and wrote 15280 full blocks (conv=sync zero filled). So the output is much much larger than the input, until you get no space left.
sync pad every input block with NULs to ibs-size; when used with
block or unblock, pad with spaces rather than NULs
To solve this, you could remove conv=sync and add iflag=fullblock.
answered 36 mins ago
frostschutz
24.9k14878
answered 36 mins ago
frostschutz
24.9k14878
answered 36 mins ago
frostschutz
24.9k14878
answered 36 mins ago
frostschutz
24.9k14878
24.9k14878
add a comment |Â
add a comment |Â
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2funix.stackexchange.com%2fquestions%2f473464%2fwhy-does-piping-tar-into-dd-not-stop-until-the-disk-is-full%23new-answer', 'question_page');
);
Post as a guest
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Do you have the disk space to try running this through
ddand writing it to another file?tar zxf disk.img.tgz -O | dd status=progress conv=sync bs=1M of=/path/to/some/file/on/disk? If so, does that get you an exact copy of the original file?– Andy Dalton
48 mins ago
Why do you have
conv=sync? Did you mean to useconv=fsyncperhaps?– Ralph Rönnquist
38 mins ago