awk: pipe output of (conditional) print to gzip

up vote
3
down vote

favorite

consider this file:

#!/usr/bin/env bash
cat > example_file.txt <<EOL
group, value
1, 3.21
1, 3.42
1, 3.5
2, 4.1
2, 4.2
EOL

in the following script, I group the rows of this file by
the values in the first column (the values in the first column are already sorted) and print each group to an individual txt file:

var=$(echo 'example_file.txt')
var2=$(echo $var|sed "s/.txt//g")
mkdir -p output
cat $var | awk -v varn="$var2" -F, 'FNR == 1 header = $0;next !seen[$1]++ print header > ("output/"varn"_"$1".txt") print > ("output/"varn"_"$1".txt");'

question

How to print the result to a compressed stream "output/"varn"_"$1".gz" (instead of an uncompressed txt file "output/"varn"_"$1".txt")?

(so the desired output is the same as that the scrip produces now, only I
want the outputed files to be compressed and saved to .txt.gz instead of plain text ones as the code does now).

(I tried using gzip > inside the print blocks but to no avail :(

(PS I'm a bit of a an awk noob and so the question might be a really dumb one.)

edited Aug 18 at 8:39

asked Aug 18 at 8:34

user2413

3,688133462

1

FYI that's a UUOC(Useless Use of Cat) and also of echo and sed: you can assign the variables simply as var='example_file.txt' and varn="var%.txt"
â€“Â steeldriver
Aug 18 at 10:42

add a commentÂ |Â

up vote
3
down vote

favorite

consider this file:

#!/usr/bin/env bash
cat > example_file.txt <<EOL
group, value
1, 3.21
1, 3.42
1, 3.5
2, 4.1
2, 4.2
EOL

in the following script, I group the rows of this file by
the values in the first column (the values in the first column are already sorted) and print each group to an individual txt file:

var=$(echo 'example_file.txt')
var2=$(echo $var|sed "s/.txt//g")
mkdir -p output
cat $var | awk -v varn="$var2" -F, 'FNR == 1 header = $0;next !seen[$1]++ print header > ("output/"varn"_"$1".txt") print > ("output/"varn"_"$1".txt");'

question

How to print the result to a compressed stream "output/"varn"_"$1".gz" (instead of an uncompressed txt file "output/"varn"_"$1".txt")?

(so the desired output is the same as that the scrip produces now, only I
want the outputed files to be compressed and saved to .txt.gz instead of plain text ones as the code does now).

(I tried using gzip > inside the print blocks but to no avail :(

(PS I'm a bit of a an awk noob and so the question might be a really dumb one.)

edited Aug 18 at 8:39

asked Aug 18 at 8:34

user2413

3,688133462

1

FYI that's a UUOC(Useless Use of Cat) and also of echo and sed: you can assign the variables simply as var='example_file.txt' and varn="var%.txt"
â€“Â steeldriver
Aug 18 at 10:42

add a commentÂ |Â

up vote
3
down vote

favorite

consider this file:

#!/usr/bin/env bash
cat > example_file.txt <<EOL
group, value
1, 3.21
1, 3.42
1, 3.5
2, 4.1
2, 4.2
EOL

in the following script, I group the rows of this file by
the values in the first column (the values in the first column are already sorted) and print each group to an individual txt file:

var=$(echo 'example_file.txt')
var2=$(echo $var|sed "s/.txt//g")
mkdir -p output
cat $var | awk -v varn="$var2" -F, 'FNR == 1 header = $0;next !seen[$1]++ print header > ("output/"varn"_"$1".txt") print > ("output/"varn"_"$1".txt");'

question

How to print the result to a compressed stream "output/"varn"_"$1".gz" (instead of an uncompressed txt file "output/"varn"_"$1".txt")?

(so the desired output is the same as that the scrip produces now, only I
want the outputed files to be compressed and saved to .txt.gz instead of plain text ones as the code does now).

(I tried using gzip > inside the print blocks but to no avail :(

(PS I'm a bit of a an awk noob and so the question might be a really dumb one.)

edited Aug 18 at 8:39

asked Aug 18 at 8:34

user2413

3,688133462

consider this file:

#!/usr/bin/env bash
cat > example_file.txt <<EOL
group, value
1, 3.21
1, 3.42
1, 3.5
2, 4.1
2, 4.2
EOL

in the following script, I group the rows of this file by
the values in the first column (the values in the first column are already sorted) and print each group to an individual txt file:

var=$(echo 'example_file.txt')
var2=$(echo $var|sed "s/.txt//g")
mkdir -p output
cat $var | awk -v varn="$var2" -F, 'FNR == 1 header = $0;next !seen[$1]++ print header > ("output/"varn"_"$1".txt") print > ("output/"varn"_"$1".txt");'

question

How to print the result to a compressed stream "output/"varn"_"$1".gz" (instead of an uncompressed txt file "output/"varn"_"$1".txt")?

(so the desired output is the same as that the scrip produces now, only I
want the outputed files to be compressed and saved to .txt.gz instead of plain text ones as the code does now).

(I tried using gzip > inside the print blocks but to no avail :(

(PS I'm a bit of a an awk noob and so the question might be a really dumb one.)

edited Aug 18 at 8:39

asked Aug 18 at 8:34

user2413

3,688133462

edited Aug 18 at 8:39

asked Aug 18 at 8:34

user2413

3,688133462

asked Aug 18 at 8:34

user2413

3,688133462

asked Aug 18 at 8:34

user2413

3,688133462

1

FYI that's a UUOC(Useless Use of Cat) and also of echo and sed: you can assign the variables simply as var='example_file.txt' and varn="var%.txt"
â€“Â steeldriver
Aug 18 at 10:42

add a commentÂ |Â

1

FYI that's a UUOC(Useless Use of Cat) and also of echo and sed: you can assign the variables simply as var='example_file.txt' and varn="var%.txt"
â€“Â steeldriver
Aug 18 at 10:42

FYI that's a UUOC(Useless Use of Cat) and also of echo and sed: you can assign the variables simply as var='example_file.txt' and varn="var%.txt"
â€“Â steeldriver
Aug 18 at 10:42

add a commentÂ |Â

1 Answer
1

active

oldest

votes

up vote
6
down vote

accepted

You can pipe to commands in GNU awk's print. From the GNU awk manual:

print items | command
It is possible to send output to another program through a pipe
instead of into a file. This redirection opens a pipe to command, and
writes the values of items through this pipe to another process
created to execute command.

The redirection argument command is actually an awk expression. Its
value is converted to a string whose contents give the shell command
to be run. For example, the following produces two files, one unsorted
list of peoplesÃ¢Â€Â™ names, and one list sorted in reverse alphabetical
order:
awk ' print $1 > "names.unsorted"
 command = "sort -r > names.sorted"
 print $1 ' mail-list

So:

awk -v varn="$var2" -F, 'FNR == 1 header = $0;next
 !seen[$1]++ print header 
 "gzip > output/"varn"_"$1".gz";'

For example:

% echo 1 2 | awk ' "gzip > "$1".gz"'
% zcat 1.gz 
2

edited Aug 19 at 22:48

answered Aug 18 at 8:54

muru

129k19271462

Thanks! Small problem (I think) running the corrected code (copy pasting the block after the So:) I get awk: 1: unexpected character '.' awk: line 1: runaway string constant ");} ...
â€“Â user2413
Aug 19 at 17:11

add a commentÂ |Â

Your Answer

StackExchange.ready(function()
var channelOptions =
tags: "".split(" "),
id: "89"
;
initTagRenderer("".split(" "), "".split(" "), channelOptions);

StackExchange.using("externalEditor", function()
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled)
StackExchange.using("snippets", function()
createEditor();
);

else
createEditor();

);

function createEditor()
StackExchange.prepareEditor(
heartbeatType: 'answer',
convertImagesToLinks: true,
noModals: false,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
);

);

draft saved

draft discarded

StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2faskubuntu.com%2fquestions%2f1066499%2fawk-pipe-output-of-conditional-print-to-gzip%23new-answer', 'question_page');

);

Post as a guest

Name

1 Answer
1

active

oldest

votes

1 Answer
1

active

oldest

votes

up vote
6
down vote

accepted

You can pipe to commands in GNU awk's print. From the GNU awk manual:

print items | command
It is possible to send output to another program through a pipe
instead of into a file. This redirection opens a pipe to command, and
writes the values of items through this pipe to another process
created to execute command.

The redirection argument command is actually an awk expression. Its
value is converted to a string whose contents give the shell command
to be run. For example, the following produces two files, one unsorted
list of peoplesÃ¢Â€Â™ names, and one list sorted in reverse alphabetical
order:
awk ' print $1 > "names.unsorted"
 command = "sort -r > names.sorted"
 print $1 ' mail-list

So:

awk -v varn="$var2" -F, 'FNR == 1 header = $0;next
 !seen[$1]++ print header 
 "gzip > output/"varn"_"$1".gz";'

For example:

% echo 1 2 | awk ' "gzip > "$1".gz"'
% zcat 1.gz 
2

edited Aug 19 at 22:48

answered Aug 18 at 8:54

muru

129k19271462

Thanks! Small problem (I think) running the corrected code (copy pasting the block after the So:) I get awk: 1: unexpected character '.' awk: line 1: runaway string constant ");} ...
â€“Â user2413
Aug 19 at 17:11

add a commentÂ |Â

up vote
6
down vote

accepted

You can pipe to commands in GNU awk's print. From the GNU awk manual:

print items | command
It is possible to send output to another program through a pipe
instead of into a file. This redirection opens a pipe to command, and
writes the values of items through this pipe to another process
created to execute command.

The redirection argument command is actually an awk expression. Its
value is converted to a string whose contents give the shell command
to be run. For example, the following produces two files, one unsorted
list of peoplesÃ¢Â€Â™ names, and one list sorted in reverse alphabetical
order:
awk ' print $1 > "names.unsorted"
 command = "sort -r > names.sorted"
 print $1 ' mail-list

So:

awk -v varn="$var2" -F, 'FNR == 1 header = $0;next
 !seen[$1]++ print header 
 "gzip > output/"varn"_"$1".gz";'

For example:

% echo 1 2 | awk ' "gzip > "$1".gz"'
% zcat 1.gz 
2

edited Aug 19 at 22:48

answered Aug 18 at 8:54

muru

129k19271462

Thanks! Small problem (I think) running the corrected code (copy pasting the block after the So:) I get awk: 1: unexpected character '.' awk: line 1: runaway string constant ");} ...
â€“Â user2413
Aug 19 at 17:11

add a commentÂ |Â

up vote
6
down vote

accepted

You can pipe to commands in GNU awk's print. From the GNU awk manual:

print items | command
It is possible to send output to another program through a pipe
instead of into a file. This redirection opens a pipe to command, and
writes the values of items through this pipe to another process
created to execute command.

The redirection argument command is actually an awk expression. Its
value is converted to a string whose contents give the shell command
to be run. For example, the following produces two files, one unsorted
list of peoplesÃ¢Â€Â™ names, and one list sorted in reverse alphabetical
order:
awk ' print $1 > "names.unsorted"
 command = "sort -r > names.sorted"
 print $1 ' mail-list

So:

awk -v varn="$var2" -F, 'FNR == 1 header = $0;next
 !seen[$1]++ print header 
 "gzip > output/"varn"_"$1".gz";'

For example:

% echo 1 2 | awk ' "gzip > "$1".gz"'
% zcat 1.gz 
2

edited Aug 19 at 22:48

answered Aug 18 at 8:54

muru

129k19271462

You can pipe to commands in GNU awk's print. From the GNU awk manual:

print items | command
It is possible to send output to another program through a pipe
instead of into a file. This redirection opens a pipe to command, and
writes the values of items through this pipe to another process
created to execute command.

The redirection argument command is actually an awk expression. Its
value is converted to a string whose contents give the shell command
to be run. For example, the following produces two files, one unsorted
list of peoplesÃ¢Â€Â™ names, and one list sorted in reverse alphabetical
order:
awk ' print $1 > "names.unsorted"
 command = "sort -r > names.sorted"
 print $1 ' mail-list

So:

awk -v varn="$var2" -F, 'FNR == 1 header = $0;next
 !seen[$1]++ print header 
 "gzip > output/"varn"_"$1".gz";'

For example:

% echo 1 2 | awk ' "gzip > "$1".gz"'
% zcat 1.gz 
2

edited Aug 19 at 22:48

answered Aug 18 at 8:54

muru

129k19271462

edited Aug 19 at 22:48

answered Aug 18 at 8:54

muru

129k19271462

answered Aug 18 at 8:54

muru

129k19271462

answered Aug 18 at 8:54

muru

129k19271462

Thanks! Small problem (I think) running the corrected code (copy pasting the block after the So:) I get awk: 1: unexpected character '.' awk: line 1: runaway string constant ");} ...
â€“Â user2413
Aug 19 at 17:11

add a commentÂ |Â

Thanks! Small problem (I think) running the corrected code (copy pasting the block after the So:) I get awk: 1: unexpected character '.' awk: line 1: runaway string constant ");} ...
â€“Â user2413
Aug 19 at 17:11

Thanks! Small problem (I think) running the corrected code (copy pasting the block after the So:) I get awk: 1: unexpected character '.' awk: line 1: runaway string constant ");} ...
â€“Â user2413
Aug 19 at 17:11

add a commentÂ |Â

draft saved

draft discarded

draft saved

draft discarded

Post as a guest

Name

Search This Blog

Iyfjky