Merging text files and adding separator
Clash Royale CLAN TAG#URR8PPP
up vote
3
down vote
favorite
I want to add a separator like this "==============" and a blank new line
I tried to do this, but failed and causes high CPU usage. i mean the cpu become rotate ery fast and noisy when i run the script
This needs to be for around 100000 text files.
this is the code that I use
#!/bin/bash
for F in *.txt ; do
type "$F"
echo .
echo ========
echo .
done >> Combined.txt;
please advice
text-processing scripting
add a comment |Â
up vote
3
down vote
favorite
I want to add a separator like this "==============" and a blank new line
I tried to do this, but failed and causes high CPU usage. i mean the cpu become rotate ery fast and noisy when i run the script
This needs to be for around 100000 text files.
this is the code that I use
#!/bin/bash
for F in *.txt ; do
type "$F"
echo .
echo ========
echo .
done >> Combined.txt;
please advice
text-processing scripting
See Concatenate multiple files with two blank lines as delimiter? and the linked questions/answers.
â don_crissti
2 hours ago
add a comment |Â
up vote
3
down vote
favorite
up vote
3
down vote
favorite
I want to add a separator like this "==============" and a blank new line
I tried to do this, but failed and causes high CPU usage. i mean the cpu become rotate ery fast and noisy when i run the script
This needs to be for around 100000 text files.
this is the code that I use
#!/bin/bash
for F in *.txt ; do
type "$F"
echo .
echo ========
echo .
done >> Combined.txt;
please advice
text-processing scripting
I want to add a separator like this "==============" and a blank new line
I tried to do this, but failed and causes high CPU usage. i mean the cpu become rotate ery fast and noisy when i run the script
This needs to be for around 100000 text files.
this is the code that I use
#!/bin/bash
for F in *.txt ; do
type "$F"
echo .
echo ========
echo .
done >> Combined.txt;
please advice
text-processing scripting
text-processing scripting
edited 1 hour ago
Jeff Schaller
33.6k851113
33.6k851113
asked 2 hours ago
Shervan
32010
32010
See Concatenate multiple files with two blank lines as delimiter? and the linked questions/answers.
â don_crissti
2 hours ago
add a comment |Â
See Concatenate multiple files with two blank lines as delimiter? and the linked questions/answers.
â don_crissti
2 hours ago
See Concatenate multiple files with two blank lines as delimiter? and the linked questions/answers.
â don_crissti
2 hours ago
See Concatenate multiple files with two blank lines as delimiter? and the linked questions/answers.
â don_crissti
2 hours ago
add a comment |Â
4 Answers
4
active
oldest
votes
up vote
1
down vote
accepted
I would simplify your commands as follows:
#!/bin/bash
for file in *.txt; do
cat $file >> Combined.txt
printf 'nn=========nn' >> Combined.txt
done
add a comment |Â
up vote
0
down vote
I edited your question to make the code readable... but also fixed some things at the same time;
- 'do' needs a matching 'done'
- & means branch while && means and
- % is cmd $ is bash
- () was a needless subshell
- one only need one write out
- use shellcheck.net
1
Ideally you'd leave OP's code intact and post any modifications/fixes as part of your answer. That way other people can see the original code/attempt without having to scroll through edit revisions...
â don_crissti
2 hours ago
add a comment |Â
up vote
0
down vote
If you're going to do it for thousands of files, you may want to avoid running several commands per file. With GNU awk
:
printf '%s' ./*.txt | xargs -r0 gawk '
BEGINFILE if (NR) print "n==========n";1' > combined.out
Don't give a .txt
extension to the output file if you're going to put it in the same directory, or it's going to be selected as an input file and causing an infinite loop (probably your problem in the first place).
Or use a shell where cat
is builtin like ksh93
:
#! /bin/ksh93
firstpass=true
for file in *.txt; do
"$firstpass" || print 'n===========n'
firstpass=false
command /opt/ast/bin/cat < "$file"
done > combined.out
All those commands in the loop are built-in, so running them doesn't involve forking new processes nor loading external executable, so that would make the performance tolerable.
add a comment |Â
up vote
0
down vote
Using FNR
and NR
in awk
#!/bin/bash
outfile="$( mktemp combined.txt.XXXXXX )"
echo "Output file: $outfile"
awk 'FNR==1 && NR>1 printf("n%snn","========") 1' *.txt > "$outfile"
echo "Finished."
A line-by-line description:
outfile="$( mktemp combined.txt.XXXXXX )"
Use mktemp
to create an empty new file with a unique name (eg, combined.txt.HDpgMn
). You can use more X
characters for a longer random suffix. Enclose the command in "$(
...)"
to store the new file's name in the variable outfile
.
echo "Saving to file: $outfile"
Print the name of the output file. (When the script has finished, you may wish to rename the output file to remove the string of random characters following the .txt
.)
awk 'FNR==1 && NR>1 printf("n%snn","========") 1' *.txt > "$outfile"
Print...
- a blank line,
- a short line of "=" characters,
- and another blank line
...at the start of each input file, except for the first input file. FNR
counts the input file line numbers, resetting at the start of each file. NR
counts the line numbers and does not reset.
In the awk
statement, the 1
just before the closing single quotation mark evaluates to TRUE
for every line, and performs the default action of printing that line. (In other words, awk '1'
works like cat
.)
echo "Finished."
Inform the user when the script is done. (Not strictly necessary, since you'll see the command prompt anyway, but it doesn't hurt.)
add a comment |Â
4 Answers
4
active
oldest
votes
4 Answers
4
active
oldest
votes
active
oldest
votes
active
oldest
votes
up vote
1
down vote
accepted
I would simplify your commands as follows:
#!/bin/bash
for file in *.txt; do
cat $file >> Combined.txt
printf 'nn=========nn' >> Combined.txt
done
add a comment |Â
up vote
1
down vote
accepted
I would simplify your commands as follows:
#!/bin/bash
for file in *.txt; do
cat $file >> Combined.txt
printf 'nn=========nn' >> Combined.txt
done
add a comment |Â
up vote
1
down vote
accepted
up vote
1
down vote
accepted
I would simplify your commands as follows:
#!/bin/bash
for file in *.txt; do
cat $file >> Combined.txt
printf 'nn=========nn' >> Combined.txt
done
I would simplify your commands as follows:
#!/bin/bash
for file in *.txt; do
cat $file >> Combined.txt
printf 'nn=========nn' >> Combined.txt
done
answered 2 hours ago
Goro
7,20753168
7,20753168
add a comment |Â
add a comment |Â
up vote
0
down vote
I edited your question to make the code readable... but also fixed some things at the same time;
- 'do' needs a matching 'done'
- & means branch while && means and
- % is cmd $ is bash
- () was a needless subshell
- one only need one write out
- use shellcheck.net
1
Ideally you'd leave OP's code intact and post any modifications/fixes as part of your answer. That way other people can see the original code/attempt without having to scroll through edit revisions...
â don_crissti
2 hours ago
add a comment |Â
up vote
0
down vote
I edited your question to make the code readable... but also fixed some things at the same time;
- 'do' needs a matching 'done'
- & means branch while && means and
- % is cmd $ is bash
- () was a needless subshell
- one only need one write out
- use shellcheck.net
1
Ideally you'd leave OP's code intact and post any modifications/fixes as part of your answer. That way other people can see the original code/attempt without having to scroll through edit revisions...
â don_crissti
2 hours ago
add a comment |Â
up vote
0
down vote
up vote
0
down vote
I edited your question to make the code readable... but also fixed some things at the same time;
- 'do' needs a matching 'done'
- & means branch while && means and
- % is cmd $ is bash
- () was a needless subshell
- one only need one write out
- use shellcheck.net
I edited your question to make the code readable... but also fixed some things at the same time;
- 'do' needs a matching 'done'
- & means branch while && means and
- % is cmd $ is bash
- () was a needless subshell
- one only need one write out
- use shellcheck.net
edited 2 hours ago
answered 2 hours ago
user1133275
2,297412
2,297412
1
Ideally you'd leave OP's code intact and post any modifications/fixes as part of your answer. That way other people can see the original code/attempt without having to scroll through edit revisions...
â don_crissti
2 hours ago
add a comment |Â
1
Ideally you'd leave OP's code intact and post any modifications/fixes as part of your answer. That way other people can see the original code/attempt without having to scroll through edit revisions...
â don_crissti
2 hours ago
1
1
Ideally you'd leave OP's code intact and post any modifications/fixes as part of your answer. That way other people can see the original code/attempt without having to scroll through edit revisions...
â don_crissti
2 hours ago
Ideally you'd leave OP's code intact and post any modifications/fixes as part of your answer. That way other people can see the original code/attempt without having to scroll through edit revisions...
â don_crissti
2 hours ago
add a comment |Â
up vote
0
down vote
If you're going to do it for thousands of files, you may want to avoid running several commands per file. With GNU awk
:
printf '%s' ./*.txt | xargs -r0 gawk '
BEGINFILE if (NR) print "n==========n";1' > combined.out
Don't give a .txt
extension to the output file if you're going to put it in the same directory, or it's going to be selected as an input file and causing an infinite loop (probably your problem in the first place).
Or use a shell where cat
is builtin like ksh93
:
#! /bin/ksh93
firstpass=true
for file in *.txt; do
"$firstpass" || print 'n===========n'
firstpass=false
command /opt/ast/bin/cat < "$file"
done > combined.out
All those commands in the loop are built-in, so running them doesn't involve forking new processes nor loading external executable, so that would make the performance tolerable.
add a comment |Â
up vote
0
down vote
If you're going to do it for thousands of files, you may want to avoid running several commands per file. With GNU awk
:
printf '%s' ./*.txt | xargs -r0 gawk '
BEGINFILE if (NR) print "n==========n";1' > combined.out
Don't give a .txt
extension to the output file if you're going to put it in the same directory, or it's going to be selected as an input file and causing an infinite loop (probably your problem in the first place).
Or use a shell where cat
is builtin like ksh93
:
#! /bin/ksh93
firstpass=true
for file in *.txt; do
"$firstpass" || print 'n===========n'
firstpass=false
command /opt/ast/bin/cat < "$file"
done > combined.out
All those commands in the loop are built-in, so running them doesn't involve forking new processes nor loading external executable, so that would make the performance tolerable.
add a comment |Â
up vote
0
down vote
up vote
0
down vote
If you're going to do it for thousands of files, you may want to avoid running several commands per file. With GNU awk
:
printf '%s' ./*.txt | xargs -r0 gawk '
BEGINFILE if (NR) print "n==========n";1' > combined.out
Don't give a .txt
extension to the output file if you're going to put it in the same directory, or it's going to be selected as an input file and causing an infinite loop (probably your problem in the first place).
Or use a shell where cat
is builtin like ksh93
:
#! /bin/ksh93
firstpass=true
for file in *.txt; do
"$firstpass" || print 'n===========n'
firstpass=false
command /opt/ast/bin/cat < "$file"
done > combined.out
All those commands in the loop are built-in, so running them doesn't involve forking new processes nor loading external executable, so that would make the performance tolerable.
If you're going to do it for thousands of files, you may want to avoid running several commands per file. With GNU awk
:
printf '%s' ./*.txt | xargs -r0 gawk '
BEGINFILE if (NR) print "n==========n";1' > combined.out
Don't give a .txt
extension to the output file if you're going to put it in the same directory, or it's going to be selected as an input file and causing an infinite loop (probably your problem in the first place).
Or use a shell where cat
is builtin like ksh93
:
#! /bin/ksh93
firstpass=true
for file in *.txt; do
"$firstpass" || print 'n===========n'
firstpass=false
command /opt/ast/bin/cat < "$file"
done > combined.out
All those commands in the loop are built-in, so running them doesn't involve forking new processes nor loading external executable, so that would make the performance tolerable.
edited 1 hour ago
answered 1 hour ago
Stéphane Chazelas
287k53531868
287k53531868
add a comment |Â
add a comment |Â
up vote
0
down vote
Using FNR
and NR
in awk
#!/bin/bash
outfile="$( mktemp combined.txt.XXXXXX )"
echo "Output file: $outfile"
awk 'FNR==1 && NR>1 printf("n%snn","========") 1' *.txt > "$outfile"
echo "Finished."
A line-by-line description:
outfile="$( mktemp combined.txt.XXXXXX )"
Use mktemp
to create an empty new file with a unique name (eg, combined.txt.HDpgMn
). You can use more X
characters for a longer random suffix. Enclose the command in "$(
...)"
to store the new file's name in the variable outfile
.
echo "Saving to file: $outfile"
Print the name of the output file. (When the script has finished, you may wish to rename the output file to remove the string of random characters following the .txt
.)
awk 'FNR==1 && NR>1 printf("n%snn","========") 1' *.txt > "$outfile"
Print...
- a blank line,
- a short line of "=" characters,
- and another blank line
...at the start of each input file, except for the first input file. FNR
counts the input file line numbers, resetting at the start of each file. NR
counts the line numbers and does not reset.
In the awk
statement, the 1
just before the closing single quotation mark evaluates to TRUE
for every line, and performs the default action of printing that line. (In other words, awk '1'
works like cat
.)
echo "Finished."
Inform the user when the script is done. (Not strictly necessary, since you'll see the command prompt anyway, but it doesn't hurt.)
add a comment |Â
up vote
0
down vote
Using FNR
and NR
in awk
#!/bin/bash
outfile="$( mktemp combined.txt.XXXXXX )"
echo "Output file: $outfile"
awk 'FNR==1 && NR>1 printf("n%snn","========") 1' *.txt > "$outfile"
echo "Finished."
A line-by-line description:
outfile="$( mktemp combined.txt.XXXXXX )"
Use mktemp
to create an empty new file with a unique name (eg, combined.txt.HDpgMn
). You can use more X
characters for a longer random suffix. Enclose the command in "$(
...)"
to store the new file's name in the variable outfile
.
echo "Saving to file: $outfile"
Print the name of the output file. (When the script has finished, you may wish to rename the output file to remove the string of random characters following the .txt
.)
awk 'FNR==1 && NR>1 printf("n%snn","========") 1' *.txt > "$outfile"
Print...
- a blank line,
- a short line of "=" characters,
- and another blank line
...at the start of each input file, except for the first input file. FNR
counts the input file line numbers, resetting at the start of each file. NR
counts the line numbers and does not reset.
In the awk
statement, the 1
just before the closing single quotation mark evaluates to TRUE
for every line, and performs the default action of printing that line. (In other words, awk '1'
works like cat
.)
echo "Finished."
Inform the user when the script is done. (Not strictly necessary, since you'll see the command prompt anyway, but it doesn't hurt.)
add a comment |Â
up vote
0
down vote
up vote
0
down vote
Using FNR
and NR
in awk
#!/bin/bash
outfile="$( mktemp combined.txt.XXXXXX )"
echo "Output file: $outfile"
awk 'FNR==1 && NR>1 printf("n%snn","========") 1' *.txt > "$outfile"
echo "Finished."
A line-by-line description:
outfile="$( mktemp combined.txt.XXXXXX )"
Use mktemp
to create an empty new file with a unique name (eg, combined.txt.HDpgMn
). You can use more X
characters for a longer random suffix. Enclose the command in "$(
...)"
to store the new file's name in the variable outfile
.
echo "Saving to file: $outfile"
Print the name of the output file. (When the script has finished, you may wish to rename the output file to remove the string of random characters following the .txt
.)
awk 'FNR==1 && NR>1 printf("n%snn","========") 1' *.txt > "$outfile"
Print...
- a blank line,
- a short line of "=" characters,
- and another blank line
...at the start of each input file, except for the first input file. FNR
counts the input file line numbers, resetting at the start of each file. NR
counts the line numbers and does not reset.
In the awk
statement, the 1
just before the closing single quotation mark evaluates to TRUE
for every line, and performs the default action of printing that line. (In other words, awk '1'
works like cat
.)
echo "Finished."
Inform the user when the script is done. (Not strictly necessary, since you'll see the command prompt anyway, but it doesn't hurt.)
Using FNR
and NR
in awk
#!/bin/bash
outfile="$( mktemp combined.txt.XXXXXX )"
echo "Output file: $outfile"
awk 'FNR==1 && NR>1 printf("n%snn","========") 1' *.txt > "$outfile"
echo "Finished."
A line-by-line description:
outfile="$( mktemp combined.txt.XXXXXX )"
Use mktemp
to create an empty new file with a unique name (eg, combined.txt.HDpgMn
). You can use more X
characters for a longer random suffix. Enclose the command in "$(
...)"
to store the new file's name in the variable outfile
.
echo "Saving to file: $outfile"
Print the name of the output file. (When the script has finished, you may wish to rename the output file to remove the string of random characters following the .txt
.)
awk 'FNR==1 && NR>1 printf("n%snn","========") 1' *.txt > "$outfile"
Print...
- a blank line,
- a short line of "=" characters,
- and another blank line
...at the start of each input file, except for the first input file. FNR
counts the input file line numbers, resetting at the start of each file. NR
counts the line numbers and does not reset.
In the awk
statement, the 1
just before the closing single quotation mark evaluates to TRUE
for every line, and performs the default action of printing that line. (In other words, awk '1'
works like cat
.)
echo "Finished."
Inform the user when the script is done. (Not strictly necessary, since you'll see the command prompt anyway, but it doesn't hurt.)
answered 9 mins ago
Gaultheria
3404
3404
add a comment |Â
add a comment |Â
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2funix.stackexchange.com%2fquestions%2f473849%2fmerging-text-files-and-adding-separator%23new-answer', 'question_page');
);
Post as a guest
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
See Concatenate multiple files with two blank lines as delimiter? and the linked questions/answers.
â don_crissti
2 hours ago