up vote
3
down vote

favorite

I want to add a separator like this "==============" and a blank new line

I tried to do this, but failed and causes high CPU usage. i mean the cpu become rotate ery fast and noisy when i run the script

This needs to be for around 100000 text files.

this is the code that I use

#!/bin/bash
for F in *.txt ; do
 type "$F"
 echo .
 echo ========
 echo . 
done >> Combined.txt;

please advice

edited 1 hour ago

Jeff Schaller

33.6k851113

asked 2 hours ago

Shervan

32010

See Concatenate multiple files with two blank lines as delimiter? and the linked questions/answers.
â€“Â don_crissti
2 hours ago

add a commentÂ |Â

up vote
3
down vote

favorite

I want to add a separator like this "==============" and a blank new line

I tried to do this, but failed and causes high CPU usage. i mean the cpu become rotate ery fast and noisy when i run the script

This needs to be for around 100000 text files.

this is the code that I use

#!/bin/bash
for F in *.txt ; do
 type "$F"
 echo .
 echo ========
 echo . 
done >> Combined.txt;

please advice

edited 1 hour ago

Jeff Schaller

33.6k851113

asked 2 hours ago

Shervan

32010

See Concatenate multiple files with two blank lines as delimiter? and the linked questions/answers.
â€“Â don_crissti
2 hours ago

add a commentÂ |Â

up vote
3
down vote

favorite

I want to add a separator like this "==============" and a blank new line

I tried to do this, but failed and causes high CPU usage. i mean the cpu become rotate ery fast and noisy when i run the script

This needs to be for around 100000 text files.

this is the code that I use

#!/bin/bash
for F in *.txt ; do
 type "$F"
 echo .
 echo ========
 echo . 
done >> Combined.txt;

please advice

edited 1 hour ago

Jeff Schaller

33.6k851113

asked 2 hours ago

Shervan

32010

I want to add a separator like this "==============" and a blank new line

I tried to do this, but failed and causes high CPU usage. i mean the cpu become rotate ery fast and noisy when i run the script

This needs to be for around 100000 text files.

this is the code that I use

#!/bin/bash
for F in *.txt ; do
 type "$F"
 echo .
 echo ========
 echo . 
done >> Combined.txt;

please advice

text-processing scripting

edited 1 hour ago

Jeff Schaller

33.6k851113

asked 2 hours ago

Shervan

32010

edited 1 hour ago

Jeff Schaller

33.6k851113

asked 2 hours ago

Shervan

32010

edited 1 hour ago

Jeff Schaller

33.6k851113

edited 1 hour ago

Jeff Schaller

33.6k851113

edited 1 hour ago

Jeff Schaller

33.6k851113

asked 2 hours ago

Shervan

32010

asked 2 hours ago

Shervan

32010

asked 2 hours ago

Shervan

32010

See Concatenate multiple files with two blank lines as delimiter? and the linked questions/answers.
â€“Â don_crissti
2 hours ago

add a commentÂ |Â

See Concatenate multiple files with two blank lines as delimiter? and the linked questions/answers.
â€“Â don_crissti
2 hours ago

See Concatenate multiple files with two blank lines as delimiter? and the linked questions/answers.
â€“Â don_crissti
2 hours ago

add a commentÂ |Â

4 Answers
4

active

oldest

votes

up vote
1
down vote

accepted

I would simplify your commands as follows:

 #!/bin/bash
 for file in *.txt; do
 cat $file >> Combined.txt
 printf 'nn=========nn' >> Combined.txt
 done

answered 2 hours ago

Goro

7,20753168

add a commentÂ |Â

up vote
0
down vote

I edited your question to make the code readable... but also fixed some things at the same time;

'do' needs a matching 'done'

& means branch while && means and

% is cmd $ is bash

() was a needless subshell

one only need one write out

use shellcheck.net

edited 2 hours ago

answered 2 hours ago

user1133275

2,297412

1

Ideally you'd leave OP's code intact and post any modifications/fixes as part of your answer. That way other people can see the original code/attempt without having to scroll through edit revisions...
â€“Â don_crissti
2 hours ago

add a commentÂ |Â

up vote
0
down vote

If you're going to do it for thousands of files, you may want to avoid running several commands per file. With GNU awk:

printf '%s' ./*.txt | xargs -r0 gawk '
 BEGINFILE if (NR) print "n==========n";1' > combined.out

Don't give a .txt extension to the output file if you're going to put it in the same directory, or it's going to be selected as an input file and causing an infinite loop (probably your problem in the first place).

Or use a shell where cat is builtin like ksh93:

#! /bin/ksh93
firstpass=true
for file in *.txt; do
 "$firstpass" || print 'n===========n'
 firstpass=false
 command /opt/ast/bin/cat < "$file"
done > combined.out

All those commands in the loop are built-in, so running them doesn't involve forking new processes nor loading external executable, so that would make the performance tolerable.

edited 1 hour ago

answered 1 hour ago

StÃ©phane Chazelas

287k53531868

add a commentÂ |Â

up vote
0
down vote

Using `FNR` and `NR` in `awk`

#!/bin/bash

outfile="$( mktemp combined.txt.XXXXXX )"

echo "Output file: $outfile"

awk 'FNR==1 && NR>1 printf("n%snn","========") 1' *.txt > "$outfile"

echo "Finished."

A line-by-line description:

outfile="$( mktemp combined.txt.XXXXXX )"

Use mktemp to create an empty new file with a unique name (eg, combined.txt.HDpgMn). You can use more X characters for a longer random suffix. Enclose the command in "$(...)" to store the new file's name in the variable outfile.

echo "Saving to file: $outfile"

Print the name of the output file. (When the script has finished, you may wish to rename the output file to remove the string of random characters following the .txt.)

awk 'FNR==1 && NR>1 printf("n%snn","========") 1' *.txt > "$outfile"

Print...

a blank line,

a short line of "=" characters,

and another blank line

...at the start of each input file, except for the first input file. FNR counts the input file line numbers, resetting at the start of each file. NR counts the line numbers and does not reset.

In the awk statement, the 1 just before the closing single quotation mark evaluates to TRUE for every line, and performs the default action of printing that line. (In other words, awk '1' works like cat.)

echo "Finished."

Inform the user when the script is done. (Not strictly necessary, since you'll see the command prompt anyway, but it doesn't hurt.)

answered 9 mins ago

Gaultheria

3404

add a commentÂ |Â

Your Answer

StackExchange.ready(function()
var channelOptions =
tags: "".split(" "),
id: "106"
;
initTagRenderer("".split(" "), "".split(" "), channelOptions);

StackExchange.using("externalEditor", function()
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled)
StackExchange.using("snippets", function()
createEditor();
);

else
createEditor();

);

function createEditor()
StackExchange.prepareEditor(
heartbeatType: 'answer',
convertImagesToLinks: false,
noModals: false,
showLowRepImageUploadWarning: true,
reputationToPostImages: null,
bindNavPrevention: true,
postfix: "",
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
);

);

draft saved

draft discarded

StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2funix.stackexchange.com%2fquestions%2f473849%2fmerging-text-files-and-adding-separator%23new-answer', 'question_page');

);

Post as a guest

Name

4 Answers
4

active

oldest

votes

4 Answers
4

active

oldest

votes

up vote
1
down vote

accepted

I would simplify your commands as follows:

 #!/bin/bash
 for file in *.txt; do
 cat $file >> Combined.txt
 printf 'nn=========nn' >> Combined.txt
 done

answered 2 hours ago

Goro

7,20753168

add a commentÂ |Â

up vote
1
down vote

accepted

I would simplify your commands as follows:

 #!/bin/bash
 for file in *.txt; do
 cat $file >> Combined.txt
 printf 'nn=========nn' >> Combined.txt
 done

answered 2 hours ago

Goro

7,20753168

add a commentÂ |Â

up vote
1
down vote

accepted

I would simplify your commands as follows:

 #!/bin/bash
 for file in *.txt; do
 cat $file >> Combined.txt
 printf 'nn=========nn' >> Combined.txt
 done

answered 2 hours ago

Goro

7,20753168

I would simplify your commands as follows:

 #!/bin/bash
 for file in *.txt; do
 cat $file >> Combined.txt
 printf 'nn=========nn' >> Combined.txt
 done

answered 2 hours ago

Goro

7,20753168

answered 2 hours ago

Goro

7,20753168

answered 2 hours ago

Goro

7,20753168

answered 2 hours ago

Goro

7,20753168

add a commentÂ |Â

up vote
0
down vote

I edited your question to make the code readable... but also fixed some things at the same time;

'do' needs a matching 'done'

& means branch while && means and

% is cmd $ is bash

() was a needless subshell

one only need one write out

use shellcheck.net

edited 2 hours ago

answered 2 hours ago

user1133275

2,297412

1

Ideally you'd leave OP's code intact and post any modifications/fixes as part of your answer. That way other people can see the original code/attempt without having to scroll through edit revisions...
â€“Â don_crissti
2 hours ago

add a commentÂ |Â

up vote
0
down vote

I edited your question to make the code readable... but also fixed some things at the same time;

'do' needs a matching 'done'

& means branch while && means and

% is cmd $ is bash

() was a needless subshell

one only need one write out

use shellcheck.net

edited 2 hours ago

answered 2 hours ago

user1133275

2,297412

1

Ideally you'd leave OP's code intact and post any modifications/fixes as part of your answer. That way other people can see the original code/attempt without having to scroll through edit revisions...
â€“Â don_crissti
2 hours ago

add a commentÂ |Â

up vote
0
down vote

I edited your question to make the code readable... but also fixed some things at the same time;

'do' needs a matching 'done'

& means branch while && means and

% is cmd $ is bash

() was a needless subshell

one only need one write out

use shellcheck.net

edited 2 hours ago

answered 2 hours ago

user1133275

2,297412

I edited your question to make the code readable... but also fixed some things at the same time;

'do' needs a matching 'done'

& means branch while && means and

% is cmd $ is bash

() was a needless subshell

one only need one write out

use shellcheck.net

edited 2 hours ago

answered 2 hours ago

user1133275

2,297412

edited 2 hours ago

answered 2 hours ago

user1133275

2,297412

answered 2 hours ago

user1133275

2,297412

answered 2 hours ago

user1133275

2,297412

1

Ideally you'd leave OP's code intact and post any modifications/fixes as part of your answer. That way other people can see the original code/attempt without having to scroll through edit revisions...
â€“Â don_crissti
2 hours ago

add a commentÂ |Â

1

Ideally you'd leave OP's code intact and post any modifications/fixes as part of your answer. That way other people can see the original code/attempt without having to scroll through edit revisions...
â€“Â don_crissti
2 hours ago

Ideally you'd leave OP's code intact and post any modifications/fixes as part of your answer. That way other people can see the original code/attempt without having to scroll through edit revisions...
â€“Â don_crissti
2 hours ago

add a commentÂ |Â

up vote
0
down vote

If you're going to do it for thousands of files, you may want to avoid running several commands per file. With GNU awk:

printf '%s' ./*.txt | xargs -r0 gawk '
 BEGINFILE if (NR) print "n==========n";1' > combined.out

Or use a shell where cat is builtin like ksh93:

#! /bin/ksh93
firstpass=true
for file in *.txt; do
 "$firstpass" || print 'n===========n'
 firstpass=false
 command /opt/ast/bin/cat < "$file"
done > combined.out

All those commands in the loop are built-in, so running them doesn't involve forking new processes nor loading external executable, so that would make the performance tolerable.

edited 1 hour ago

answered 1 hour ago

StÃ©phane Chazelas

287k53531868

add a commentÂ |Â

up vote
0
down vote

If you're going to do it for thousands of files, you may want to avoid running several commands per file. With GNU awk:

printf '%s' ./*.txt | xargs -r0 gawk '
 BEGINFILE if (NR) print "n==========n";1' > combined.out

Or use a shell where cat is builtin like ksh93:

#! /bin/ksh93
firstpass=true
for file in *.txt; do
 "$firstpass" || print 'n===========n'
 firstpass=false
 command /opt/ast/bin/cat < "$file"
done > combined.out

All those commands in the loop are built-in, so running them doesn't involve forking new processes nor loading external executable, so that would make the performance tolerable.

edited 1 hour ago

answered 1 hour ago

StÃ©phane Chazelas

287k53531868

add a commentÂ |Â

up vote
0
down vote

If you're going to do it for thousands of files, you may want to avoid running several commands per file. With GNU awk:

printf '%s' ./*.txt | xargs -r0 gawk '
 BEGINFILE if (NR) print "n==========n";1' > combined.out

Or use a shell where cat is builtin like ksh93:

#! /bin/ksh93
firstpass=true
for file in *.txt; do
 "$firstpass" || print 'n===========n'
 firstpass=false
 command /opt/ast/bin/cat < "$file"
done > combined.out

All those commands in the loop are built-in, so running them doesn't involve forking new processes nor loading external executable, so that would make the performance tolerable.

edited 1 hour ago

answered 1 hour ago

287k53531868

If you're going to do it for thousands of files, you may want to avoid running several commands per file. With GNU awk:

printf '%s' ./*.txt | xargs -r0 gawk '
 BEGINFILE if (NR) print "n==========n";1' > combined.out

Or use a shell where cat is builtin like ksh93:

#! /bin/ksh93
firstpass=true
for file in *.txt; do
 "$firstpass" || print 'n===========n'
 firstpass=false
 command /opt/ast/bin/cat < "$file"
done > combined.out

All those commands in the loop are built-in, so running them doesn't involve forking new processes nor loading external executable, so that would make the performance tolerable.

edited 1 hour ago

answered 1 hour ago

287k53531868

edited 1 hour ago

answered 1 hour ago

287k53531868

answered 1 hour ago

287k53531868

answered 1 hour ago

287k53531868

add a commentÂ |Â

up vote
0
down vote

Using `FNR` and `NR` in `awk`

#!/bin/bash

outfile="$( mktemp combined.txt.XXXXXX )"

echo "Output file: $outfile"

awk 'FNR==1 && NR>1 printf("n%snn","========") 1' *.txt > "$outfile"

echo "Finished."

A line-by-line description:

outfile="$( mktemp combined.txt.XXXXXX )"

echo "Saving to file: $outfile"

Print the name of the output file. (When the script has finished, you may wish to rename the output file to remove the string of random characters following the .txt.)

awk 'FNR==1 && NR>1 printf("n%snn","========") 1' *.txt > "$outfile"

Print...

a blank line,

a short line of "=" characters,

and another blank line

...at the start of each input file, except for the first input file. FNR counts the input file line numbers, resetting at the start of each file. NR counts the line numbers and does not reset.

echo "Finished."

Inform the user when the script is done. (Not strictly necessary, since you'll see the command prompt anyway, but it doesn't hurt.)

answered 9 mins ago

Gaultheria

3404

add a commentÂ |Â

up vote
0
down vote

Using `FNR` and `NR` in `awk`

#!/bin/bash

outfile="$( mktemp combined.txt.XXXXXX )"

echo "Output file: $outfile"

awk 'FNR==1 && NR>1 printf("n%snn","========") 1' *.txt > "$outfile"

echo "Finished."

A line-by-line description:

outfile="$( mktemp combined.txt.XXXXXX )"

echo "Saving to file: $outfile"

Print the name of the output file. (When the script has finished, you may wish to rename the output file to remove the string of random characters following the .txt.)

awk 'FNR==1 && NR>1 printf("n%snn","========") 1' *.txt > "$outfile"

Print...

a blank line,

a short line of "=" characters,

and another blank line

...at the start of each input file, except for the first input file. FNR counts the input file line numbers, resetting at the start of each file. NR counts the line numbers and does not reset.

echo "Finished."

Inform the user when the script is done. (Not strictly necessary, since you'll see the command prompt anyway, but it doesn't hurt.)

answered 9 mins ago

Gaultheria

3404

add a commentÂ |Â

up vote
0
down vote

Using `FNR` and `NR` in `awk`

#!/bin/bash

outfile="$( mktemp combined.txt.XXXXXX )"

echo "Output file: $outfile"

awk 'FNR==1 && NR>1 printf("n%snn","========") 1' *.txt > "$outfile"

echo "Finished."

A line-by-line description:

outfile="$( mktemp combined.txt.XXXXXX )"

echo "Saving to file: $outfile"

Print the name of the output file. (When the script has finished, you may wish to rename the output file to remove the string of random characters following the .txt.)

awk 'FNR==1 && NR>1 printf("n%snn","========") 1' *.txt > "$outfile"

Print...

a blank line,

a short line of "=" characters,

and another blank line

...at the start of each input file, except for the first input file. FNR counts the input file line numbers, resetting at the start of each file. NR counts the line numbers and does not reset.

echo "Finished."

Inform the user when the script is done. (Not strictly necessary, since you'll see the command prompt anyway, but it doesn't hurt.)

answered 9 mins ago

Gaultheria

3404

Using `FNR` and `NR` in `awk`

#!/bin/bash

outfile="$( mktemp combined.txt.XXXXXX )"

echo "Output file: $outfile"

awk 'FNR==1 && NR>1 printf("n%snn","========") 1' *.txt > "$outfile"

echo "Finished."

A line-by-line description:

outfile="$( mktemp combined.txt.XXXXXX )"

echo "Saving to file: $outfile"

Print the name of the output file. (When the script has finished, you may wish to rename the output file to remove the string of random characters following the .txt.)

awk 'FNR==1 && NR>1 printf("n%snn","========") 1' *.txt > "$outfile"

Print...

a blank line,

a short line of "=" characters,

and another blank line

...at the start of each input file, except for the first input file. FNR counts the input file line numbers, resetting at the start of each file. NR counts the line numbers and does not reset.

echo "Finished."

Inform the user when the script is done. (Not strictly necessary, since you'll see the command prompt anyway, but it doesn't hurt.)

answered 9 mins ago

Gaultheria

3404

answered 9 mins ago

Gaultheria

3404

answered 9 mins ago

Gaultheria

3404

answered 9 mins ago

Gaultheria

3404

add a commentÂ |Â

draft saved

draft discarded

draft saved

draft discarded

Post as a guest

Name

Search This Blog

Iyfjky

Merging text files and adding separator

4 Answers
4

Using `FNR` and `NR` in `awk`

Your Answer

Post as a guest

4 Answers
4

4 Answers
4

Using `FNR` and `NR` in `awk`

Using `FNR` and `NR` in `awk`

Using `FNR` and `NR` in `awk`

Using `FNR` and `NR` in `awk`

Post as a guest

Comments

Post a Comment

Popular posts from this blog

Long meetings (6-7 hours a day): Being â€œbabysatâ€ by supervisor

Is the Concept of Multiple Fantasy Races Scientifically Flawed? [closed]

Confectionery

Category

Random preview

Merging text files and adding separator

4 Answers 4

Using FNR and NR in awk

Your Answer

Sign up or log in

Post as a guest

Post as a guest

4 Answers 4

4 Answers 4

Using FNR and NR in awk

Using FNR and NR in awk

Using FNR and NR in awk

Using FNR and NR in awk

Sign up or log in

Post as a guest

Post as a guest

Sign up or log in

Post as a guest

Sign up or log in

Post as a guest

Sign up or log in

Post as a guest

Comments

Post a Comment

Popular posts from this blog

Long meetings (6-7 hours a day): Being â€œbabysatâ€ by supervisor

Is the Concept of Multiple Fantasy Races Scientifically Flawed? [closed]

Confectionery

4 Answers
4

Using `FNR` and `NR` in `awk`

4 Answers
4

4 Answers
4

Using `FNR` and `NR` in `awk`

Using `FNR` and `NR` in `awk`

Using `FNR` and `NR` in `awk`

Using `FNR` and `NR` in `awk`

Long meetings (6-7 hours a day): Being â€œbabysatâ€ by supervisor