Loop over lines in file and subtract previous line from current line

up vote
3
down vote

favorite

I have a file that contains some numbers

$ cat file.dat
0.092593
0.048631
0.027957
0.030699
0.026250
0.038156
0.011823
0.013284
0.024529
0.022498
0.013217
0.007105
0.018916
0.014079

I want to make a new file that contains the difference of the current line with the previous line. Expected output should be

$ cat newfile.dat
-0.043962
-0.020674
0.002742
-0.004449
0.011906
-0.026333
0.001461
0.011245
-0.002031
-0.009281
-0.006112
0.011811
-0.004837

Thinking this was trivial, I started with this piece of code

f="myfile.dat" 
while read line; do
 curr=$line
 prev=

 bc <<< "$line - $prev" >> newfile.dat
done < $f

but I realized quickly that I have no idea how to access the previous line in the file. I guess I also need to account for that no subtraction should take place when reading the first line. Any guidance on how to proceed is appreciated!

asked 28 mins ago

Yoda

14816

add a commentÂ |Â

up vote
3
down vote

favorite

I have a file that contains some numbers

$ cat file.dat
0.092593
0.048631
0.027957
0.030699
0.026250
0.038156
0.011823
0.013284
0.024529
0.022498
0.013217
0.007105
0.018916
0.014079

I want to make a new file that contains the difference of the current line with the previous line. Expected output should be

$ cat newfile.dat
-0.043962
-0.020674
0.002742
-0.004449
0.011906
-0.026333
0.001461
0.011245
-0.002031
-0.009281
-0.006112
0.011811
-0.004837

Thinking this was trivial, I started with this piece of code

f="myfile.dat" 
while read line; do
 curr=$line
 prev=

 bc <<< "$line - $prev" >> newfile.dat
done < $f

asked 28 mins ago

Yoda

14816

add a commentÂ |Â

up vote
3
down vote

favorite

I have a file that contains some numbers

$ cat file.dat
0.092593
0.048631
0.027957
0.030699
0.026250
0.038156
0.011823
0.013284
0.024529
0.022498
0.013217
0.007105
0.018916
0.014079

I want to make a new file that contains the difference of the current line with the previous line. Expected output should be

$ cat newfile.dat
-0.043962
-0.020674
0.002742
-0.004449
0.011906
-0.026333
0.001461
0.011245
-0.002031
-0.009281
-0.006112
0.011811
-0.004837

Thinking this was trivial, I started with this piece of code

f="myfile.dat" 
while read line; do
 curr=$line
 prev=

 bc <<< "$line - $prev" >> newfile.dat
done < $f

asked 28 mins ago

Yoda

14816

I have a file that contains some numbers

$ cat file.dat
0.092593
0.048631
0.027957
0.030699
0.026250
0.038156
0.011823
0.013284
0.024529
0.022498
0.013217
0.007105
0.018916
0.014079

I want to make a new file that contains the difference of the current line with the previous line. Expected output should be

$ cat newfile.dat
-0.043962
-0.020674
0.002742
-0.004449
0.011906
-0.026333
0.001461
0.011245
-0.002031
-0.009281
-0.006112
0.011811
-0.004837

Thinking this was trivial, I started with this piece of code

f="myfile.dat" 
while read line; do
 curr=$line
 prev=

 bc <<< "$line - $prev" >> newfile.dat
done < $f

bash shell-script scripting

asked 28 mins ago

Yoda

14816

asked 28 mins ago

Yoda

14816

asked 28 mins ago

Yoda

14816

asked 28 mins ago

Yoda

14816

asked 28 mins ago

Yoda

14816

add a commentÂ |Â

3 Answers
3

active

oldest

votes

up vote
6
down vote

$ awk 'NR > 1 print $0 - prev prev = $0 ' <file.dat
-0.043962
-0.020674
0.002742
-0.004449
0.011906
-0.026333
0.001461
0.011245
-0.002031
-0.009281
-0.006112
0.011811
-0.004837

Doing this in a shell loop calling bc is cumbersome. The above uses a simple awk script that reads the values off of the file one by one and for any line past the first one, it prints the difference as you describe.

The first block, NR > 1 print $0 - prev , conditionally prints the difference between this and the previous line if we've reached line two or further (NR is the number of records read so far, and a "record" is by default a line).

The second block, prev = $0 , unconditionally sets prev to the value on the current line.

Redirect the output to newfile.dat to save the result there:

$ awk 'NR > 1 print $0 - prev prev = $0 ' <file.dat >newfile.dat

Why is using a shell loop to process text considered bad practice?

edited 7 mins ago

answered 23 mins ago

Kusalananda

106k14209328

If prev is the value on the current line, how does this actually work? Because if I substitute prev = $0 into the first block, we get print $0 - $0.
â€“Â Yoda
13 mins ago

@Yoda prev is the value on the previous line. For the first line, the first block is not executed (due to the NR > 1 condition), so prev only gets the value of the first line (and nothing else happens). For the second line, the difference between the second and first line is printed before prev is set to the value of the second line. The blocks are executed in order, and for each line.
â€“Â Kusalananda
12 mins ago

add a commentÂ |Â

up vote
2
down vote

If you wanted to try and force the shell script into working, you were just missing some initialization:

f=myfile.dat
prev=0
while read line; do
 bc <<< "$line - $prev"
 prev=$line
done < $f > newfile.dat

... where I also moved the redirection outside of the loop, just to save some I/O.

The bc solution does not print leading zeroes, while the awk solution does.

answered 14 mins ago

Jeff Schaller

32.6k849110

add a commentÂ |Â

up vote
2
down vote

You could use an exec redirection to read successive lines of the input file from multiple points in the script - once before the loop (to set up the initial value), then repeatedly during it (for each new value to subtract):

exec 3<file.dat
read prev<&3
while read curr ; do
 bc <<< "$curr - $prev" >> newfile.dat
 prev=$curr
done <&3

answered 10 mins ago

JigglyNaga

2,651623

add a commentÂ |Â

Your Answer

StackExchange.ready(function()
var channelOptions =
tags: "".split(" "),
id: "106"
;
initTagRenderer("".split(" "), "".split(" "), channelOptions);

StackExchange.using("externalEditor", function()
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled)
StackExchange.using("snippets", function()
createEditor();
);

else
createEditor();

);

function createEditor()
StackExchange.prepareEditor(
heartbeatType: 'answer',
convertImagesToLinks: false,
noModals: false,
showLowRepImageUploadWarning: true,
reputationToPostImages: null,
bindNavPrevention: true,
postfix: "",
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
);

);

draft saved

draft discarded

StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2funix.stackexchange.com%2fquestions%2f470031%2floop-over-lines-in-file-and-subtract-previous-line-from-current-line%23new-answer', 'question_page');

);

Post as a guest

Name

3 Answers
3

active

oldest

votes

3 Answers
3

active

oldest

votes

up vote
6
down vote

$ awk 'NR > 1 print $0 - prev prev = $0 ' <file.dat
-0.043962
-0.020674
0.002742
-0.004449
0.011906
-0.026333
0.001461
0.011245
-0.002031
-0.009281
-0.006112
0.011811
-0.004837

The second block, prev = $0 , unconditionally sets prev to the value on the current line.

Redirect the output to newfile.dat to save the result there:

$ awk 'NR > 1 print $0 - prev prev = $0 ' <file.dat >newfile.dat

Why is using a shell loop to process text considered bad practice?

edited 7 mins ago

answered 23 mins ago

Kusalananda

106k14209328

If prev is the value on the current line, how does this actually work? Because if I substitute prev = $0 into the first block, we get print $0 - $0.
â€“Â Yoda
13 mins ago

@Yoda prev is the value on the previous line. For the first line, the first block is not executed (due to the NR > 1 condition), so prev only gets the value of the first line (and nothing else happens). For the second line, the difference between the second and first line is printed before prev is set to the value of the second line. The blocks are executed in order, and for each line.
â€“Â Kusalananda
12 mins ago

add a commentÂ |Â

up vote
6
down vote

$ awk 'NR > 1 print $0 - prev prev = $0 ' <file.dat
-0.043962
-0.020674
0.002742
-0.004449
0.011906
-0.026333
0.001461
0.011245
-0.002031
-0.009281
-0.006112
0.011811
-0.004837

The second block, prev = $0 , unconditionally sets prev to the value on the current line.

Redirect the output to newfile.dat to save the result there:

$ awk 'NR > 1 print $0 - prev prev = $0 ' <file.dat >newfile.dat

Why is using a shell loop to process text considered bad practice?

edited 7 mins ago

answered 23 mins ago

Kusalananda

106k14209328

If prev is the value on the current line, how does this actually work? Because if I substitute prev = $0 into the first block, we get print $0 - $0.
â€“Â Yoda
13 mins ago

@Yoda prev is the value on the previous line. For the first line, the first block is not executed (due to the NR > 1 condition), so prev only gets the value of the first line (and nothing else happens). For the second line, the difference between the second and first line is printed before prev is set to the value of the second line. The blocks are executed in order, and for each line.
â€“Â Kusalananda
12 mins ago

add a commentÂ |Â

up vote
6
down vote

$ awk 'NR > 1 print $0 - prev prev = $0 ' <file.dat
-0.043962
-0.020674
0.002742
-0.004449
0.011906
-0.026333
0.001461
0.011245
-0.002031
-0.009281
-0.006112
0.011811
-0.004837

The second block, prev = $0 , unconditionally sets prev to the value on the current line.

Redirect the output to newfile.dat to save the result there:

$ awk 'NR > 1 print $0 - prev prev = $0 ' <file.dat >newfile.dat

Why is using a shell loop to process text considered bad practice?

edited 7 mins ago

answered 23 mins ago

Kusalananda

106k14209328

$ awk 'NR > 1 print $0 - prev prev = $0 ' <file.dat
-0.043962
-0.020674
0.002742
-0.004449
0.011906
-0.026333
0.001461
0.011245
-0.002031
-0.009281
-0.006112
0.011811
-0.004837

The second block, prev = $0 , unconditionally sets prev to the value on the current line.

Redirect the output to newfile.dat to save the result there:

$ awk 'NR > 1 print $0 - prev prev = $0 ' <file.dat >newfile.dat

Why is using a shell loop to process text considered bad practice?

edited 7 mins ago

answered 23 mins ago

Kusalananda

106k14209328

edited 7 mins ago

answered 23 mins ago

Kusalananda

106k14209328

answered 23 mins ago

Kusalananda

106k14209328

answered 23 mins ago

Kusalananda

106k14209328

If prev is the value on the current line, how does this actually work? Because if I substitute prev = $0 into the first block, we get print $0 - $0.
â€“Â Yoda
13 mins ago

@Yoda prev is the value on the previous line. For the first line, the first block is not executed (due to the NR > 1 condition), so prev only gets the value of the first line (and nothing else happens). For the second line, the difference between the second and first line is printed before prev is set to the value of the second line. The blocks are executed in order, and for each line.
â€“Â Kusalananda
12 mins ago

add a commentÂ |Â

If prev is the value on the current line, how does this actually work? Because if I substitute prev = $0 into the first block, we get print $0 - $0.
â€“Â Yoda
13 mins ago

@Yoda prev is the value on the previous line. For the first line, the first block is not executed (due to the NR > 1 condition), so prev only gets the value of the first line (and nothing else happens). For the second line, the difference between the second and first line is printed before prev is set to the value of the second line. The blocks are executed in order, and for each line.
â€“Â Kusalananda
12 mins ago

If prev is the value on the current line, how does this actually work? Because if I substitute prev = $0 into the first block, we get print $0 - $0.
â€“Â Yoda
13 mins ago

@Yoda prev is the value on the previous line. For the first line, the first block is not executed (due to the NR > 1 condition), so prev only gets the value of the first line (and nothing else happens). For the second line, the difference between the second and first line is printed before prev is set to the value of the second line. The blocks are executed in order, and for each line.
â€“Â Kusalananda
12 mins ago

add a commentÂ |Â

up vote
2
down vote

If you wanted to try and force the shell script into working, you were just missing some initialization:

f=myfile.dat
prev=0
while read line; do
 bc <<< "$line - $prev"
 prev=$line
done < $f > newfile.dat

... where I also moved the redirection outside of the loop, just to save some I/O.

The bc solution does not print leading zeroes, while the awk solution does.

answered 14 mins ago

Jeff Schaller

32.6k849110

add a commentÂ |Â

up vote
2
down vote

If you wanted to try and force the shell script into working, you were just missing some initialization:

f=myfile.dat
prev=0
while read line; do
 bc <<< "$line - $prev"
 prev=$line
done < $f > newfile.dat

... where I also moved the redirection outside of the loop, just to save some I/O.

The bc solution does not print leading zeroes, while the awk solution does.

answered 14 mins ago

Jeff Schaller

32.6k849110

add a commentÂ |Â

up vote
2
down vote

If you wanted to try and force the shell script into working, you were just missing some initialization:

f=myfile.dat
prev=0
while read line; do
 bc <<< "$line - $prev"
 prev=$line
done < $f > newfile.dat

... where I also moved the redirection outside of the loop, just to save some I/O.

The bc solution does not print leading zeroes, while the awk solution does.

answered 14 mins ago

Jeff Schaller

32.6k849110

If you wanted to try and force the shell script into working, you were just missing some initialization:

f=myfile.dat
prev=0
while read line; do
 bc <<< "$line - $prev"
 prev=$line
done < $f > newfile.dat

... where I also moved the redirection outside of the loop, just to save some I/O.

The bc solution does not print leading zeroes, while the awk solution does.

answered 14 mins ago

Jeff Schaller

32.6k849110

answered 14 mins ago

Jeff Schaller

32.6k849110

answered 14 mins ago

Jeff Schaller

32.6k849110

answered 14 mins ago

Jeff Schaller

32.6k849110

add a commentÂ |Â

up vote
2
down vote

exec 3<file.dat
read prev<&3
while read curr ; do
 bc <<< "$curr - $prev" >> newfile.dat
 prev=$curr
done <&3

answered 10 mins ago

JigglyNaga

2,651623

add a commentÂ |Â

up vote
2
down vote

exec 3<file.dat
read prev<&3
while read curr ; do
 bc <<< "$curr - $prev" >> newfile.dat
 prev=$curr
done <&3

answered 10 mins ago

JigglyNaga

2,651623

add a commentÂ |Â

up vote
2
down vote

exec 3<file.dat
read prev<&3
while read curr ; do
 bc <<< "$curr - $prev" >> newfile.dat
 prev=$curr
done <&3

answered 10 mins ago

JigglyNaga

2,651623

exec 3<file.dat
read prev<&3
while read curr ; do
 bc <<< "$curr - $prev" >> newfile.dat
 prev=$curr
done <&3

answered 10 mins ago

JigglyNaga

2,651623

answered 10 mins ago

JigglyNaga

2,651623

answered 10 mins ago

JigglyNaga

2,651623

answered 10 mins ago

JigglyNaga

2,651623

add a commentÂ |Â

draft saved

draft discarded

draft saved

draft discarded

Post as a guest

Name

Search This Blog

Iyfjky