Rename a subset of files in a directory

up vote
4
down vote

favorite

I have many hundreds of thousands of files in a directory. These files are named as follows:

left-00001.tiff
left-00002.tiff
...
left-99999.tiff
left-100000.tiff
...
left-245000.tiff

I would like to rename the files as follows:

left-000001.tiff
...
left-099999.tiff
...
left-245000.tiff

I found elegant solution to this problem here.

This solution implements a bash script called zeropad.sh. the bash is coded as follows:

#!/bin/bash
num=`expr match "$1" '[^0-9]*([0-9]+).*'`
paddednum=`printf "%06d" $num`
echo $1/$num/$paddednum

and can be applied iteratively using for loop as follows:

for i in *.tiff;do mv $i `./zeropad.sh $i`; done

However, this solution takes a very long time because it does much unnecessary work renaming all the files which are already properly padded. i.e. as %06d type numbers. For my own purposes this is solution is very slow.

I have two questions:

1- How can I modify the iterator to only apply zeropad.sh on files which need to be zero padded?

2- How can I use the command touch in a for loop to generate test data? It is crucial to verify that this script works before I apply it on the original data.

edited Sep 5 at 23:16

asked Sep 5 at 18:20

kevinkayaks

1295

New contributor

It is slow because you are calling your shell script for every file. It would be quicker if you did the looping and renaming in the same script.
â€“Â Kusalananda
Sep 5 at 19:19

2

Posting as comment because question asks for bash, but I think Python is a better fit: from pathlib import Path; for f in Path().glob('left-?????.tiff'): f.rename(f.name.replace('-', '-0')) Note that touch left-00001..99999.tiff can be used to quickly generate files.
â€“Â BoppreH
Sep 5 at 21:43

add a commentÂ |Â

up vote
4
down vote

favorite

I have many hundreds of thousands of files in a directory. These files are named as follows:

left-00001.tiff
left-00002.tiff
...
left-99999.tiff
left-100000.tiff
...
left-245000.tiff

I would like to rename the files as follows:

left-000001.tiff
...
left-099999.tiff
...
left-245000.tiff

I found elegant solution to this problem here.

This solution implements a bash script called zeropad.sh. the bash is coded as follows:

#!/bin/bash
num=`expr match "$1" '[^0-9]*([0-9]+).*'`
paddednum=`printf "%06d" $num`
echo $1/$num/$paddednum

and can be applied iteratively using for loop as follows:

for i in *.tiff;do mv $i `./zeropad.sh $i`; done

I have two questions:

1- How can I modify the iterator to only apply zeropad.sh on files which need to be zero padded?

2- How can I use the command touch in a for loop to generate test data? It is crucial to verify that this script works before I apply it on the original data.

edited Sep 5 at 23:16

asked Sep 5 at 18:20

kevinkayaks

1295

New contributor

It is slow because you are calling your shell script for every file. It would be quicker if you did the looping and renaming in the same script.
â€“Â Kusalananda
Sep 5 at 19:19

2

Posting as comment because question asks for bash, but I think Python is a better fit: from pathlib import Path; for f in Path().glob('left-?????.tiff'): f.rename(f.name.replace('-', '-0')) Note that touch left-00001..99999.tiff can be used to quickly generate files.
â€“Â BoppreH
Sep 5 at 21:43

add a commentÂ |Â

up vote
4
down vote

favorite

I have many hundreds of thousands of files in a directory. These files are named as follows:

left-00001.tiff
left-00002.tiff
...
left-99999.tiff
left-100000.tiff
...
left-245000.tiff

I would like to rename the files as follows:

left-000001.tiff
...
left-099999.tiff
...
left-245000.tiff

I found elegant solution to this problem here.

This solution implements a bash script called zeropad.sh. the bash is coded as follows:

#!/bin/bash
num=`expr match "$1" '[^0-9]*([0-9]+).*'`
paddednum=`printf "%06d" $num`
echo $1/$num/$paddednum

and can be applied iteratively using for loop as follows:

for i in *.tiff;do mv $i `./zeropad.sh $i`; done

I have two questions:

1- How can I modify the iterator to only apply zeropad.sh on files which need to be zero padded?

2- How can I use the command touch in a for loop to generate test data? It is crucial to verify that this script works before I apply it on the original data.

edited Sep 5 at 23:16

asked Sep 5 at 18:20

kevinkayaks

1295

New contributor

I have many hundreds of thousands of files in a directory. These files are named as follows:

left-00001.tiff
left-00002.tiff
...
left-99999.tiff
left-100000.tiff
...
left-245000.tiff

I would like to rename the files as follows:

left-000001.tiff
...
left-099999.tiff
...
left-245000.tiff

I found elegant solution to this problem here.

This solution implements a bash script called zeropad.sh. the bash is coded as follows:

#!/bin/bash
num=`expr match "$1" '[^0-9]*([0-9]+).*'`
paddednum=`printf "%06d" $num`
echo $1/$num/$paddednum

and can be applied iteratively using for loop as follows:

for i in *.tiff;do mv $i `./zeropad.sh $i`; done

I have two questions:

1- How can I modify the iterator to only apply zeropad.sh on files which need to be zero padded?

2- How can I use the command touch in a for loop to generate test data? It is crucial to verify that this script works before I apply it on the original data.

edited Sep 5 at 23:16

asked Sep 5 at 18:20

kevinkayaks

1295

New contributor

edited Sep 5 at 23:16

asked Sep 5 at 18:20

kevinkayaks

1295

New contributor

asked Sep 5 at 18:20

kevinkayaks

1295

asked Sep 5 at 18:20

kevinkayaks

1295

New contributor

kevinkayaks is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.

It is slow because you are calling your shell script for every file. It would be quicker if you did the looping and renaming in the same script.
â€“Â Kusalananda
Sep 5 at 19:19

2

Posting as comment because question asks for bash, but I think Python is a better fit: from pathlib import Path; for f in Path().glob('left-?????.tiff'): f.rename(f.name.replace('-', '-0')) Note that touch left-00001..99999.tiff can be used to quickly generate files.
â€“Â BoppreH
Sep 5 at 21:43

add a commentÂ |Â

It is slow because you are calling your shell script for every file. It would be quicker if you did the looping and renaming in the same script.
â€“Â Kusalananda
Sep 5 at 19:19

2

Posting as comment because question asks for bash, but I think Python is a better fit: from pathlib import Path; for f in Path().glob('left-?????.tiff'): f.rename(f.name.replace('-', '-0')) Note that touch left-00001..99999.tiff can be used to quickly generate files.
â€“Â BoppreH
Sep 5 at 21:43

It is slow because you are calling your shell script for every file. It would be quicker if you did the looping and renaming in the same script.
â€“Â Kusalananda
Sep 5 at 19:19

Posting as comment because question asks for bash, but I think Python is a better fit: from pathlib import Path; for f in Path().glob('left-?????.tiff'): f.rename(f.name.replace('-', '-0')) Note that touch left-00001..99999.tiff can be used to quickly generate files.
â€“Â BoppreH
Sep 5 at 21:43

add a commentÂ |Â

6 Answers
6

active

oldest

votes

up vote
6
down vote

accepted

The bulk of the time spent by your loop is probably in calling your zeropad.sh script.

Instead, do it all in one script:

#!/bin/bash

for filename in left-*.tiff; do
 if [[ "$filename" =~ ^left-0*([1-9]?[0-9]+).tiff$ ]]; then
 num=$BASH_REMATCH[1]
 newname="left-$( printf '%06d' "$num" ).tiff"
 if [ "$filename" != "$newname" ] && [ ! -e "$newname" ]; then
 echo mv "$filename" "$newname"
 fi
 fi
done

Remove the echo once you have verified that the script is doing the correct thing.

answered Sep 5 at 19:44

Kusalananda

105k14209326

works great and is easiest for me to understand @kusalananda. Thanks very much.
â€“Â kevinkayaks
Sep 5 at 23:15

1

Still forks a mv(1) for each rename(2) system call it needs to make; a dedicated renaming tool like the perl rename script, available as prename on some distros. Or for some patterns, mmv -n '*-foo*' '#1-bar#2 *.tiff` is nice and easier to type. I find it worth installing.
â€“Â Peter Cordes
Sep 6 at 6:44

add a commentÂ |Â

up vote
7
down vote

This is how I usually do it (manually on the shell):

rename left- left-0 left-?.png # for 0-9
rename left- left-0 left-??.png # for 00-99
rename left- left-0 left-???.png # for 000-999
# result left-0000.png - left-9999.png

This is easy to do in an interactive shell session... just repeat the last command with one additional ? added.

However, with a large number of files, you'll eventually end up with a too long list of arguments. And obviously it's not the most efficient choice, as it ends up renaming the same file multiple times (left-1.png -> left-01.png -> left-001.png -> ...).

Also there are two flavors for rename about, one with perl regular expressions and one without. Depending on distro you end up with rename.ul or perl-rename or other names for them. Basically it renders any script using the rename command unportable since you never know what to expect.

I'm using the the util-linux rename and your question is actually one of their examples, from the man page:

EXAMPLES

Given the files foo1, ..., foo9, foo10, ..., foo278, the commands

 rename foo foo00 foo?
 rename foo foo0 foo??

will turn them into foo001, ..., foo009, foo010, ..., foo278.

Which is the more efficient method (each file renamed only once) but you have to figure out the correct distribution of 000 vs ??? or you'll end up with a wrong result.

To me, the inefficient method is the more practical one, on an interactive shell, when dealing with a reasonable small set of files.

The advantage of rename over scripting it yourself is that it doesn't have to spawn one mv process for each file, or as in your case, a sub-script just to figure out a filename. It's not clear what has more overhead, the process spawning, or repeated renaming, and I'm too lazy to benchmark it.

Actually the answer you linked already contains the "optimal" solution at the very end... using perl-rename:

rename 's/d+/sprintf("%04d",$&)/e' *.png

Well, one can argue about the regular expression, but the point is, it's possible to do it all in one go, without unnecessary mv, or spawning of processes. If you still need to improve on that, write a tool that reads directory contents directly instead of using shell globbing (which sorts, which is slow) and performs the renaming as needed.

Maybe that's actually the answer you linked to and maybe that's why you're getting downvoted. ;)

edited Sep 5 at 19:22

answered Sep 5 at 19:15

frostschutz

24.7k14774

add a commentÂ |Â

up vote
5
down vote

For the first part, consider:

for i in left-?????.tiff left-????.tiff left-???.tiff left-??.tiff left-?.tiff ...`

If that generates too many files, then break it up into sections:

for i in left-?????.tiff ...`

...

for i in left-????.tiff ...`

...

The above works by using the ? glob character to substitute any single character whenever it appears. Here, I've specifically requested 5, 4, 3, 2, and then 1 digits after the leading left-.

For the second part, one option is:

dir=$(mktemp)
cd "$dir"
for i in $(seq 10); do touch $(printf 'left-%05d.tiff' $((RANDOM % 10000))); done

Adjust the seq 10 to generate more or fewer filenames. Adjust the % 10000 to generate smaller or larger numbers. Note that bash's $RANDOM generates numbers between 0 and 32,767.

answered Sep 5 at 18:57

Jeff Schaller

32.1k849109

Thanks Jeff. I will try this as soon as I'm back to my pc
â€“Â kevinkayaks
Sep 5 at 19:03

Thakns jeff, I used the dummy data generator you suggested. The renaming script led to a generating too many files bug and I didn't really troubleshoot. Sorry I didn't accept it. @kusalanda's solution is just easier to understand with my limited unix knowledge
â€“Â kevinkayaks
Sep 5 at 23:18

add a commentÂ |Â

up vote
4
down vote

What's costly is to fork so many processes and run so many commands for each file.

With zsh:

zmodload zsh/files # make mv builtin to speed things up
autoload zmv
zmv -n '(*-)(<->)(.tiff)' '$1$(l:6::0:)2$3'

(remove -n when happy)

That's all with builtins, so doesn't fork any process nor execute any file.

Or with perl's rename:

rename -n 's/d+(?=.tiffz)/sprintf "%06d", $&/e' ./*[0-9].tiff

answered Sep 5 at 22:22

StÃ©phane Chazelas

283k53521858

add a commentÂ |Â

up vote
3
down vote

You can rename all the files in parallel. Do the following trivial changes on the same slow code that you had provided in your question, as follows:

cd data_folder # cd the folder where you put the *.tiff files
for i in *.tiff;do 

mv $i `./zeropad.sh $i`;
&

This will rename all the files at once. Please be aware that you must have enough memory resources on your workstation before you run this code inside the folder that contain the *.tiff files. No enough memory resources may lead to memory crash. But given that the process is only renaming files you should be fine!

In order to take into acccount memory resources on your workstation. Save the following code in a file called code, give it permissions then run it:

mem=$(free -m | awk 'NR==2printf "Memory Usage: %s/%sMB (%.2f%%)n", $3,$2,$3*100/$2 ' | grep Memory | awk 'print $3' | tr -d "()%MB" | cut -d / -f 2 )

for i in *.tiff;do 

mv $i `./zeropad.sh $i`;
&

if [ $mem -lt 100000 ] 
 then 
 if (( "$i" % 75 == 0 ))
 then 
 sleep 4
 fi
 fi
if [ $mem -gt 100000 ] 
 then 
 if (( "$i" % 300 == 0 ))
 then 
 sleep 3
 fi
 fi
done

When you run code, it will check memory resources on your workstation using the variable mem. If memory is less 100000MB, then it will rename 75 files at once. If memory resources are more than 100000MB, then it will rename up to 300 files at once. However, you can adjust all the variables as you want.

edited Sep 6 at 0:31

answered Sep 5 at 20:14

TNT

309111

New contributor

Hi TNT: I did not try this because @Kusalananda's solution was simply easier for me to understand with limited knowledge. However, how much memory would this require? Does it actually hold the files in RAM? The folder of files is of the order of ~500GB
â€“Â kevinkayaks
Sep 5 at 23:19

@kevinkayaks. 500GB is to much. Please see my edits. I had updated the code. Now it will sense how much memory resources are available and then it will rename the files accordingly. You can adjust the numbers in the code as you want!
â€“Â TNT
Sep 6 at 0:15

1

@kevinkayaks: Renaming a file within the same directory (or on the same filesystem) doesn't require touch the file's data at all. Renaming a 1TB file is as fast as renaming an empty file; the rename(2) system call just has to modify directory entries (and update the ctime in the file's own inode). The memory consumption here doesn't come from filesize, it just comes from the couple MB of RAM for mv, but mostly the zeropad.sh in parallel many times. This doesn't optimize away the bash startup cost of running zeropad.sh once per file.
â€“Â Peter Cordes
Sep 6 at 6:49

add a commentÂ |Â

up vote
1
down vote

I love Perl one-liners:

ls left-*.tiff | perl -ne 'if(m/(S+)-(d+).tiff/)chomp;printf "mv $_ left-%06d.tiffn", $2' | bash

PS, make sure to double check the output before piping into bash. Just to be safe.

answered Sep 5 at 22:59

idnavid

1113

New contributor

2

perl has a rename function built-in. Printing out bash commands is always going to be slower. Or better, use the existing rename wrapper script like Stephane's answer.
â€“Â Peter Cordes
Sep 6 at 6:54

add a commentÂ |Â

Your Answer

StackExchange.ready(function()
var channelOptions =
tags: "".split(" "),
id: "106"
;
initTagRenderer("".split(" "), "".split(" "), channelOptions);

StackExchange.using("externalEditor", function()
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled)
StackExchange.using("snippets", function()
createEditor();
);

else
createEditor();

);

function createEditor()
StackExchange.prepareEditor(
heartbeatType: 'answer',
convertImagesToLinks: false,
noModals: false,
showLowRepImageUploadWarning: true,
reputationToPostImages: null,
bindNavPrevention: true,
postfix: "",
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
);

);

kevinkayaks is a new contributor. Be nice, and check out our Code of Conduct.

draft saved

draft discarded

StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2funix.stackexchange.com%2fquestions%2f467097%2frename-a-subset-of-files-in-a-directory%23new-answer', 'question_page');

);

Post as a guest

Name

6 Answers
6

active

oldest

votes

6 Answers
6

active

oldest

votes

up vote
6
down vote

accepted

The bulk of the time spent by your loop is probably in calling your zeropad.sh script.

Instead, do it all in one script:

#!/bin/bash

for filename in left-*.tiff; do
 if [[ "$filename" =~ ^left-0*([1-9]?[0-9]+).tiff$ ]]; then
 num=$BASH_REMATCH[1]
 newname="left-$( printf '%06d' "$num" ).tiff"
 if [ "$filename" != "$newname" ] && [ ! -e "$newname" ]; then
 echo mv "$filename" "$newname"
 fi
 fi
done

Remove the echo once you have verified that the script is doing the correct thing.

answered Sep 5 at 19:44

Kusalananda

105k14209326

works great and is easiest for me to understand @kusalananda. Thanks very much.
â€“Â kevinkayaks
Sep 5 at 23:15

1

Still forks a mv(1) for each rename(2) system call it needs to make; a dedicated renaming tool like the perl rename script, available as prename on some distros. Or for some patterns, mmv -n '*-foo*' '#1-bar#2 *.tiff` is nice and easier to type. I find it worth installing.
â€“Â Peter Cordes
Sep 6 at 6:44

add a commentÂ |Â

up vote
6
down vote

accepted

The bulk of the time spent by your loop is probably in calling your zeropad.sh script.

Instead, do it all in one script:

#!/bin/bash

for filename in left-*.tiff; do
 if [[ "$filename" =~ ^left-0*([1-9]?[0-9]+).tiff$ ]]; then
 num=$BASH_REMATCH[1]
 newname="left-$( printf '%06d' "$num" ).tiff"
 if [ "$filename" != "$newname" ] && [ ! -e "$newname" ]; then
 echo mv "$filename" "$newname"
 fi
 fi
done

Remove the echo once you have verified that the script is doing the correct thing.

answered Sep 5 at 19:44

Kusalananda

105k14209326

works great and is easiest for me to understand @kusalananda. Thanks very much.
â€“Â kevinkayaks
Sep 5 at 23:15

1

Still forks a mv(1) for each rename(2) system call it needs to make; a dedicated renaming tool like the perl rename script, available as prename on some distros. Or for some patterns, mmv -n '*-foo*' '#1-bar#2 *.tiff` is nice and easier to type. I find it worth installing.
â€“Â Peter Cordes
Sep 6 at 6:44

add a commentÂ |Â

up vote
6
down vote

accepted

The bulk of the time spent by your loop is probably in calling your zeropad.sh script.

Instead, do it all in one script:

#!/bin/bash

for filename in left-*.tiff; do
 if [[ "$filename" =~ ^left-0*([1-9]?[0-9]+).tiff$ ]]; then
 num=$BASH_REMATCH[1]
 newname="left-$( printf '%06d' "$num" ).tiff"
 if [ "$filename" != "$newname" ] && [ ! -e "$newname" ]; then
 echo mv "$filename" "$newname"
 fi
 fi
done

Remove the echo once you have verified that the script is doing the correct thing.

answered Sep 5 at 19:44

Kusalananda

105k14209326

The bulk of the time spent by your loop is probably in calling your zeropad.sh script.

Instead, do it all in one script:

#!/bin/bash

for filename in left-*.tiff; do
 if [[ "$filename" =~ ^left-0*([1-9]?[0-9]+).tiff$ ]]; then
 num=$BASH_REMATCH[1]
 newname="left-$( printf '%06d' "$num" ).tiff"
 if [ "$filename" != "$newname" ] && [ ! -e "$newname" ]; then
 echo mv "$filename" "$newname"
 fi
 fi
done

Remove the echo once you have verified that the script is doing the correct thing.

answered Sep 5 at 19:44

Kusalananda

105k14209326

answered Sep 5 at 19:44

Kusalananda

105k14209326

answered Sep 5 at 19:44

Kusalananda

105k14209326

answered Sep 5 at 19:44

Kusalananda

105k14209326

works great and is easiest for me to understand @kusalananda. Thanks very much.
â€“Â kevinkayaks
Sep 5 at 23:15

1

Still forks a mv(1) for each rename(2) system call it needs to make; a dedicated renaming tool like the perl rename script, available as prename on some distros. Or for some patterns, mmv -n '*-foo*' '#1-bar#2 *.tiff` is nice and easier to type. I find it worth installing.
â€“Â Peter Cordes
Sep 6 at 6:44

add a commentÂ |Â

works great and is easiest for me to understand @kusalananda. Thanks very much.
â€“Â kevinkayaks
Sep 5 at 23:15

1

Still forks a mv(1) for each rename(2) system call it needs to make; a dedicated renaming tool like the perl rename script, available as prename on some distros. Or for some patterns, mmv -n '*-foo*' '#1-bar#2 *.tiff` is nice and easier to type. I find it worth installing.
â€“Â Peter Cordes
Sep 6 at 6:44

works great and is easiest for me to understand @kusalananda. Thanks very much.
â€“Â kevinkayaks
Sep 5 at 23:15

Still forks a mv(1) for each rename(2) system call it needs to make; a dedicated renaming tool like the perl rename script, available as prename on some distros. Or for some patterns, mmv -n '*-foo*' '#1-bar#2 *.tiff` is nice and easier to type. I find it worth installing.
â€“Â Peter Cordes
Sep 6 at 6:44

add a commentÂ |Â

up vote
7
down vote

This is how I usually do it (manually on the shell):

rename left- left-0 left-?.png # for 0-9
rename left- left-0 left-??.png # for 00-99
rename left- left-0 left-???.png # for 000-999
# result left-0000.png - left-9999.png

This is easy to do in an interactive shell session... just repeat the last command with one additional ? added.

I'm using the the util-linux rename and your question is actually one of their examples, from the man page:

EXAMPLES

Given the files foo1, ..., foo9, foo10, ..., foo278, the commands

 rename foo foo00 foo?
 rename foo foo0 foo??

will turn them into foo001, ..., foo009, foo010, ..., foo278.

Which is the more efficient method (each file renamed only once) but you have to figure out the correct distribution of 000 vs ??? or you'll end up with a wrong result.

To me, the inefficient method is the more practical one, on an interactive shell, when dealing with a reasonable small set of files.

Actually the answer you linked already contains the "optimal" solution at the very end... using perl-rename:

rename 's/d+/sprintf("%04d",$&)/e' *.png

Maybe that's actually the answer you linked to and maybe that's why you're getting downvoted. ;)

edited Sep 5 at 19:22

answered Sep 5 at 19:15

frostschutz

24.7k14774

add a commentÂ |Â

up vote
7
down vote

This is how I usually do it (manually on the shell):

rename left- left-0 left-?.png # for 0-9
rename left- left-0 left-??.png # for 00-99
rename left- left-0 left-???.png # for 000-999
# result left-0000.png - left-9999.png

This is easy to do in an interactive shell session... just repeat the last command with one additional ? added.

I'm using the the util-linux rename and your question is actually one of their examples, from the man page:

EXAMPLES

Given the files foo1, ..., foo9, foo10, ..., foo278, the commands

 rename foo foo00 foo?
 rename foo foo0 foo??

will turn them into foo001, ..., foo009, foo010, ..., foo278.

Which is the more efficient method (each file renamed only once) but you have to figure out the correct distribution of 000 vs ??? or you'll end up with a wrong result.

To me, the inefficient method is the more practical one, on an interactive shell, when dealing with a reasonable small set of files.

Actually the answer you linked already contains the "optimal" solution at the very end... using perl-rename:

rename 's/d+/sprintf("%04d",$&)/e' *.png

Maybe that's actually the answer you linked to and maybe that's why you're getting downvoted. ;)

edited Sep 5 at 19:22

answered Sep 5 at 19:15

frostschutz

24.7k14774

add a commentÂ |Â

up vote
7
down vote

This is how I usually do it (manually on the shell):

rename left- left-0 left-?.png # for 0-9
rename left- left-0 left-??.png # for 00-99
rename left- left-0 left-???.png # for 000-999
# result left-0000.png - left-9999.png

This is easy to do in an interactive shell session... just repeat the last command with one additional ? added.

I'm using the the util-linux rename and your question is actually one of their examples, from the man page:

EXAMPLES

Given the files foo1, ..., foo9, foo10, ..., foo278, the commands

 rename foo foo00 foo?
 rename foo foo0 foo??

will turn them into foo001, ..., foo009, foo010, ..., foo278.

Which is the more efficient method (each file renamed only once) but you have to figure out the correct distribution of 000 vs ??? or you'll end up with a wrong result.

To me, the inefficient method is the more practical one, on an interactive shell, when dealing with a reasonable small set of files.

Actually the answer you linked already contains the "optimal" solution at the very end... using perl-rename:

rename 's/d+/sprintf("%04d",$&)/e' *.png

Maybe that's actually the answer you linked to and maybe that's why you're getting downvoted. ;)

edited Sep 5 at 19:22

answered Sep 5 at 19:15

frostschutz

24.7k14774

This is how I usually do it (manually on the shell):

rename left- left-0 left-?.png # for 0-9
rename left- left-0 left-??.png # for 00-99
rename left- left-0 left-???.png # for 000-999
# result left-0000.png - left-9999.png

This is easy to do in an interactive shell session... just repeat the last command with one additional ? added.

I'm using the the util-linux rename and your question is actually one of their examples, from the man page:

EXAMPLES

Given the files foo1, ..., foo9, foo10, ..., foo278, the commands

 rename foo foo00 foo?
 rename foo foo0 foo??

will turn them into foo001, ..., foo009, foo010, ..., foo278.

Which is the more efficient method (each file renamed only once) but you have to figure out the correct distribution of 000 vs ??? or you'll end up with a wrong result.

To me, the inefficient method is the more practical one, on an interactive shell, when dealing with a reasonable small set of files.

Actually the answer you linked already contains the "optimal" solution at the very end... using perl-rename:

rename 's/d+/sprintf("%04d",$&)/e' *.png

Maybe that's actually the answer you linked to and maybe that's why you're getting downvoted. ;)

edited Sep 5 at 19:22

answered Sep 5 at 19:15

frostschutz

24.7k14774

edited Sep 5 at 19:22

answered Sep 5 at 19:15

frostschutz

24.7k14774

answered Sep 5 at 19:15

frostschutz

24.7k14774

answered Sep 5 at 19:15

frostschutz

24.7k14774

add a commentÂ |Â

up vote
5
down vote

For the first part, consider:

for i in left-?????.tiff left-????.tiff left-???.tiff left-??.tiff left-?.tiff ...`

If that generates too many files, then break it up into sections:

for i in left-?????.tiff ...`

...

for i in left-????.tiff ...`

...

The above works by using the ? glob character to substitute any single character whenever it appears. Here, I've specifically requested 5, 4, 3, 2, and then 1 digits after the leading left-.

For the second part, one option is:

dir=$(mktemp)
cd "$dir"
for i in $(seq 10); do touch $(printf 'left-%05d.tiff' $((RANDOM % 10000))); done

Adjust the seq 10 to generate more or fewer filenames. Adjust the % 10000 to generate smaller or larger numbers. Note that bash's $RANDOM generates numbers between 0 and 32,767.

answered Sep 5 at 18:57

Jeff Schaller

32.1k849109

Thanks Jeff. I will try this as soon as I'm back to my pc
â€“Â kevinkayaks
Sep 5 at 19:03

Thakns jeff, I used the dummy data generator you suggested. The renaming script led to a generating too many files bug and I didn't really troubleshoot. Sorry I didn't accept it. @kusalanda's solution is just easier to understand with my limited unix knowledge
â€“Â kevinkayaks
Sep 5 at 23:18

add a commentÂ |Â

up vote
5
down vote

For the first part, consider:

for i in left-?????.tiff left-????.tiff left-???.tiff left-??.tiff left-?.tiff ...`

If that generates too many files, then break it up into sections:

for i in left-?????.tiff ...`

...

for i in left-????.tiff ...`

...

The above works by using the ? glob character to substitute any single character whenever it appears. Here, I've specifically requested 5, 4, 3, 2, and then 1 digits after the leading left-.

For the second part, one option is:

dir=$(mktemp)
cd "$dir"
for i in $(seq 10); do touch $(printf 'left-%05d.tiff' $((RANDOM % 10000))); done

Adjust the seq 10 to generate more or fewer filenames. Adjust the % 10000 to generate smaller or larger numbers. Note that bash's $RANDOM generates numbers between 0 and 32,767.

answered Sep 5 at 18:57

Jeff Schaller

32.1k849109

Thanks Jeff. I will try this as soon as I'm back to my pc
â€“Â kevinkayaks
Sep 5 at 19:03

Thakns jeff, I used the dummy data generator you suggested. The renaming script led to a generating too many files bug and I didn't really troubleshoot. Sorry I didn't accept it. @kusalanda's solution is just easier to understand with my limited unix knowledge
â€“Â kevinkayaks
Sep 5 at 23:18

add a commentÂ |Â

up vote
5
down vote

For the first part, consider:

for i in left-?????.tiff left-????.tiff left-???.tiff left-??.tiff left-?.tiff ...`

If that generates too many files, then break it up into sections:

for i in left-?????.tiff ...`

...

for i in left-????.tiff ...`

...

The above works by using the ? glob character to substitute any single character whenever it appears. Here, I've specifically requested 5, 4, 3, 2, and then 1 digits after the leading left-.

For the second part, one option is:

dir=$(mktemp)
cd "$dir"
for i in $(seq 10); do touch $(printf 'left-%05d.tiff' $((RANDOM % 10000))); done

Adjust the seq 10 to generate more or fewer filenames. Adjust the % 10000 to generate smaller or larger numbers. Note that bash's $RANDOM generates numbers between 0 and 32,767.

answered Sep 5 at 18:57

Jeff Schaller

32.1k849109

For the first part, consider:

for i in left-?????.tiff left-????.tiff left-???.tiff left-??.tiff left-?.tiff ...`

If that generates too many files, then break it up into sections:

for i in left-?????.tiff ...`

...

for i in left-????.tiff ...`

...

The above works by using the ? glob character to substitute any single character whenever it appears. Here, I've specifically requested 5, 4, 3, 2, and then 1 digits after the leading left-.

For the second part, one option is:

dir=$(mktemp)
cd "$dir"
for i in $(seq 10); do touch $(printf 'left-%05d.tiff' $((RANDOM % 10000))); done

Adjust the seq 10 to generate more or fewer filenames. Adjust the % 10000 to generate smaller or larger numbers. Note that bash's $RANDOM generates numbers between 0 and 32,767.

answered Sep 5 at 18:57

Jeff Schaller

32.1k849109

answered Sep 5 at 18:57

Jeff Schaller

32.1k849109

answered Sep 5 at 18:57

Jeff Schaller

32.1k849109

answered Sep 5 at 18:57

Jeff Schaller

32.1k849109

Thanks Jeff. I will try this as soon as I'm back to my pc
â€“Â kevinkayaks
Sep 5 at 19:03

Thakns jeff, I used the dummy data generator you suggested. The renaming script led to a generating too many files bug and I didn't really troubleshoot. Sorry I didn't accept it. @kusalanda's solution is just easier to understand with my limited unix knowledge
â€“Â kevinkayaks
Sep 5 at 23:18

add a commentÂ |Â

Thanks Jeff. I will try this as soon as I'm back to my pc
â€“Â kevinkayaks
Sep 5 at 19:03

Thakns jeff, I used the dummy data generator you suggested. The renaming script led to a generating too many files bug and I didn't really troubleshoot. Sorry I didn't accept it. @kusalanda's solution is just easier to understand with my limited unix knowledge
â€“Â kevinkayaks
Sep 5 at 23:18

Thanks Jeff. I will try this as soon as I'm back to my pc
â€“Â kevinkayaks
Sep 5 at 19:03

Thakns jeff, I used the dummy data generator you suggested. The renaming script led to a generating too many files bug and I didn't really troubleshoot. Sorry I didn't accept it. @kusalanda's solution is just easier to understand with my limited unix knowledge
â€“Â kevinkayaks
Sep 5 at 23:18

add a commentÂ |Â

up vote
4
down vote

What's costly is to fork so many processes and run so many commands for each file.

With zsh:

zmodload zsh/files # make mv builtin to speed things up
autoload zmv
zmv -n '(*-)(<->)(.tiff)' '$1$(l:6::0:)2$3'

(remove -n when happy)

That's all with builtins, so doesn't fork any process nor execute any file.

Or with perl's rename:

rename -n 's/d+(?=.tiffz)/sprintf "%06d", $&/e' ./*[0-9].tiff

answered Sep 5 at 22:22

StÃ©phane Chazelas

283k53521858

add a commentÂ |Â

up vote
4
down vote

What's costly is to fork so many processes and run so many commands for each file.

With zsh:

zmodload zsh/files # make mv builtin to speed things up
autoload zmv
zmv -n '(*-)(<->)(.tiff)' '$1$(l:6::0:)2$3'

(remove -n when happy)

That's all with builtins, so doesn't fork any process nor execute any file.

Or with perl's rename:

rename -n 's/d+(?=.tiffz)/sprintf "%06d", $&/e' ./*[0-9].tiff

answered Sep 5 at 22:22

283k53521858

add a commentÂ |Â

up vote
4
down vote

What's costly is to fork so many processes and run so many commands for each file.

With zsh:

zmodload zsh/files # make mv builtin to speed things up
autoload zmv
zmv -n '(*-)(<->)(.tiff)' '$1$(l:6::0:)2$3'

(remove -n when happy)

That's all with builtins, so doesn't fork any process nor execute any file.

Or with perl's rename:

rename -n 's/d+(?=.tiffz)/sprintf "%06d", $&/e' ./*[0-9].tiff

answered Sep 5 at 22:22

283k53521858

What's costly is to fork so many processes and run so many commands for each file.

With zsh:

zmodload zsh/files # make mv builtin to speed things up
autoload zmv
zmv -n '(*-)(<->)(.tiff)' '$1$(l:6::0:)2$3'

(remove -n when happy)

That's all with builtins, so doesn't fork any process nor execute any file.

Or with perl's rename:

rename -n 's/d+(?=.tiffz)/sprintf "%06d", $&/e' ./*[0-9].tiff

answered Sep 5 at 22:22

283k53521858

answered Sep 5 at 22:22

283k53521858

answered Sep 5 at 22:22

283k53521858

answered Sep 5 at 22:22

283k53521858

add a commentÂ |Â

up vote
3
down vote

You can rename all the files in parallel. Do the following trivial changes on the same slow code that you had provided in your question, as follows:

cd data_folder # cd the folder where you put the *.tiff files
for i in *.tiff;do 

mv $i `./zeropad.sh $i`;
&

In order to take into acccount memory resources on your workstation. Save the following code in a file called code, give it permissions then run it:

mem=$(free -m | awk 'NR==2printf "Memory Usage: %s/%sMB (%.2f%%)n", $3,$2,$3*100/$2 ' | grep Memory | awk 'print $3' | tr -d "()%MB" | cut -d / -f 2 )

for i in *.tiff;do 

mv $i `./zeropad.sh $i`;
&

if [ $mem -lt 100000 ] 
 then 
 if (( "$i" % 75 == 0 ))
 then 
 sleep 4
 fi
 fi
if [ $mem -gt 100000 ] 
 then 
 if (( "$i" % 300 == 0 ))
 then 
 sleep 3
 fi
 fi
done

edited Sep 6 at 0:31

answered Sep 5 at 20:14

TNT

309111

New contributor

Hi TNT: I did not try this because @Kusalananda's solution was simply easier for me to understand with limited knowledge. However, how much memory would this require? Does it actually hold the files in RAM? The folder of files is of the order of ~500GB
â€“Â kevinkayaks
Sep 5 at 23:19

@kevinkayaks. 500GB is to much. Please see my edits. I had updated the code. Now it will sense how much memory resources are available and then it will rename the files accordingly. You can adjust the numbers in the code as you want!
â€“Â TNT
Sep 6 at 0:15

1

@kevinkayaks: Renaming a file within the same directory (or on the same filesystem) doesn't require touch the file's data at all. Renaming a 1TB file is as fast as renaming an empty file; the rename(2) system call just has to modify directory entries (and update the ctime in the file's own inode). The memory consumption here doesn't come from filesize, it just comes from the couple MB of RAM for mv, but mostly the zeropad.sh in parallel many times. This doesn't optimize away the bash startup cost of running zeropad.sh once per file.
â€“Â Peter Cordes
Sep 6 at 6:49

add a commentÂ |Â

up vote
3
down vote

You can rename all the files in parallel. Do the following trivial changes on the same slow code that you had provided in your question, as follows:

cd data_folder # cd the folder where you put the *.tiff files
for i in *.tiff;do 

mv $i `./zeropad.sh $i`;
&

In order to take into acccount memory resources on your workstation. Save the following code in a file called code, give it permissions then run it:

mem=$(free -m | awk 'NR==2printf "Memory Usage: %s/%sMB (%.2f%%)n", $3,$2,$3*100/$2 ' | grep Memory | awk 'print $3' | tr -d "()%MB" | cut -d / -f 2 )

for i in *.tiff;do 

mv $i `./zeropad.sh $i`;
&

if [ $mem -lt 100000 ] 
 then 
 if (( "$i" % 75 == 0 ))
 then 
 sleep 4
 fi
 fi
if [ $mem -gt 100000 ] 
 then 
 if (( "$i" % 300 == 0 ))
 then 
 sleep 3
 fi
 fi
done

edited Sep 6 at 0:31

answered Sep 5 at 20:14

TNT

309111

New contributor

Hi TNT: I did not try this because @Kusalananda's solution was simply easier for me to understand with limited knowledge. However, how much memory would this require? Does it actually hold the files in RAM? The folder of files is of the order of ~500GB
â€“Â kevinkayaks
Sep 5 at 23:19

@kevinkayaks. 500GB is to much. Please see my edits. I had updated the code. Now it will sense how much memory resources are available and then it will rename the files accordingly. You can adjust the numbers in the code as you want!
â€“Â TNT
Sep 6 at 0:15

1

@kevinkayaks: Renaming a file within the same directory (or on the same filesystem) doesn't require touch the file's data at all. Renaming a 1TB file is as fast as renaming an empty file; the rename(2) system call just has to modify directory entries (and update the ctime in the file's own inode). The memory consumption here doesn't come from filesize, it just comes from the couple MB of RAM for mv, but mostly the zeropad.sh in parallel many times. This doesn't optimize away the bash startup cost of running zeropad.sh once per file.
â€“Â Peter Cordes
Sep 6 at 6:49

add a commentÂ |Â

up vote
3
down vote

You can rename all the files in parallel. Do the following trivial changes on the same slow code that you had provided in your question, as follows:

cd data_folder # cd the folder where you put the *.tiff files
for i in *.tiff;do 

mv $i `./zeropad.sh $i`;
&

In order to take into acccount memory resources on your workstation. Save the following code in a file called code, give it permissions then run it:

mem=$(free -m | awk 'NR==2printf "Memory Usage: %s/%sMB (%.2f%%)n", $3,$2,$3*100/$2 ' | grep Memory | awk 'print $3' | tr -d "()%MB" | cut -d / -f 2 )

for i in *.tiff;do 

mv $i `./zeropad.sh $i`;
&

if [ $mem -lt 100000 ] 
 then 
 if (( "$i" % 75 == 0 ))
 then 
 sleep 4
 fi
 fi
if [ $mem -gt 100000 ] 
 then 
 if (( "$i" % 300 == 0 ))
 then 
 sleep 3
 fi
 fi
done

edited Sep 6 at 0:31

answered Sep 5 at 20:14

TNT

309111

New contributor

You can rename all the files in parallel. Do the following trivial changes on the same slow code that you had provided in your question, as follows:

cd data_folder # cd the folder where you put the *.tiff files
for i in *.tiff;do 

mv $i `./zeropad.sh $i`;
&

In order to take into acccount memory resources on your workstation. Save the following code in a file called code, give it permissions then run it:

mem=$(free -m | awk 'NR==2printf "Memory Usage: %s/%sMB (%.2f%%)n", $3,$2,$3*100/$2 ' | grep Memory | awk 'print $3' | tr -d "()%MB" | cut -d / -f 2 )

for i in *.tiff;do 

mv $i `./zeropad.sh $i`;
&

if [ $mem -lt 100000 ] 
 then 
 if (( "$i" % 75 == 0 ))
 then 
 sleep 4
 fi
 fi
if [ $mem -gt 100000 ] 
 then 
 if (( "$i" % 300 == 0 ))
 then 
 sleep 3
 fi
 fi
done

edited Sep 6 at 0:31

answered Sep 5 at 20:14

TNT

309111

New contributor

edited Sep 6 at 0:31

answered Sep 5 at 20:14

TNT

309111

New contributor

answered Sep 5 at 20:14

TNT

309111

answered Sep 5 at 20:14

TNT

309111

New contributor

TNT is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.

Hi TNT: I did not try this because @Kusalananda's solution was simply easier for me to understand with limited knowledge. However, how much memory would this require? Does it actually hold the files in RAM? The folder of files is of the order of ~500GB
â€“Â kevinkayaks
Sep 5 at 23:19

@kevinkayaks. 500GB is to much. Please see my edits. I had updated the code. Now it will sense how much memory resources are available and then it will rename the files accordingly. You can adjust the numbers in the code as you want!
â€“Â TNT
Sep 6 at 0:15

1

@kevinkayaks: Renaming a file within the same directory (or on the same filesystem) doesn't require touch the file's data at all. Renaming a 1TB file is as fast as renaming an empty file; the rename(2) system call just has to modify directory entries (and update the ctime in the file's own inode). The memory consumption here doesn't come from filesize, it just comes from the couple MB of RAM for mv, but mostly the zeropad.sh in parallel many times. This doesn't optimize away the bash startup cost of running zeropad.sh once per file.
â€“Â Peter Cordes
Sep 6 at 6:49

add a commentÂ |Â

Hi TNT: I did not try this because @Kusalananda's solution was simply easier for me to understand with limited knowledge. However, how much memory would this require? Does it actually hold the files in RAM? The folder of files is of the order of ~500GB
â€“Â kevinkayaks
Sep 5 at 23:19

@kevinkayaks. 500GB is to much. Please see my edits. I had updated the code. Now it will sense how much memory resources are available and then it will rename the files accordingly. You can adjust the numbers in the code as you want!
â€“Â TNT
Sep 6 at 0:15

1

@kevinkayaks: Renaming a file within the same directory (or on the same filesystem) doesn't require touch the file's data at all. Renaming a 1TB file is as fast as renaming an empty file; the rename(2) system call just has to modify directory entries (and update the ctime in the file's own inode). The memory consumption here doesn't come from filesize, it just comes from the couple MB of RAM for mv, but mostly the zeropad.sh in parallel many times. This doesn't optimize away the bash startup cost of running zeropad.sh once per file.
â€“Â Peter Cordes
Sep 6 at 6:49

Hi TNT: I did not try this because @Kusalananda's solution was simply easier for me to understand with limited knowledge. However, how much memory would this require? Does it actually hold the files in RAM? The folder of files is of the order of ~500GB
â€“Â kevinkayaks
Sep 5 at 23:19

@kevinkayaks. 500GB is to much. Please see my edits. I had updated the code. Now it will sense how much memory resources are available and then it will rename the files accordingly. You can adjust the numbers in the code as you want!
â€“Â TNT
Sep 6 at 0:15

@kevinkayaks: Renaming a file within the same directory (or on the same filesystem) doesn't require touch the file's data at all. Renaming a 1TB file is as fast as renaming an empty file; the rename(2) system call just has to modify directory entries (and update the ctime in the file's own inode). The memory consumption here doesn't come from filesize, it just comes from the couple MB of RAM for mv, but mostly the zeropad.sh in parallel many times. This doesn't optimize away the bash startup cost of running zeropad.sh once per file.
â€“Â Peter Cordes
Sep 6 at 6:49

add a commentÂ |Â

up vote
1
down vote

I love Perl one-liners:

ls left-*.tiff | perl -ne 'if(m/(S+)-(d+).tiff/)chomp;printf "mv $_ left-%06d.tiffn", $2' | bash

PS, make sure to double check the output before piping into bash. Just to be safe.

answered Sep 5 at 22:59

idnavid

1113

New contributor

2

perl has a rename function built-in. Printing out bash commands is always going to be slower. Or better, use the existing rename wrapper script like Stephane's answer.
â€“Â Peter Cordes
Sep 6 at 6:54

add a commentÂ |Â

up vote
1
down vote

I love Perl one-liners:

ls left-*.tiff | perl -ne 'if(m/(S+)-(d+).tiff/)chomp;printf "mv $_ left-%06d.tiffn", $2' | bash

PS, make sure to double check the output before piping into bash. Just to be safe.

answered Sep 5 at 22:59

idnavid

1113

New contributor

2

perl has a rename function built-in. Printing out bash commands is always going to be slower. Or better, use the existing rename wrapper script like Stephane's answer.
â€“Â Peter Cordes
Sep 6 at 6:54

add a commentÂ |Â

up vote
1
down vote

I love Perl one-liners:

ls left-*.tiff | perl -ne 'if(m/(S+)-(d+).tiff/)chomp;printf "mv $_ left-%06d.tiffn", $2' | bash

PS, make sure to double check the output before piping into bash. Just to be safe.

answered Sep 5 at 22:59

idnavid

1113

New contributor

I love Perl one-liners:

ls left-*.tiff | perl -ne 'if(m/(S+)-(d+).tiff/)chomp;printf "mv $_ left-%06d.tiffn", $2' | bash

PS, make sure to double check the output before piping into bash. Just to be safe.

answered Sep 5 at 22:59

idnavid

1113

New contributor

answered Sep 5 at 22:59

idnavid

1113

New contributor

answered Sep 5 at 22:59

idnavid

1113

answered Sep 5 at 22:59

idnavid

1113

New contributor

idnavid is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.

2

perl has a rename function built-in. Printing out bash commands is always going to be slower. Or better, use the existing rename wrapper script like Stephane's answer.
â€“Â Peter Cordes
Sep 6 at 6:54

add a commentÂ |Â

2

perl has a rename function built-in. Printing out bash commands is always going to be slower. Or better, use the existing rename wrapper script like Stephane's answer.
â€“Â Peter Cordes
Sep 6 at 6:54

perl has a rename function built-in. Printing out bash commands is always going to be slower. Or better, use the existing rename wrapper script like Stephane's answer.
â€“Â Peter Cordes
Sep 6 at 6:54

add a commentÂ |Â

kevinkayaks is a new contributor. Be nice, and check out our Code of Conduct.

draft saved

draft discarded

kevinkayaks is a new contributor. Be nice, and check out our Code of Conduct.

draft saved

draft discarded

Post as a guest

Name

Search This Blog

Iyfjky