How do you copy many files?

[jjordan]

I work with large numbers of files and can't find an easy way to copy them from one location to another.

After trying many different filemanagers, I discovered that Endeavour2 can show directories with large number of files in it quickly, but can't copy them. It just hangs.

The built-in linux command 'cp' says argument list too long.

Does anyone know of either a linux command or a filemanager that can copy large numbers of files.

I usually work with anywhere from 10,000 to 20,000 files, so please don't respond saying that you use so-in-so, but haven't tried it with 10,000 files. I need something that works :-)

I used to work mainframe operating systems in the past, and there didn't seem to be any limitations on the number of files.

Better yet, is there some way to modify 'cp' so that it CAN handle large numbers of files?

[Icarus]

So what happens when you do a "cp -v * /target/dir"? Does it hang or show it is it copying all the files one at a time?

Are the files small or large? Larger ones would naturally take longer

[hlrguy]

Are they 10,000+ unique filenames, or do you need to copy subsets of each directory, not all files at all times?

If I was faced with that, I would simply
tar the entire directory on a regular basis and untar it into the desired location then burn/archive the tar for safety. Even if only one file changed per day, if that was the frequency of the backup, it would simply overwrite the unchanged files if continually untarred into the same location.

cp can use wildcards, which you probably know like *.ppt cow?.ppt, etc.

cp also offers recursive copy, so you can start the copy at the top of the source directory with a * wildcard and recursively, everything is copied.

Are all the files in the same directory, or spread over multiple directories, etc. Provide some more details and I will see if I can think of more. For example, if you were looking to simply backup say, 75 distinct filetypes in a myriad of directories, a simple cp script can be created just to recursively backup and store those filetypes.

hlrguy

[zdude255]

If the * argument still returns too long you can write a script to do:

cp a* /target
cp b* /target
...
cp A* /target
cp B* /target

[jjordan]

Wow, Talk about response time. Thanks

When I use cp, with or without any options, it simply comes back with argument list too long.

The tar idea might work. That makes the most sense, even if it takes awhile.

All files have the same extension. The size varies from about 3MB to about 10MB each. They are all within the same directory.

The way I am doing it now is to use the cp command with wildcards, but I leaves too much room for error by missing a file that may have a filename that is irregular.

Can whatever controls the lenght of the argument list for 'cp' be changed?

[baldguy]

maybe ...
tar cf *.ext | (cd /new/path && tar xf -)

from the book UNIX Power Tools (http://www.oreilly.com/catalog/upt2/)

[jjordan]

No matter what I try in tar, it still comes back with 'argument list too long'. Same as 'cp'. Unless I limit the number of files someway, which I don't want to have to do.

When I do a ulinit -a, it shows almost everything as 'unlimited'.

I'm kind of surprised in this day and age of huge disks and Terabytes of data, that there is a limit to the number of files that filemanagers and commands can handle.

At least in DOS, you could do either a COPY or XCOPY command and it would handle any size list of files. There has to be some way around this ridicuous limitation. It reminds me of the early days of UNIX!

Thanks so far for the feedback and I welcome any others. It looks like I just need to find out how to set some parameter that will allow me to change the 'cp' command.

Does anyone know how hard it would be to write my own 'copy' command;-)

_____________________
When all you got is a bunch of nails and no hammer, you can't get any work done.

[baldguy]

Okay, in that case try this
ls *.ext | xargs cp /that/path

[jjordan]

Baldbuy, didn't work.

But, in the last couple of hours, I learned how to program in bash and wrote my own new 'cp' command. I tested it with 13,000 files and it worked just great.

Here it is if anyone is interested. Mayby the moderator can move it to the 'how I did it' forum.

It's not elegant. What do you expect for 2 hours of learning bash scripting, writing the program and testing it. It currently copies ALL files in the source directory. You're welcome to make it more elegant or flexible.

#!/bin/sh
# This will copy a large number of files from sourcedir to destinationdir
ARGS=2
NO_ARGS=65

if [ $# -ne $ARGS ]
then
echo "Usage: `basename $0` sourcedir destdir"
exit $NO_ARGS
fi

for dir in $1/*
do echo "$2/`basename "$dir"`";
cp "$dir" "$2/`basename "$dir"`"; done
echo "Done"
exit 0

[baldguy]

sorry I answered off the top of my head, I forgot that with cp you need to include the destination directory as part of the xargs command

ls *.ext | xargs -p cp --target-directory=/that/directory/

or if you have spaces/carriage returns

find . -type f -name "*.ext" -print0 | xargs -0 -p cp --target-directory=/that/directory/