perl: removing the first line in HUGE files

[thegreatorangepeel]

I'm working with some VERY large files (around 128M) and as I process each line in the files, I want to remove it for two reasons. First, to gradually make the file more manageable, and second, so that I can pick up where I left off should my power/connection get interrupted.

I've done this sort of thing before using splice, but we are talking about WAY too much file I/O to do it that way.

Is there a different way to remove the first line that is better suited to my needs?

[apeekaboo]

Something like this could probably work:
cat file.txt |perl -pe '$pass=0; while(<>){if($pass == 0){shift} print; $pass ++}' >newfile.txt

Perhaps there is a more elegant solution to do it?

[Sepero]

Originally posted by apeekaboo
Something like this could probably work:
cat file.txt |perl -pe '$pass=0; while(<>){if($pass == 0){shift} print; $pass ++}' >newfile.txt

Perhaps there is a more elegant solution to do it? Is that the same as this?
sed 1d file.txt > newfile.txt
If so, it would seem to be very inefficient.

thegreatorangepeel, I'm not fluent in perl, but perhaps it would be easier for you to reverse the file and snip from the end?

[apeekaboo]

[QUOTE]Originally posted by Sepero
Is that the same as this?
sed 1d file.txt > newfile.txt


You cheated! That's not perl... ;)
I'm by no means a perl guru so I think it could be done in a more simple way, but I doubt it can top sed.
But please prove me wrong. :)

[dchidelf]

perl -ne 'print if $. > 1' filename > filename.new


P.S. I think VERY big is > 1GB ... yours are just very big.

[chrism01]

To get a copy of a file, without the first line of the file:

lines=`wc -l filename|awk '{print $1}'`
lines=$(( $lines - 1 ))
tail -$lines filename > newfile


HTH

[Sepero]

Programming Challenge

Is there no possible way of cutting one line from the file without creating a separate file? (Preferably using perl)

[apeekaboo]

It's possible with the -i swith in perl.
Using dchidelf's solution it could look like this:
perl -i -ne 'print if $. > 1' filename
This would do an in-place replacement, writing back to the same file.