File size limitations?

[jerbear]

One of the software packages I use at work occasionally attempts to write files that can be a few GB in size for some of the designs I've been using. Currently the program crashes with a file size error if it gets up to 2GB. So far this has not really been an issue since the files get sent to a VMS machine which can only handle 1GB files. But this limitation may be fixed in the near future, which brings me to my question: I've heard rumors that 32bit systems in general and linux in particular have an inherent 2GB file size limit. However, I'm not sure if this has all gotten confused with the old hard drive size limitations on old DOS systems. I'm running RHEL3 with ext3 filesystems, and I was wondering if there are any limitations or if the errors are due to a problem in the program itself.
I just realized another factor that may be important. The files are often written directly to a smb-mounted NTFS drive across the network (perhaps not the fastest or best way to do all this, but the engineer that typically does all this has it set up in a way that works well for her for most cases). Are there file size limitations in the smb protocall?

Thanks for your help,
JB

[stumbles]

Originally posted by jerbear
One of the software packages I use at work occasionally attempts to write files that can be a few GB in size for some of the designs I've been using. Currently the program crashes with a file size error if it gets up to 2GB. So far this has not really been an issue since the files get sent to a VMS machine which can only handle 1GB files. But this limitation may be fixed in the near future, which brings me to my question: I've heard rumors that 32bit systems in general and linux in particular have an inherent 2GB file size limit. However, I'm not sure if this has all gotten confused with the old hard drive size limitations on old DOS systems. I'm running RHEL3 with ext3 filesystems, and I was wondering if there are any limitations or if the errors are due to a problem in the program itself.
I just realized another factor that may be important. The files are often written directly to a smb-mounted NTFS drive across the network (perhaps not the fastest or best way to do all this, but the engineer that typically does all this has it set up in a way that works well for her for most cases). Are there file size limitations in the smb protocall?

Thanks for your help,
JB


This might answer your question;

http://www.webservertalk.com/message468571.html


Google is your freind.

[soulestream]

i may have misread the article but it discusses the file limitaion of fat32 which is 2gb. samba doesnt have this limitation

maybe this will help


http://batleth.sapienti-sat.org/projects/FAQs/ext3-faq.html


Ext3 can support files up to 1TB. With a 2.4 kernel the filesystem size is limited by the maximal block device size, which is 2TB. In 2.6 the maximum (32-bit CPU) limit is of block devices is 16TB, but ext3 supports only up to 4TB.


soule

[retsaw]

I've had problems writing a file >4GB to a smb share on a Windows machine (I'll test this again tomorrow to confirm it), but no problems reading files over 4GB. There isn't any general 2GB file size limit, the 2GB file size limitation depends on how the program was written I know wget can't dowload files over 2GB, but other programs can.

[retsaw]

Originally posted by retsaw
I've had problems writing a file >4GB to a smb share on a Windows machine (I'll test this again tomorrow to confirm it), but no problems reading files over 4GB.

I've tested this again and hit the "File size limit exceeded" error at 2GB. The share I was copying to was on a NTFS drive.

[jerbear]

That makes sense. Next time I write the file I'll do it locally and see if that gets rid of the problem. Thanks for the help.

JB

[Alex Cavnar, aka alc6379]

Originally posted by retsaw
I've tested this again and hit the "File size limit exceeded" error at 2GB. The share I was copying to was on a NTFS drive.

That's really odd, you know. I've got a Windows XP pro machine with an 80GB NTFS volume on it, and I've had no problem whatsoever putting 5+GB files on it, whether it's over the network or locally. :confused:

I've done it with a plethora of OSes, to and from: NetBSD/FreeBSD, Linux (2.4 and 2.6 kernels), Windows 2000, 98, XP, and server 2003. I guess YMMV, huh?

[bwkaz]

Maybe it depends on the Samba version in use? Alex, were you using SMB or CIFS when it worked for you? It could be that CIFS allows files larger than 2GB, but SMB doesn't.

[retsaw]

Originally posted by Alex Cavnar, aka alc6379
That's really odd, you know. I've got a Windows XP pro machine with an 80GB NTFS volume on it, and I've had no problem whatsoever putting 5+GB files on it, whether it's over the network or locally. :confused:

I've done it with a plethora of OSes, to and from: NetBSD/FreeBSD, Linux (2.4 and 2.6 kernels), Windows 2000, 98, XP, and server 2003. I guess YMMV, huh?

I was doing it with a share mounted using smbfs with a 2.6.8 kernel. I tried it with smbclient and it worked fine, so smbfs is the problem. I never thought to try smbclient before as the file size limitation wasn't a problem for me as I rarely need to transfer large files to the Windows machine, it's usually just the other way round which never caused me a problem.

[mrBen]

Just to clear something up:

FAT32 maximum file size is 4GB, not 2GB

Linux filesystems should handle big file sizes, although you will need a newish kernel.

XFS is recommended in particular. I _think_ ext2 used to be max of 2GB, but not sure. ext3 is apparently 2TB. Not sure about Reiser

[EnigmaOne]

http://list.nmlug.org/nmlug/2002-April/msg00091.html

http://lists.suse.com/archive/suse-linux-e/2001-Jan/1413.html

http://sourceforge.net/mailarchive/message.php?msg_id=1758879

http://www.redhat.com/archives/fedora-list/2003-November/msg03031.html

http://lists.naos.co.nz/pipermail/wellylug/2004-May/007045.html

http://www.ussg.iu.edu/hypermail/linux/kernel/9803.0/0525.html

ALSO:

http://www.suse.de/~aj/linux_lfs.html

and:

Try this search (http://www.google.com/search?as_q=Reiser&num=100&hl=en&btnG=Google+Search&as_epq=file+size+limit&as_oq=&as_eq=&lr=&as_ft=i&as_filetype=&as_qdr=all&as_nlo=&as_nhi=&as_occt=any&as_dt=i&as_sitesearch=&safe=images)