Can't write to /var!?!!

[NUtack97]

Hey Guys, I was playing around with my SAMBA setup, decided to reboot my server, and now I cannot write ANYTHING to my /var directory :confused: . I checked out my disk free space with df and I have about 59% used in my /var directory (about 500 meg's free). Any ideas, 'cause I'm really at a loss, and rather frustrated :mad: . Any help would be most appreciated....Thanks :D

-Tack

[mpetnuch]

You are root right. I know that might be obvious but I tried setting up a mail thing to /var/mail/mpetnuch but it always failed. Then I realized that I didn't have the permission to right to /var as a normal user ;-)

[Ig0r]

One other thing to check:
It sounds like you're mounting /var as a seperate partition. Check the output of the 'mount' command to make sure it isnt mounted as read-only.

[NUtack97]

Yes, I'm logged in as root for all of this, but the problem came up during my booting sequence. When crond and httpd try to start up, they fail because they can't write to the /var directory.
Also, the mount command gives me the following results:

/dev/sdb5 on / type ext2 (rw)
none on /proc type proc (rw)
none on /dev/pts type devpts (rw,mode=0620)
/dev/sdc1 on /home type ext2 (rw)
/dev/sdb6 on /usr type ext2 (rw)
/dev/sdb7 on /var type ext2 (rw)
automount(pid688) on /misc type autofs (rw,fd=5,pgrp=688,minproto=2,maxproto=3)

Now, I'm assuming that the (rw) means that I have read/write access to /var. Any ideas? thanks again!
-Tackleberry

[NUtack97]

If this helps, here's what it spits back at me when I type crond at the command prompt (when logged in as root):

crond: can't open or create /var/run/crond.pid: No space left on device
crond: can't open log file
/var/log/cron: No space left on device
crond: can't write to log file
CRON (02/27-02:30:43-7117) DEATH (can't open or create /var/run/crond.pid: No sp
ace left on device)

Again, any ideas guys? I'm REALLY confused at this point.

-Tack

[PLBlaze]

Free up some space on your drive and try agin the error its self explenatory "No space left on device".Hope this helps.

[Lorithar]

*grins*

the difference here is MB of space versus the almighty inode...
you may see x free space in df ...by itself ... but try df -i ...

[NUtack97]

That's it! Here's the output when I type in df -i:

Filesystem Inodes IUsed IFree IUse% Mounted on
/dev/sdb5 61k 8.7k 52k 14% /
/dev/sdc1 519k 6.9k 512k 1% /home
/dev/sdb6 261k 64k 197k 25% /usr
/dev/sdb7 178k 178k 0 100% /var

Now, my next question is this. How do I change this inode thing? I'm a total newbie at this, so I really don't know how to fix it. Thanks again!

-Tack

[NUtack97]

One problem with that, PLBlaze, I've tried deleting files and it doesn't seem to help this inode situation at all. After deleting several files, I still have a 100% used inode space, as my above posting shows. How can I fix this? :confused: :confused:

-Tack

[Lorithar]

*okies ... *

I thought so ... now. ..
du /var
somewhere there are several thousand files lying about that are tying up space....
I personally would start by tarring up messages syslog and debug and putting them somewhere else for the time being ... these are likely the largest of your files here..
Then take a good look at /var/mail.
If you use this, you likely have a bit of data there too...

[NUtack97]

Thanks for the suggestion, Lorithar. I tried doing this, and although my disk space is being cleared up, my inode space is not changing. After removing nearly 100 MB, my inode status is still @ 100% usage :confused: . Is there something else I'm doing/not doing properly?

-Tack

[Lorithar]

*hmms*

/var is a completely separate partition

so .. check lost+found ...
The only thing that would help from here would be the output of du /var .. in full..

which would be massive.

once you've deleted something you might want to (as root) sync ..
This will force the drive buffers to be flushed..
next .. ext2fsk the drive and see if it has anything to report..

I can only suspect that something in there is corrupting the filesystem...
Inodes aren't like sectors ... their more like ... ... the pointer to the head of a file. ... it is not the SIZE of the file, but rather hte number of files that will chew up inodes ..
every ext2 filesystem should have at the top a lost+found dir... it is possible that you recovered from a bad shutdown, and ext2fsck pulled the individual sectors of one large file into lost+found as individual files... if the file was large enough this would quickly use up 100k in inodes..

all I can think of is to df -i the individual subdirectories of /var to find out which one is using all the inodes...

[edit]
Skip the above, ... it shows the total for the device.
I'm gonna try flooding one of my work directories (that is its lying around being used for temp work) and see what I can figure out...
I find it *very* odd that you've used all those inodes... I have /usr -- 3.8 GIG partition of which I have used 2.8 GIG and of 1 Million + inodes I have only used 300,000 or so... and I have a LOT of crap in there ... I like experimenting..

[ 28 February 2001: Message edited by: Lorithar ]

[Lorithar]

*hmms*

You said you were working on Samba .. what did you setup in samba .... ???

Ummmms... I wonder if you used samba to create smbfs links to the windows shares you can see ...
I seem to recall somewhere in my reading of samba setups that one can create mappings of windows shares as mounted devices in linux, and that one can then access the filesystem of the share directly ...
I canNOT recall if the mounted windows share uses up inodes ...
ANYONE have any ideas on this ..
(thinking that /var/<windows share mount> could/would be the problem???


[edit]
*kicks self*

LOOK IN /var/lock/samba ...
this is where samba sticks the lock files .. it SHOULD be chmod 755 ... BUT .. if it were not .. it is likely that allllll those lock files are sitting in there using up indoes... and since the files weren't cleaned out before .....*grins* if samba is able to start, ..it is likely tieing the files off ...

[end edit and hope that's the critter]

[ 28 February 2001: Message edited by: Lorithar ]

[Lorithar]

*grins*

after 3.5 hours my little script touched out 127,318 files -- incidentally it used 128,045 inodes ... total space used however was less than 2Mb
However .. (neat side effect that it had) you cannot delete 127,000 dummyfile.xxxxx files by rm dummyfile* ...
(too many arguments)
It is interesting to note that midnight commander doesn't care .. it took them out .. it just took a while.
One other point .. inodes ' An inode (short for index node) contains information about a file such as its owner, permissions, timestamps, and location on the disk.'
quote from man df ...
(duh bounce head on desk)

Nutack .. let me know how this goes.. I can't think of any more for ya ...

[NUtack97]

Hey Lorithar,

I found the problem...Apparently my /var/log/mail directory was flooded with thousands of files that apparently took all of the inode allocations. There's a whole bunch of mail.warn.* files that cluttered things up...I'm rm'ing them slowly. What's this midnight commander thing? I'll probably have this whole directory cleaned out by the time I get a response, but I would like to know for the future.

On a similar note, I need to find out why my system generated so many logs. It's a mail thing, and I don't really use the mail on my server (it was installed automatically by Mandrake :D ), but I'll figure that out sometime in the future.....Thanks for all the help!!

-Tack

[Lorithar]

*grins*

Midnight Commander ..
Ever recall Norton Commander from DOS days?
Midnight Commander is the same kinda creature .. great tool for file management.. run from the command line interface as mc -- gnome file manager is actually gmc ... the gnome front end for .. you guessed ot ... mignight commander...

I figured it might be something from samba cause that's what you were futzing with .. but... mail will deinately tie up space .. I think I suggested looking at var/mail at one point .. but maybe I was half asleep.. and only thought it ...

[NUtack97]

Got the Midnight Commander working, thanks dude! Well, I had about 120,000 files in my /var/log/mail directory, most of 'em with sizes of 0 bytes. Go figure. Well, my system's back to normal operations, and I think I've stopped it from generating so much crud. Thanks for the help everyone!

-Tack
:cool: