Beefy Boxes and Bandwidth Generously Provided by pair Networks
"be consistent"
 
PerlMonks  

Script to show space usage by users in home partition

by OfficeLinebacker (Chaplain)
on Jan 12, 2007 at 06:50 UTC ( #594318=perlquestion: print w/ replies, xml ) Need Help??
OfficeLinebacker has asked for the wisdom of the Perl Monks concerning the following question:

Greetings, esteemed monks!

I kind of had the last straw yesterday with one of my users filling up our home/ partition and then conveniently going home for the evening immediately thereafter. We don't do hard quotas, but I am thinking of ways to nip these situations in the bud, as it were. One method I thought of is a CGI script that displays everyone's overall space usage in home/ so that when stuff breaks, everyone in our section knows whodunit. Other ideas include sending friendly email reminders, etc. Anyway I got kind of caught up in parsing the output of dhog (not sure if that's a standard builtin) and converting the output to "human readable" form (aka dividing by the appropriate power of 1024 and appending K,M,or G as appropriate (the partition is 20G so T isn't necessary (YET!!).

Without further ado (very raw yet):
#!/opt/local/bin/perl #script that pesters users who use up too much space in /home use strict; use warnings; use Readonly; local $\ = "\n"; #use POSIX 'strftime'; Readonly my $QUOTA => 2_000_000; #2G is reasonable, IMO (1k blo +cks) Readonly my $HOME_DIR => '/fst/home'; my @HOM_USE_BY_DIR = `sudo dhog -o $HOME_DIR`; die "Error: sudo dhog failed: $! status $? " if ($?); #print @HOM_USE_BY_DIR; my %HOM_USE_BY_DIR; foreach my $i (@HOM_USE_BY_DIR) { chomp $i; #print $i; if ( my ( $sp, $uid ) = $i =~ m{\s?(\d+)\s+/(?:\w+/)*(m\d[a-z]{3}\ +d{2})} ) { my $KMG = 0; while ( $sp > 1023 ) { #print $sp; $sp = $sp / 1024; ++$KMG; } my $mag_ind = ( $KMG == 0 ) ? 'K' : ( $KMG == 1 ) ? 'M' : ( $KMG == 2 ) ? 'G' : 'some huge multiple of 1024'; $sp = "$sp $mag_ind"; print "$uid: $sp"; } ## end if ( my ( $sp, $uid ) = $i =~ m{\s?(\d+)\s+/(?:\w+/)*(m\d +[a-z]{3}\d{2})} ) } ## end foreach my $i (@HOM_USE_BY_DIR)

As you can see I originally intended to create a hash with logonid=>space use pairs but haven't got around to that yet (any ideas welcome).

So of course I'll have to consider the overall plan and truncate all the insignificant digits, but it's a start. What experience have you guys had when begging/exhorting/recommending use of partitions other than home for users' heavy-duty data work?

I already have a cron job that runs a `df` every five minutes and sends me a warning message if any of my partitions hits 95% usage, so I was thinking of adding another one or modifying that one so that the particular user gets an email if and when their personal space usage in the home partition exceeds $QUOTA. I was also thinking of ccing our boss if the situation isn't resolved in a certain number of hours or days or if it's worsening rapidly.

BTW ChemBoy had some great ideas in the chatterbox which I hope he will post here as I am about to doze off here so pardon the quality of the code and writing.

UPDATE since ChemBoy seems to have beat me to sleep here's some of what he typed in the cb:

first
my $KiB = <>; my @pref = qw(M G T P);my $i = 0; while($i<@pref){last i +f $KiB/1024 < 1; $KiB/=1024;$i++} print "$KiB$pref[$i]iB\n"
then
perl -ne'chomp(my $KiB =$_); my @pref = qw(K M G T P);my $i = 0; while +($i<@pref){ last if $KiB/1024 < 1; $KiB/=1024;$i++} printf "${_}KiB = + %.3f$pref[$i]iB\n",$KiB'
and then
(oh dammit... too clever again. Take out the chomp and use perl -l...

UPDATE: Final form should have been (via /msg from ChemBoy)

perl -lne 'my $KiB =$_; my @pref = qw(K M G T P); my $i = 0; while($i<@pref){ last if $KiB/1024 < 1; $KiB/=1024; $i++ } printf "${_}KiB = %.3f$pref[$i]iB\n ",$KiB'

I like computer programming because it's like Legos for the mind.

Comment on Script to show space usage by users in home partition
Select or Download Code
Re: Script to show space usage by user in home partition
by gaal (Parson) on Jan 12, 2007 at 06:57 UTC
      > We don't do hard quotas, but I am thinking of ways to nip these situations in the bud, as it were.

    Perhaps you should do hard quotas. :-)

    I'm no fan of the bureaucratic process of looking for someone to grant me more space. But if it happens often that (for example) developers on your server accidentally leave some tool running in a very high debug level, it's easier for them to ask for more space when they need it than for you to start hunting the offending dev when *you* (and everyone else) needs it.

    This depends on the usage of the server, of course -- I'm only recommending this because it sounds like you've had this occurrence frequently. On feather, Juerd++ just made a note asking people nicely to clean up after themselves and I'm pretty sure they did. Of course that requires noticing a problem before it turns critical.

      I guess I should have said "they won't do hard quotas." I'm not a sysadmin so I don't have the right to do that, and when I requested it the admin guys said it was too "Big Brother" ish or something. Well, when some inconsiderate person is just too lazy or "in a hurry" to do their work in one of the other partitions that are there for exactly this kind of thing (I believe the cuplrit(s) was/were large dataset(s) and/or log file(s))causing problems where my other users can't even log in to their workstations, it's kind of annoying.

      I don't want to have to run around to peoples' offices at 6 pm asking them to delete some files so that things won't get even more fubar. At the very least this program or some incarnation thereof will end up as a bookmarked CGI script so that myself or someone else can look up the space usage and ask someone person to move their music files out of their roaming profile and on to the workstation hard drive or delete their log files or move their datasets or whatever.

      Sorry if I am a little cranky, but I had to work late today because of this and missed getting to spend time with my wife and kid.

      My boss is a big fan of the public accountability thing; he had me write a web page where each person records if they've tested access to certain websites that they're supposed to be able to get to in an emergency. When he outlined it for me he said like "the checklist on the back of bathroom doors at fast food restaurants." I figure publicly posting disk space usage is something that will be right up his alley.


      I like computer programming because it's like Legos for the mind.

        I see from one of your other posts that you're restricted by sudo to a small set of commands, so you're unlikely to be able to run this directly, but...

        durep is really nice for this sort of thing:

        durep creates disk usage reports with bar graphs, allowing one to easily deduce which directories are using the most space. Although durep can produce text output similar to du, its real power lies in the ability to store reports in a file, which can then be viewed as a web page with the supplied cgi script

        Who knows, your boss may like this enough to possibly persuade your sysadmins to install it ;-)

Re: Script to show space usage by user in home partition
by OfficeLinebacker (Chaplain) on Jan 12, 2007 at 07:27 UTC
    tye posted this in the CB:
    my $size= -s; my $u= "TGMk"; my $p= ''; while( $u && 512 < $size ) { $p= chop $u; $size /= 1024 } $size= 10 <= $size ? int($size) : sprintf "%.1f", $size; print $size, $p, "B";

    I like computer programming because it's like Legos for the mind.
Re: Script to show space usage by users in home partition
by chargrill (Parson) on Jan 12, 2007 at 08:11 UTC

    TIMTOWTDI and all, but I'd just parse the output of:

    # most linuxes ship with du that has --max-depth $ du -h --max-depth 1 /home # OR *bsds ship with du that has -d $ du -h -d 1 /home

    ... and look for likely offenders by grepping for high values of M or just G, i.e. output from above on my system:

    518K /usr/home/colette 7.5M /usr/home/uvnet 74M /usr/home/dsia 1.3G /usr/home/kcowgill 4.0K /usr/home/kimc

    At least you don't have to do a lot of work that du can do for you - which is more portable (installed on every *nix) than dhog (?) which I've honestly never heard of before.

    As the output of du with the -h (human readable) flag (along with some kind of depth flag) is easily readable, implementation is left as an exercise for the, uh ... reader. :-)



    --chargrill
    s**lil*; $*=join'',sort split q**; s;.*;grr; &&s+(.(.)).+$2$1+; $; = qq-$_-;s,.*,ahc,;$,.=chop for split q,,,reverse;print for($,,$;,$*,$/)
      I use at work a loop over the home directories and 'du', too. Thats
      ... ($users{uid}) = `/usr/bin/du -s0 --block-size=1048576 $uid ` =~ /^(\d* +)/; ...
      And %users contains the "wasted" space in MB. Additionally I "nice" the process.
Re: Script to show space usage by users in home partition
by cdarke (Prior) on Jan 12, 2007 at 13:01 UTC
    You might want to consider Filesys::Df from CPAN.
Re: Script to show space usage by users in home partition
by OfficeLinebacker (Chaplain) on Jan 12, 2007 at 13:32 UTC
    Ach! I have sudo privileges on only a small subset of commands, dhog apparently being some home-rolled version of du. du is not a program I can `sudo.` So
    %du --max-depth 1 -h /fst/home/ du: `/fst/home/lost+found': Permission denied <bunch more permissions denied messages> ... 5.9G /fst/home/user1 <bunch more permissions denied messages> ... 3.4G /fst/home/user2 etc.
    I imagine anything that actually tries to look inside of users' filesystems won't work (and, rightly so). The admins have carefully created dhog so I can't find out much, I think, and of course dhog doesn't have a -h flag.

    I like computer programming because it's like Legos for the mind.
Re: Script to show space usage by users in home partition
by neilwatson (Curate) on Jan 12, 2007 at 19:48 UTC
    Quotas are really the best way to go. However, I did write something long ago that might suite you: spacehog

    Neil Watson
    watson-wilson.ca

Re: Script to show space usage by users in home partition
by OfficeLinebacker (Chaplain) on Jan 13, 2007 at 06:07 UTC
    Greetings, esteemed monks!

    I went ahead and checked with one my my close contacts in the server admin section and got this surprising response (they have over 30 users to our seven (soon to be 9), and a 30-GB home partition to our 20):

    We've been using a quota on our /home now for months, with only a few 
    problems.  Of course, these problems can tend to be really big problems, 
    since it's on the home filesystem (half-written windows profiles, lost 
    email, etc.).  On the other hand, we haven't had /arc/home fill up once 
    since we set it up (and it was happening just about on a weekly basis).  
    The soft quota (enforced by bugging the bad user until they quit) could 
    certainly work.
    
    If you'd like us to set up real quotas (i.e. enforced linux filesystem 
    quotas), I think we might consider it, but there would probably have to be 
    some pretty high level discussion (i.e. section chief to section chief, 
    or at least section chief to Greg), just making sure that everyone 
    involved is aware of the potential problems, and working out exactly how 
    to do it.
    
    Anyway, I think I am going to start out with a link on our section home page to a simple web page listing the sorted by-home-directory listing of space usage in home as oputput by dhog-o; I might tag those over 2GB in bold. I will probably augment with an emailer of some kind.

    My users in general are reasonable, cooperative, good people. I don't have the kind of horror stories most tech support people have. I think I will first try seeing the effect the shame factor of having that info "out there" for all to see and then go from there. I am also thinking about having the program that calculates the sizes post complimentary text for users with the smallest directories, and maybe even track info like largest 24 hour change in both relative and absolute terms and create a "most improved award" for the person who reduces their space consumption the most or something (the "Prodigal Son" treatment if you will). I want to focus on positive reinforcement. If things don't improve, it's time to start the email barrages. This is all of course subject to boss approval, but when home/ fills up, work doesn't get done. I think as long as I am not verbally berating people in the hallways or really going out of my way to hurt peoples' feelings my boss can't deny me.

    Thanks for all the suggestions. I'll try to keep y'all updated. This is an interesting experiment that is really just as much psychology as it is Perl....or are the two closer than I once thought?


    I like computer programming because it's like Legos for the mind.

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: perlquestion [id://594318]
Approved by ysth
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others musing on the Monastery: (3)
As of 2014-10-26 05:22 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    For retirement, I am banking on:










    Results (151 votes), past polls