Adding/Committing Files to CVS

[bassplayer]

Greetings Exalted Monks,

I have been tasked with adding a huge amount of files to CVS. This large web directory was created by others who did not employ CVS (as they should have). I started adding a few files, before realizing what a tremendous amount of time was involved. Then I decided to write some Perl to make my life easier.

Basically, I now have a program that I run in the main directory, and it is supposed to find all acceptible files (.pl, .cgi, .htm, .html, .shtml) and then add and commit them to the repository. It also adds and commits images using the binary setting of CVS. If it finds a directory, it should recurse through it and add and commit files found inside this directory, etc.

I initially found that within directories, the files were not being added or committed, because CVS was being passed file names without full path names. I am using IO::Dir to get directory information. It is not giving me full path names, just file names.

I am storing the directory names in an array, then adding and committing later because I was getting a Deep recursion on subroutine warning.

I apologize for the crappy code, but I wanted to get it working before I later optimized it (and I'm no Perl genius anyway).

I guess my question is, can I even do this the way that I have set out (IO::Dir, Shell)? Is there another way that someone can suggest? Does CVS offer any automation like this?

This should be a simple bit of code, I know, and I'm embarrassed to post this, but at least I'll learn. Anyway, here's the code:

#! /usr/bin/perl -w

use strict;
use IO::Dir;
use Shell;

open (LOG_FILE, ">/home/bassplayer/add_to_cvs_log.txt") || die "Couldn
+'t open log file";
my @all_directories;
my @directories = get_directories(
    dir => '.',
    first => '1'
);
while ( @directories ) {
    my @more_directories = get_directories(
        dir => $directories[0],
        first => '0'
    );
    push @directories, @more_directories;
    push @all_directories, $directories[0];
    shift @directories;
}

foreach ( @all_directories ) {
    print STDOUT "ALL_DIR: $_\n";
    check_files( dir => $_ );
}

close LOG_FILE;

exit 0;
 




#####################
sub get_directories {
#####################

    my %arg = @_;
    my @directories;

    my $d = new IO::Dir $arg{dir};
    if ( defined $d ) {
        while ( defined( $_ = $d->read ) ) {
            unless ( $_ eq '..' || ( $_ eq '.' && $arg{first} ne '1' )
+ || $_ eq 'CVS' ) {;
                if ( -d $_ ) {
                    print STDOUT "DIR: $_\n";
                    push @directories, $_;
                }
            }
        }
        undef $d;
    }

    return @directories;

}





#################
sub check_files {
#################

    my %arg = @_;

    my $d = new IO::Dir $arg{dir};
    if ( defined $d ) {
        while ( defined( $_ = $d->read ) ) {
            next if $_ eq '..' || $_ eq '.' || $_ eq 'CVS';
            if ( -d $_ ) {
                print LOG_FILE "ADDING DIR: $_\n";
                my $cvs_output = cvs( 'add', $_ );
                print LOG_FILE "COMMITTING DIR: $_\n";
                $cvs_output .= cvs( 'commit', '-m _', $_ );
                print LOG_FILE $cvs_output;
            }
            if ( ( -B $_ ) && ( $_ =~ m/\.jpg$/ || $_ =~ m/\.jpeg$/ ||
+ $_ =~ m/\.gif$/ ) ) {
                print LOG_FILE "ADDING FILE: $_\n";
                my $cvs_output = cvs( 'add', '-kb', $_ );
                print LOG_FILE "COMMITTING FILE: $_\n";
                $cvs_output .= cvs( 'commit', '-m _', $_ );
                print LOG_FILE $cvs_output;
            }
            elsif ( $_ =~ m/\.pl$/ || m/\.cgi$/ || m/\.htm$/ || $_ =~ 
+m/\.html$/ || $_ =~ m/\.shtml$/ ) {
                print LOG_FILE "ADDING FILE: $_\n";
                my $cvs_output = cvs( 'add', $_ );
                print LOG_FILE "COMMITTING FILE: $_\n";
                $cvs_output .= cvs( 'commit', '-m _', $_ );
                print LOG_FILE $cvs_output;
            }
        }
        undef $d;
    }

}

Your help is, of course, very much appreciated. Optimization suggestions are also welcome, as I do not get code review from anywhere, and am very interested in improving my Perl.

bassplayer

[jkahn]

CVS itself offers some features to deal with this. A bit dangerous in this forum, but Perl may not be the Right Thing here.

Instead, I strongly recommend taking a good hard look at the cvs import subcommand, which is a sort of sophisticated recursive cvs add, and reading the documentation for CVS to see how to use its configuration file to skip files.

In summary, since CVS already handles this it might be a Bad Idea to write your own recursive add routines.

[mfriedman]

I would reccomend using File::Find to locate all the files you want to commit. It will impliment the recursion for you and return full pathnames.

[rbc]

You may be able to use cvs's import command if the code is not already in the repository.

[TStanley]

As mentioned above, the File::Find module is a good one to use. And in the September issue of Sys Admin magazine, merlyn shows how well it can be used.

TStanley
--------
It is God's job to forgive Osama Bin Laden. It is our job to arrange the meeting -- General Norman Schwartzkopf

[csotzing]

In perl, anytime you need to parse a directory tree, File::Find is a good answer. You have options like changing to the directory you're parsing, or going depth-first. You can also get the file's name, parent directory, path, etc.

Definitely check out the cvs options first.

-C

---

print(map(lc(chr),split(6,qw/99672682673683684689632658645641610607/)));

[bassplayer]

Well, I fiddled with File::Find for a bit, but I couldn't get it to do what I needed. I ended up using opendir. Here is the code I wound up with, in case it might actually benefit someone:


#! /usr/bin/perl -w

use strict;
use IO::Dir;
use Shell;

open (LOG_FILE, ">/home/bassplayer/add_to_cvs/htdocs_out.txt") || die 
+"Couldn't open log file";
open (DIR_FILE, ">/home/bassplayer/add_to_cvs/htdocs_dir.txt") || die 
+"Couldn't open dir file";
my @all_directories;

my @directories = ( '.' );

while ( @directories ) {
    my @more_directories = get_directories(
        dir => $directories[0],
        first => '0'
    );

    push @directories, @more_directories;
    push @all_directories, $directories[0];
    shift @directories;
}

print LOG_FILE "TOTAL NUM DIR: $#all_directories\n";
foreach ( @all_directories ) {
    check_files( dir => $_ );
}

close DIR_FILE;
close LOG_FILE;

exit 0;





#####################
sub get_directories {
#####################

    my %arg = @_;
    my @directories;
    my $path = $arg{dir};

    opendir(DIR, $arg{dir}) || die "can't opendir $arg{dir}: $!";
    my @output = grep { -d "$arg{dir}/$_" } readdir(DIR);
    for ( my $i = 0 ; $i <= $#output ; $i++ ) {
        if ( $output[$i] ne '.' && $output[$i] ne '..' && $output[$i] 
+!~ m/CVS/ && $arg{dir} !~ m/CVS/ ) {
            my $directory = "$arg{dir}/$output[$i]";
            my $cvs_output;
            print LOG_FILE "ADDING DIR: $directory\n";
            $cvs_output = cvs( 'add', $directory );
            print LOG_FILE "COMMITTING DIR: $directory\n";
            $cvs_output .= cvs( 'commit', '-m _', $directory );
            print LOG_FILE $cvs_output;
            push @directories, $directory;
            print DIR_FILE "$directory\n";
        }
    }

    return @directories;

}





#################
sub check_files {
#################

    my %arg = @_;

    my $d = new IO::Dir $arg{dir};
    if ( defined $d ) {
        while ( defined( $_ = $d->read ) ) {
            next if $_ eq '..' || $_ eq '.' || $_ =~ m/CVS/;
            my $file = "$arg{dir}/$_";
            if ( -d $_ ) {
                print LOG_FILE "SKIPPING DIR: $file\n";
            }
            elsif (
                $_ =~ m/\.jpg$/ ||
                $_ =~ m/\.jpeg$/ ||
                $_ =~ m/\.gif$/ ||
                $_ =~ m/\.xls$/ ||
                $_ =~ m/\.doc$/ ||
                $_ =~ m/\.pdf$/ ||
                $_ =~ m/\.swf$/ ||
                $_ =~ m/\.cgi$/
            ) {
                my $cvs_output;
                print LOG_FILE "ADDING BINARY FILE: $file\n";
                $cvs_output = cvs( 'add', '-kb', $file );
                print LOG_FILE "COMMITTING FILE: $file\n";
                $cvs_output .= cvs( 'commit', '-m _', $file );
                print LOG_FILE $cvs_output;
            }
            elsif (
                $_ =~ m/\.htm$/ ||
                $_ =~ m/\.html$/ ||
                $_ =~ m/\.shtml$/ ||
                $_ =~ m/\.js$/ ||
                $_ =~ m/\.css$/ ||
                $_ =~ m/\.pl$/ ||
                $_ =~ m/^\.htaccess$/ ||
                $_ =~ m/^\..*passwd$/ ||
                $_ =~ m/^\.users$/ ||
                $_ =~ m/^\.txt$/
            ) {
                my $cvs_output;
                print LOG_FILE "ADDING FILE: $file\n";
                $cvs_output = cvs( 'add', $file );
                print LOG_FILE "COMMITTING FILE: $file\n";
                $cvs_output .= cvs( 'commit', '-m _', $file );
                print LOG_FILE $cvs_output;
            }
        }
        undef $d;
    }

}



Thanks to everyone who took the time to help me with this.

bassplayer