Create type-safe Collections in Java... with Perl

[Falkkin]

Background

Java's analog to the containers in C++'s STL (Standard Template Library) is the Collections framework -- a hierarchy of container classes, such as Vectors and Hashtables, that are used for holding various objects. However, Java does not include support for templates, which means that collections are not properly type-safe. This leads to three problems:

• 1) Since the accessing methods for containers return values of type Object, a cast is usually needed to do anything useful with the returned object.
• 2) You are allowed to add any arbitrary type of object to any Java container. If I've written an algorithm that expects a container to have a bunch of Shape objects, and someone has added a String to it, and I've not written special code to handle this case, the thread my algorithm's running on will probably die, due to a ClassCastException when I try to cast the String to type Shape.
• 3) Since basic data types aren't Objects, they can't be added to Java containers directly, but need to first be wrapped in an object.

So... what does this have to do with Perl?

To help remedy these problems, I'm working on a filter for Java code, called java-templater.pl. It allows usage of C++-like template syntax, and converts the code into real .java files, which can be understood by the compiler. Basically, it's "faking" templates by running the code through a preprocessor. So far, I've (at least partially) solved problems 1 and 2. With the templating system, casts are no longer needed, and an attempt to add a String to a vector of Something Which Is Not A String results in a compile-time error.

What's it look like?

Here's the complete source to Test.javat, a simple class that demonstrates java-templater.pl.

public class Test {
    public static void main(String[] args) {

        // Looks like C++...
        Vector<String> v1 = new Vector<String>();

        // works like Java...
        v1.add("Foo");
        v1.add("Bar");
        v1.add(1, "Baz");

        // ... but no casting needed when extracting elements.
        String baz = v1.elementAt(1);
        
        // Adding a mere Object to a Vector of Strings is an error,
        // and it's caught at compile-time.
        v1.add(new Object());
    }
}

The code uses a templated version of Java's Vector class. Specifically, it needs a Vector of String objects. In order for java-templater.pl to know what to do with this code, it needs a file named VectorTemplate.tmpl. Here's an excerpt of the file. (Monks who have been properly studying their scrolls of Better Homes and Monasteries will notice that this code uses the Decorator pattern to alter the behavior of Java's built-in Vector.)

public class __CLASSNAME__  {
    private java.util.Vector impl;

    public __CLASSNAME__() {
        impl = new java.util.Vector();
    } 

    public void add(int index, __TYPE1__ element) {
        impl.add(index, element);
    }

    public boolean add(__TYPE1__ o) {
        return impl.add(o);
    }

    public __TYPE1__ elementAt(int index) {
        return (__TYPE1__) impl.elementAt(index);
    }

    // The rest of java.util.Vector's interface here...
}

So, when template-filler.pl is run, it will generate a new class called StringVector. As you might expect, "__TYPE1__" from the template will get replaced by "String", and "__CLASSNAME__" will be replaced by "StringVector".

Finally, (gasp...) the Perl bit:

#!/usr/bin/perl -w

# java-templater.pl, version 0.01, 2001/06/13
# Makes type-safe Java classes from templates.
# Author: Colin McMillen ([e-mail])

=head2 TODO

* Better error tolerance (current code barfs on whitespace, among othe
+r things)
* Get nested templates working.
* Check to make sure the number of arguments within a template declara
+tion
  (like Foo<class1, class2>) matches the actual number of arguments ne
+eded
  to create a templated class of the appropriate type.
* Make into a module. (?)
* Support for templated classes of Java built-in types, such as int an
+d double.
* Support for recursively processing a directory structure's worth of 
+files.
* Support for putting template instantiations into a separate package,
  which would then be imported by files that use templates.
* Better documentation.
* Implement in Java (ack.)
* Creation of the proper templates to replace Java's entire Collection
+s hierarchy.
* Much, much more, I'm sure...

=cut

use strict;

my $VERSION = "0.01";
my @templates = qw(Vector);

# We need at least two command-line arguments (a directory and a
# .tmpl file) to do anything.
unless (@ARGV >= 2) {
  usageMessage();
  exit(0);
}

# Used to make sure we don't generate the same templated class twice.
my %createdClasses;

# Get the code directory and make it if needed.
my $codeDir = shift @ARGV;
mkdir $codeDir;

# Create all our .java files!
createTemplatedClasses($_) foreach @ARGV;

# Given a .javat file, parses the file, looking for uses of templates.
# Translates code which uses templates into valid Java and generates n
+ew
# .java files for every different templated class used by the .javat f
+ile.
sub createTemplatedClasses {
  my ($filename) = @_;

  # We make sure the filename ends with the proper extension so we don
+'t
  # accidentally try to parse non-templated files.
  die "Filename $filename must end in '.javat'" unless $filename =~ /\
+.javat$/i;

  # Read the file.
  local $/ = undef;
  open (CODE, $filename) or die "Could not open template file: $!";
  my $code = <CODE>;

  # For each type of template, look for template syntax.
  # Replace template syntax with valid Java code, and create new
  # templated classes.
  foreach my $template (@templates) {
    while ($code =~ s/$template<(.*?)>/getClassName($template, extract
+Types($1))/e) {
      my @types = extractTypes($1);
      createTemplate($template, \@types);
    }
  }

  # Write the file to the proper location.
  $filename =~ s/\.javat$/\.java/;
  print "Creating $codeDir/$filename...";
  open (CODE, ">$codeDir/$filename") 
      or die "Could not output to $codeDir/$filename: $!";
  print CODE $code;
  close(CODE);
  print " done.\n";
}

# Given a comma-separated list of class names, returns a list of the n
+ames.
sub extractTypes {
  my $argString = shift;
  my @types = split /,\s*/, $argString;
}

# Given a template and a list of Type arguments, 
# generates a descriptive class name.
sub getClassName {
  my ($template, @types) = @_;
  @types = map {
    stripPackagePart($_);
  } @types;
  return join("", @types) . $template;
}

# Given a class name, strips off the package qualifier and returns
# the result. (i.e. "java.util.Vector" -> "Vector")
sub stripPackagePart {
  my $name = shift;
  $name =~ s/.*\.(.*?)/$1/;
  return $name;
}

sub createTemplate {
  my ($template, $types_ref) = @_;
  my @types = @$types_ref;

  # Get the name of the class we'll be creating.
  my $newClassName = getClassName($template, @types);

  # Return if we've already created a class with this name.
  return if defined $createdClasses{$newClassName};

  # Prevent duplicate classes from occurring.
  $createdClasses{$newClassName} = 1;

  # Read the template from disk.
  print "Creating $codeDir/$newClassName.java...";
  local $/ = undef;
  open (TEMPLATE, $template."Template.tmpl") 
    or die "Could not open template file: $!";
  my $text = <TEMPLATE>;
  close TEMPLATE;

  # Create a hash that will be used to perform replacements 
  # on the template to generate the new class.
  my %replacements;
  $replacements{__CLASSNAME__} = $newClassName;
  for (my $i = 1; $i <= @types; $i++) {
    $replacements{"__TYPE".$i."__"} = $types[$i-1];
  }

  # Replace the text.
  replace(\$text, \%replacements);

  # Write the results to disk.
  open (OUTPUT, ">$codeDir/$newClassName.java")
    or die "Could not write new class to file: $!";
  print OUTPUT $text;
  close OUTPUT;
  print " done.\n";
}

# Generic replacement subroutine. Takes in a reference to a string of
# text and a reference to a hask of replacements.
# For every key in the hash, we attempt to replace the key with its
# associated value in the text.
sub replace {
  my ($text_ref, $replacements_ref) = @_;
  my %replacements = %{ $replacements_ref };
  foreach (keys %replacements) {
    $$text_ref =~ s/$_/$replacements{$_}/g;
  }
}

# Helpful usage message.
sub usageMessage {
  print <<"DONE";
Usage: java-templater.pl code_dir file1.javat file2.javat ...

Makes type-safe Java classes from templates.
code_dir is the directory in which to place generated .java files.
file*.javat are normal Java source files that use the special template
+ syntax.

Version $VERSION by Colin McMillen (mcmillen\@tc.umn.edu)
DONE
}

To run the whole shebang, do something like this:

./java-templater.pl ./generated_code Test.javat
cd generated_code
javac *.java
java Test

The code has several limitations, as noted in the TODO list at the top, but it's not bad for a couple hours' worth of work. It was fun to write too. :) Let me know if you have any suggestions, or if you think this is/isn't useful. I'm pondering starting a project on sourceforge to work on a fully-templated version of the Collections framework; let me know if you'd be interested in helping at all.

The more astute readers may have noticed that I'm not using the Template Toolkit (or a similar) in my code. The main reason for this is that I figure I'll eventually be rewriting the code in Java... and said libraries do not exist for Java (AFAIK, if you know of some, please let me know ;)).

The full version of the VectorTemplate.tmpl file is available here for now -- I'll probably be creating more .tmpl files in the near future and putting them online at that address.

Enjoy!
- colin

[ariels]

Completely unPerl-related, but relevant. Take a look at Philip Wadler's Generic Java (aka "GJ"), which sounds similar, but lives in the compiler rather than a preprocessor.

[coreolyn]

I hate to throw any chance of showers on your great work, as I really think that there is much room for Perl to shine as a Java support language, and I can't find a link at the moment, but at Java1 it was announced that Java 1.4 will support Generics. A google search under "Java Generics" will yield more about about the subject. I could be all wrong here (This isn't my strong suit and I haven't really done my homework before posting -- shame on me) but it was my understanding that generics are they type of thing your doing here.

Sun's renaming of wheels only makes it harder to get into thier code.

coreolyn

[John M. Dlugosz]

Many years ago, before C++ templates were implemented, I wrote a preprocessor for that, so I could play with them ahead of time. It didn't look anything like modern template syntax—I used the @ symbol, which is not used anywhere else in C++, for stuff that my program manipulated. It wasn't as limited as modern templates, either, as it was a more general-purpose macro system.

I see you're not using a grammar parser at all, but just looking for a pattern on each line. Won't that be very difficult to do "right" for quotes, comments, etc.?

Why does it look only for known template types? Why not first recognise the template syntax, then check to see if that template is known (such as in a like-named file)?

—John