A multilingual solution for PERL CGI web applications

[Zzenmonk]

Despite an incredible amount of literature on this topic, I could not find a proper solution for this issue. Accordingly I post my one here.

I had to realize a relatively complex web application. This application bases on CGI.pm and mod_perl for performance reasons and supports several languages. I wanted to have one single code for the logic of the application. An additional requirement was be to be able to add languages easily. After reading a lot about solutions involving templates or dictionaries (typically gettext), I figured out using PERL language packages would fulfill most of my requirements. The problem was to load the appropriate language module at application start up time and to allow the user to change the language for example for print outs. Here is the solution I worked out.

First make a set of simple language packages with names like EN.pm, FR.pm, DE.pm. Each of these modules holds hashes with the texts for the application's forms. For this solution to work properly, the hash names must be in lower case. Example for a log in form:

- French in FR.pm: $login{msg1} = "Identifiant";

- German in DE.pm: $login{msg1} = "Benutzername";

- English in EN.pm: $login{msg1} = "User name";

Next use the following at the beginning of each language package to export the content. Example for EN.pm:

foreach $key (sort keys %EN::) { if ( $key !~ /[A-Z]+/ ) { push @EXPORT, "\%$key"; } }

This will export all hashes in lower case and avoid for example the EXPORT array. Next you will have to import the correct languages package in your cgi script with a statement like use [Modulename];.

Unfortunaltely you can hardly load modules dynamically and can hardly reference a variable in a module dynamically. Constructs like use $language; or ${language}::login{msg1}; will not work.

Despite lot of posts on this topic, I haven't found a solution to solve this issue. This means so or so I will not be able to load the languages dynamically but will have to manage the supported languages in the code of the application's logic.

My solution bases on a language variable I called syslang. This variable is either set by the calling cgi script or defined by the selected language of the client's browser. I begin my cgi scripts the following way and include a global language variable:

    # ----------------------------------------------------------------
+----------------
    # Script header
    # ----------------------------------------------------------------
+----------------
    # Loading perl modules
    my $start = time();
    use FindBin qw($Bin);
    use File::Basename;
    use Cwd;
    use CGI qw/:standard *table tr td select/;
    use CGI::Carp;
    use CGI::Session;
    use Data::Dumper;
    use strict;
    
    # Our global language variable
    our $syslang = undef;

Next I add the path to my application or packages to @INC array. My application packages are always in a lib directory. This is also the location of the language packages. For the solution to work the @INC array must be enhanced with a begin block or the script will not compile:

    # Notice: this will work with Apache, cheap web-server might be a 
+serious issue!
    # a) Detect if we are called by the web-server or not
    # b) Set the path accordingty
    BEGIN {
        # Adding application lib path
        my $libdir = undef;
        if ( not defined($ENV{SERVER_NAME}) ) { $libdir = getcwd; }
        else { $libdir = File::Basename::dirname($ENV{SCRIPT_FILENAME}
+); }
        # I know.... but sometimes I am lazy!
        chdir "$libdir/../lib";
        $libdir = getcwd;
        if ( not grep(/$libdir/, @INC) ) { push (@INC, $libdir); }
        undef $libdir;
    }

Now we need to manage the languages. We need to load the package with the correct texts, knowing the user can change the language of the application. Because of mod_perl we need to unload the language modules first and the reload the correct module to force Apache to recompile the script. Here my solution for this:

    # Setting the language 
    # This one is to fool the compiler! Don't ask!
    use EN;

    # Testing if the language has allready be set or using the one def
+ines by the browser
    # frdlang is my cgi language parameter
    my $q = new CGI;
    if ( $q->param("frdlang") ) { $syslang = uc($q->param("frdlang"));
+ }
    elsif ( not $syslang ) { $syslang = uc(substr($ENV{HTTP_ACCEPT_LAN
+GUAGE}, 0, 2)); }

    # Unloading language modules to force recompiling under mod_perl
    foreach my $lang ( grep(/DE.pm$|FR.pm$|EN.pm$/, keys(%INC)) ) { de
+lete $INC {$lang}; }

    # Loading apropriate language module
    if ( uc($syslang) =~ /^DE/ ) { require DE; DE->import(); }
    elsif ( uc($syslang) =~ /^FR/ ) { require FR; FR->import(); }
    elsif ( uc($syslang) =~ /^EN/ ) { require EN; EN->import(); }

Now we might have the issue that some packages return some part of the HTML form, tipically the menus. The language of these parts musst also be changed each time the user switches the language. So we need to reload all the modules impacted. Here my solution for this:

    # Unloading the application language related module to force recom
+pile under mod perl
    foreach my $module ( grep(/utils.pm$|install.pm$|login.pm$/, keys(
+%INC)) ) { delete $INC{$module}; }
    
    # setting languages for each modules to load
    $utils::syslang = $syslang;
    $install::syslang = $syslang;
    $login::syslang = $syslang;
    
    # Loading language dependent modules
    require utils; utils->import();
    require install; install->import();
    require login; login->import();

    # ... your cgi code.

This solution has beed tested extensively however any suggestion for something more decent is welcome.

K.

[kbrannen]

In our product we use a DB to hold the translation tokens. However, if you're going to do it with files, why not just create a hash for each language, use Freeze then write it to store it, and later pull it in and Thaw it?

I'd probably create a CGI app to maintain the data, rows for each token, a column for each language, and when saving it, each column is a hash with the row name as the key and that's what's frozen for the real app to use later. Use the same hash for each one so you have the same name (e.g. %tokens).

Then your normal app can slurp in the frozen hash by opening the needed language file and it has all the correct token values. Then you can substitute $tokens{msg1} or whatever you need without worrying about the language and off you go.

In fact, once you have your tokens in memory and assuming you have a unique pattern for them (e.g. [[msg1]]), it's a matter of doing something like:


$page =~ s/\[\[(\w+)\]\]/$tokens{$1}/ge;

and you're good to go. No need to manipulate @INC or anything else tricky.

That also means that adding new languages is easy. Just create an empty file for the new language in your language dir, your CGI editing app will see it and create a new column of empty cells, then when you've filled it out and save, you'll get a new frozen hash in it's own file. If you want to remove a language, just remove the file.

[Anonymous Monk]

Store your language specific strings in a database and use templates with place holders for the page content. Before rendering the page just do a mass replacement of the place holders with the proper strings. Then you can provide a tool for translators to make changes to the strings in the database without mucking around in your .pm files.

[Zzenmonk]

This is definitely also a solution. However it requires a database which needs maintenance. In my case a database is not an appropriate storage solution because the system has to cope with text elements. The best way to find text elements is to store them as files and index the files with web search engine. So the implementation of the whole system is easier this way. In addition I noticed this solution is damned fast. This is due to the fact I replace all texts in one shot.

K