Beefy Boxes and Bandwidth Generously Provided by pair Networks
XP is just a number
 
PerlMonks  

comment on

( #3333=superdoc: print w/replies, xml ) Need Help??
A couple of weeks ago I released Data::Sync. The basic idea behind the module is to construct a simple open source metadirectory. This makes 'copy from one database to another' applications very quick and easy to develop. An example is set out below:

You have two databases as follows:

Database1:

NAME VARCHAR(30), POSTALADDRESS VARCHAR(50), TELEPHONE VARCHAR(15), CUSTOMERNO NUM

Database2:

FULLNAME VARCHAR(30), ADDRESS VARCHAR(100), HOMEPHONE VARCHAR(20), OFFICEPHONE VARCHAR(20)

and you want to copy name, address and phonenumber for those entries from database1 whose customer number is less than 1000. The code would look something like this (assuming SQLite for convenience)

use strict; use Data::Sync; use DBI; my $db1 = DBI->connect("DBI:SQLite:dbname=db1"); my $db2 = DBI->connect("DBI:SQLite:dbname=db2"); my $sync = Data::Sync->new(); $sync->source($db1,{select=>"SELECT NAME,POSTALADDRESS,TELEPHONE FROM +sourcetable WHERE CUSTOMERNO < 1000"}); $sync->target($db2,{table=>"targettable",index=>"FULLNAME"); $sync->mappings(NAME=>"FULLNAME", POSTALADDRESS=>"ADDRESS", TELEPHONE=>"HOMEPHONE"); $sync->run();

This will copy all records matching the select statement from the source database to the target database. If the entry exists, it will be updated - if it doesn't exist it will be created.

Nothing is done until the 'run' method is called - that calls the read, mappings, buildattributes, transforms, and write methods in turn (see below for transforms and buildattributes).

But perhaps the name attribute is formatted differently between the databases. Perhaps db1 has "Firstname Lastname", whereas db2 has "Lastname, Firstname". You can overcome this with a transformation:

$sync->transformations(FULLNAME=>'s/(\w*?)\s+(\w*?)/$2,$1/');
(You can pass a single quoted string representation of a regex, a coderef or a string in here - strings are for existing defined transformations like 'stripnewlines', 'stripspaces' etc - see the perldocs. Note that 'transforms' acts on the MAPPED names of the attributes - i.e. the names in the target, not the names in the source).

You can also specify an LDAP handle rather than a DBI handle for source or target, in which case the syntax changes to reflect the difference. The perldocs detail this (and it's more specialised, so I've omitted discussion here), but there's one point that's worth illustrating:

Writing to an LDAP directory requires a DN (Distinguished Name) to identify the record, and an objectclass. Reading from DBI to LDAP, you're unlikely to have either, so the 'buildattributes' function allows you to create them:

$sync->buildattributes(dn=>'cn=%NAME%, ou=container,dc=testorg,dc=net' +, objectclass=>'organizationalPerson');

Once created, you can use transformations on this attribute like any other, so more complex processing can be done.

Rationale

This is all analogous to a number of commercial products: Critical Path IMD, Maxware DSE, IBM Metamerge among others. Implementing these types of systems is how I normally earn my keep. I figured that given my fondness for perl I was going to end up implementing something like this eventually, so decided to do it while I have free time between projects - neatly avoiding implementing it in a project and hence being unable to release/reuse it. The upshot of that is that unlike many CPAN modules, this code, although thoroughly tested with Test is not in live use. AFAIK anyway - someone might have adopted it in the last couple of weeks I suppose. There are undoubtedly criticisms and comments that can be raised against the code as well as the syntax etc. All comments are welcome.

I'm currently at work on a GUI for job definition, and a standalone "run this saved job" script to allow sync jobs to be created and used without writing any perl code, but I'm polling at this point for opinions and feedback - any functionality people would like to see included or any other comments you may have?

--------------------------------------------------------------

$perlquestion=~s/Can I/How do I/g;


In reply to RFC: Data::Sync by g0n

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post; it's "PerlMonks-approved HTML":



  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.
  • Log In?
    Username:
    Password:

    What's my password?
    Create A New User
    Chatterbox?
    and the web crawler heard nothing...

    How do I use this? | Other CB clients
    Other Users?
    Others exploiting the Monastery: (8)
    As of 2020-05-26 18:17 GMT
    Sections?
    Information?
    Find Nodes?
    Leftovers?
      Voting Booth?
      If programming languages were movie genres, Perl would be:















      Results (150 votes). Check out past polls.

      Notices?