Beefy Boxes and Bandwidth Generously Provided by pair Networks
Your skill will accomplish
what the force of many cannot
 
PerlMonks  

ipod-backup

by Ctrl-z (Friar)
on Jun 29, 2005 at 09:38 UTC ( #470942=sourcecode: print w/replies, xml ) Need Help??
Category: Audio Related Programs
Author/Contact Info Ctrl-z
Description: Utility to resynchronise iTunes with tracks orphaned on an iPod, or alternatively do a full iPod backup. Big caveats - see POD (err, not iPod) before using

=pod

=head1 NAME

iPod-backup: reclaim tracks orphaned on an iPod 

=head1 SYNOPSIS

    perl ipod-backup.pl -do MODE -out DIR -mnt DRIVE [-top DIR]

    -out    Output directory for any reclaimed files
    -mnt    iPod mount path or drive
    -top    iTunes top-level directory
    -do     perform one of the following operations:

            stat    print iPod vs iTunes statistics (default)
            sync    reclaimed tracks not in iTunes
            bak     backup full iPod

=head1 DISCUSSION

Man, dont even get me started on a rant about Apple software...

When disk files are lost, iTunes helpfully does nothing then silently 
removes them from your iPod next time you plug it in. While the latter
+ 
can be avoided by disabling automatic sync'ing, there is no way to 
import tracks on the iPod back into iTunes...ho hum

This script checks your iPod's database against iTune's and imports an
+y 
tracks not available in iTunes back onto the harddisk.

It can also do a full backup of the iPod in a human friendly manner, i
+e:

    /Artist/Album/Track

rather than Apple's quirky 

    /F<DIGITS>/<ARBITRARY TRUNCATED FILENAME>

=head1 BUGS & CAVEATS

This is quick and dirty programming...use at your own risk.

In particular, the iPod database is a binary format. Rather than caref
+ully 
reverse engineer by hand, I just weild the Swiss Army Chainsaw and hac
+k off 
what I dont need. Your mileage may vary, as they say.

Only tested on Windows XP. Linux requires 2.6 kernel or a FAT32 format
+ted 
iPod. You would probably be better off looking at Gnupod for that.

=cut


use strict;
use utf8;

usage() if @ARGV % 2;

my %ARGS  = @ARGV;
my @KEYS  = (undef, "Name", "Path", "Album", "Artist", "Genre", "Kind"
+, undef, "Comments");
my $TYPE  = qr/\.(?:mp3|aux)/i; 
my @STAT  = ("sync'd","iPod","iTunes");
my $IPOD  =  1;
my $SYNC  =  0;
my $DISK  = -1;

my $mode          = $ARGS{-do}  || "stat";
my $ipod_temp     = $ARGS{-out} || "./ipod-temp";
my $ipod_mount    = $ARGS{-mnt} || usage();
my $itunes_root   = $ARGS{-top};

my $ipod_itunesdb = "$ipod_mount/iPod_Control/iTunes/iTunesDB";
my $ipod_sysinfo  = "$ipod_mount/iPod_Control/Device/SysInfo";
my $ipod_music    = "$ipod_mount/iPod_Control/Music";
my $itunes_lib    = "$itunes_root/iTunes Music Library.xml";
my $itunes_music  = "$itunes_root/iTunes Music";

die "'$ipod_itunesdb' does not exist\n"     unless -f $ipod_itunesdb;
die "'$itunes_root' is not a directory\n"   unless !$itunes_root || -d
+ $itunes_root;
die "'$itunes_lib' does not exist\n"        unless !$itunes_root || -f
+ $itunes_lib;

if($mode eq "stat"){
    stat_ipod($ipod_itunesdb, $itunes_lib);
}
elsif($mode eq "bak"){
    import_ipod($ipod_temp, $ipod_itunesdb);
}
elsif($mode eq "sync"){
    import_ipod($ipod_temp, $ipod_itunesdb, $itunes_lib);
}
else{
    usage();
}

#
#
#
sub usage
{
    my $src = fslurp( fopen($0) );
    print "\n", $1,"\n\n",$2,"\n\n"
        if $src =~ /=head1 NAME\s*?(.*?)\s*?=head1 SYNOPSIS\s*?(.*?)\s
+*?=head1 DISCUSSION/s;
    exit 0;
}

#
# Print shared and unique tracks on iPod and iTunes
#
sub stat_ipod
{
    my $ipod  = shift || die "Error: No path to iTunesDB\n";
    my $local = shift || die "Error: No path to XML library\n";
    my $dup   = {};
    my ($disk, $pod, $syn);

    $ipod  = parse_itunesdb($ipod);
    $local = parse_itunes_library($local);

    $dup->{ $_->{Artist}." - ".$_->{Album}." - ".$_->{Name} }++ foreac
+h @$ipod;
    $dup->{ $_->{Artist}." - ".$_->{Album}." - ".$_->{Name} }-- foreac
+h @$local;

    foreach(sort keys %$dup){
        my $s = $dup->{$_};
        printf "%-70s %-6s\n", (length $_ < 70) ? $_ : substr($_,0,67)
+."...", $STAT[$s];
        ($s == $DISK) ? $disk :
        ($s == $IPOD) ? $pod : $syn += 1;
    }

    my $format = "%-6s: %5d / %-5d\n";
    print "\n\n";
    printf $format, "iPod",  $pod,  scalar(@$ipod);
    printf $format, "iTunes",$disk, scalar(@$local);
    printf $format, "Sync'd",$syn,  scalar(@$ipod)+scalar(@$local);
}

#
# Import tracks on an iPod
# If $local is provided, only imports those not currently in iTunes
# else, imports the lot for backup
#
sub import_ipod
{
    my $dir   = shift || "./ipod-temp";
    my $ipod  = shift || die "Error: No path to iTunesDB\n";
    my $local = shift;

    my $dup   = {};
    my @err   = ();
    my $prog  = 0;
    my $totl  = 0;

    $ipod  = parse_itunesdb($ipod);
    $local = parse_itunes_library($local) if $local;
    $local = [] unless $local;

    print "(ipod)  scanned tracks: ",scalar(@$ipod),"\n";
    print "(local) scanned tracks: ",scalar(@$local),"\n";

    $dup->{ $_->{Artist}."/".$_->{Album}."/".$_->{Name} } = $_    fore
+ach @$ipod;
    $dup->{ $_->{Artist}."/".$_->{Album}."/".$_->{Name} } = undef fore
+ach @$local;
    (defined $dup->{$_}) ? $totl++ : delete $dup->{$_} foreach(keys %$
+dup);

    mkdir($dir) unless -d $dir;

    $|++;
    foreach(sort keys %$dup)
    {
        my $track = $dup->{$_} or next;
        my $path  = $track->{Path};
        my $ext   = $1 if $track->{Path} =~ /\.(\S+)$/;

        my $in    = "$ipod_mount$track->{Path}";
        my $d1    = "$dir/$track->{Artist}";
        my $d2    = "$d1/$track->{Album}";
        my $out   = "$d2/$track->{Name}.$ext";

        my $ui    = $track->{Artist}." - ".$track->{Album}." - ".$trac
+k->{Name};

        mkdir($d1) unless -d $d1;
        mkdir($d2) unless -d $d2;

        printf "%-11s %-65s", ++$prog."/".$totl, 
               (length($ui) < 60) ? $ui : substr($ui,0,60)."...";

        $@ ='';
        eval{
            fcopy($in, $out) unless( -f $out && (stat($in))[7] == (sta
+t($out))[7] );
            print "ok\n";
        };
        if($@)
        {
            push @err, "$out: $@";
            print "--\n";
        }
    }

    print STDERR "\nErrors:\n";
    print @err ? join('\n', @err) : "none", "\n";
}

#
# BRRRRRRRR!-BRRRRRRR!
#
sub parse_itunesdb
{
    my $f = fopen(shift);
    my $s = fslurp($f);
    my $fields;

    my @records = ();
    my $record  = {};

    foreach( split(/mhod/, $s) )
    {
        my ($eor, $t, $k);
        next unless length $_;

        s/\000\000+/ /sgo;
        s/\000//sgo;

        # this doesnt work...???
        # s/^(..(.).(.).....)//;
        # $c = escape($1);
        # $l = chr($2);
        # $t = $K[ord($3)];

        $t = substr($_, 0, 10);
        $t = $KEYS[ord(substr($t, 4, 5))];
        $_ = substr($_, 10, length $_);        

        next if !$t;

        if($t eq "Path")
        {
            $eor++; 
            $_ = $1 if m/^(.*?$TYPE)/;
            $_ =~ s/:/\//sg;
        }

        # TODO: Some final checks for non-printable pathnames...

        $record->{$t} = $_ ;
        if($eor)
        {
            $record->{Artist} = "Unknown"  unless defined $record->{Ar
+tist};
            $record->{Album}  = "Unknown"  unless defined $record->{Al
+bum};
            push @records, $record         if     defined $record->{Na
+me};
            $record = {};
        }
    }
    return \@records;
}

#
# YAHRXP...
#
sub parse_itunes_library
{
    my $lib = fopen(shift);
    my $ok  = 0;

    my @records = ();
    my $record  = {};

    while(<$lib>)
    {
        chomp;
        last  if m/^\s*?<key>Playlists<\/key>/;
        $ok++ if m/^\s*?<key>Tracks<\/key>/;
        next unless $ok;

        if(m/^\s*?<key>(.*?)<\/key><(\S+)>(.*?)<\/\2>/)
        {
            $record->{$1} = xml_decode($3) ;
        }
        elsif(m/^\s*?<key>(\d+)<\/key>/)
        {
            $record->{Artist} = "Unknown"  unless defined $record->{Ar
+tist};
            $record->{Album}  = "Unknown"  unless defined $record->{Al
+bum};
            push @records, $record         if     defined $record->{Na
+me};
            $record = {};
        }
    }
    return \@records;
}

#
# decode filename urls in XML
#
sub url_decode 
{
    my $str = shift;
    $str =~ tr/+/ /;
    $str =~ s/%(..)/pack("C",hex($1))/eg;
    return $str;
}

#
# TODO: decode XML-entities in fields
#
sub xml_decode 
{
    my $str = shift;
    $str =~ s/\&\#(\d+)\;/chr($1)/esg;
    return $str;
}

#
#
#
sub fcopy
{
    my $in  = shift;
    my $out = shift;
    my $buf = shift || 1024;
    my $fin  = fopen($in);
    my $fout = fopen(">".$out);
    my ($b, $r, $w);
    
    while($r = sysread($fin, $b, $buf))
    {
        $w = syswrite($fout, $b, $r);
        last if $r < $buf;
    }
}

#
#
#
sub fopen
{
    my $path = shift or die "fopen: no path\n";
    my $f;
    open $f,$path or die "$path: $!";
    binmode $f;
    return  $f;
}

#
#
#
sub fslurp
{
    my $f    = shift;
    local $/ = undef;
    return <$f>;
}
Replies are listed 'Best First'.
Re: ipod-backup
by shiza (Hermit) on Oct 25, 2005 at 22:25 UTC
    Well, re-installed the OS on my wife's computer this past weekend. Her iPod was basically her backup for her music because I thought (as insane as it sounds) that iTunes could suck the music back in after the install. It doesn't.

    Yesterday evening I hear screams coming from our office. iTunes apparently doesn't like a dirty iPod and decided to clean it up which entailed deleting 7 Gigs of music and photos. Thanks Apple!

    I immediately had my wife call Apple to see if they could help her restore her data. Well, they wouldn't even answer a question until we gave them a credit card number?! Keep in mind it's still under a 1 yr warranty. Thanks again Apple!

    My next step was to do some data recovery. I d/l some software and ran it. It found the missing songs and I restored them to my hard drive. Whew, i'm a hero now! Wrong. The restored files are all screwed up. Mis-labled songs, overlapping songs, etc...

    CTRL-Z, do you think your script can help me with my situation?


    P.S. - IMO Apple purposely weaved this inconvenience into iTunes...LAWSUIT!? I'm just kidding about the lawsuit, but seriously, why leave out simple and obviously needed functionality. If they didn't do it on purpose, maybe they need to re-structure their QA dept. I'm betting it was purposeful though.
      The restored files are all screwed up. Mis-labled songs, overlapping songs, etc...

      On my iPod, the tracks are stored in a seemingly random grouping in several folders - like some kind of hash table. The track names are truncated, but the mp3 data is not "overlapped" (is that what you meant?). Maybe your iPod used the Apple HFS+ filesystem? If that is the case, all I can do is point you at the Gnupod/Linux kernel 2.6 references in the POD. Sorry.

      If the mp3 data is ok, then you might be able to use the import_ipod function to read the binary iTunesDB restored off the ipod. Just point $ipod_itunesdb at the restored iTunesDB. The original caveats still apply though...

      Check out the Gnupod link - it is written in perl, by people who have actually studied the iPod. They could be able to advise you further.

      Good luck, man. /msg me if you need more help.




      time was, I could move my arms like a bird and...
Re: ipod-backup
by Anonymous Monk on Apr 24, 2006 at 14:05 UTC
    Very cool, could I request fleshing out the TODO's though? I half hacked the non-printing character TODO but it's still a bit wonky (my code).
      Hi Anonymous Monk,

      Perhaps if you posted the code you have somone may help you with it.

      Martin
Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: sourcecode [id://470942]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others perusing the Monastery: (5)
As of 2020-05-27 06:30 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?
    If programming languages were movie genres, Perl would be:















    Results (152 votes). Check out past polls.

    Notices?