Beefy Boxes and Bandwidth Generously Provided by pair Networks
more useful options

Parsing Gutenberg Catalog Index

by hacker (Priest)
on Aug 30, 2004 at 05:04 UTC ( [id://386850]=perlquestion: print w/replies, xml ) Need Help??

hacker has asked for the wisdom of the Perl Monks concerning the following question:

I've been thrown a curveball in a community project I signed myself up for.. a linguist contacted me with the intent of rolling through all of Project Gutenberg, importing every ebook found, and doing some analysis of the contents of the texts, for his paper.

He found me, because I suggested a project to convert all of Project Gutenberg to Plucker format, in a professional, scalable, automated fashion.

The combined talents of a "Professional Screen Scraper" and a Linguist, would be ideal here, which is how the two of us managed to hook up on this project.

But we ran into a snag.. there is ZERO consistency in the Project Gutenberg etexts, after editing them by hundreds of volunteers, each with their own ideas. Project Gutenberg's Distributed Proofreaders is a good first step, but it isn't quite there yet.

How does this relate to Perl? Well, that would be our engine, to roll through each book, unzip it, catalog it, store it in MySQL, and output it into a format we can grok with other tools.

Our first step is to take the Gutenberg Master Index, and parse it for the title, filename, document number, and so on, to populate the initial tables in the database. From there, we'll query the db and fetch each ebook in succession to perform our tests, analysis, import, and conversion of the documents.

But we're stuck on the index. The only thing that seems to remain constant in there, is the document number, a 5-digit number in the right-most column. It is always at the end of the very first line of text describing a new book entry.

What is the best way to approach parsing this? I don't even know where to begin to scan this file for the right textual members in a way that doesn't overlap another previous or following entry, during our import.

I've seen the Glossarymaker and this entry from SpritusMaximus, but neither of them seem to do what we need.

Pseudo-examples or real examples we can use as a starting point would be great. Thanks again, everyone.

Replies are listed 'Best First'.
Re: Parsing Gutenberg Catalog Index
by kvale (Monsignor) on Aug 30, 2004 at 05:30 UTC
    For the other readers, here is a snippet of the index:
    Anna Karenina, by Lev Nikolaevica Tolstoi + 13214 [Language: Dutch] Night Before Christmas & Other Popular Stories For Children, by Variou +s 13213 The Wild Olive, by Basil King 13212 The Pearl, by Sophie Jewett 13211 El Comendador Mendoza, by Juan Valera 13210 [Subtitle: Obras Completas Tomo VII] [Language: Spanish]
    It seems that there is a good bit of structure here. Each new entry starts on a new line. The title and author are separated by /, by/. The ID is at the end of the first line of the entry. Combining these, a first stab at a regexp would be
    $line =~ /^(\w.*?), by (.*?)\s+(\d+)$/; $author = $1; $title = $2; $id = $3;


      Woudlnt it just make sense to use their XML formatted catalog? ...

        I would second that, given that the RDF looks like:

        <rdf:Description rdf:ID="etext13218"> <dc:publisher>&pg;</dc:publisher> <dc:title rdf:parseType="Literal">Don Orsino</dc:title> <dc:creator>Crawford, F. Marion (Francis Marion) (1854-1909)</dc:cre +ator> <dc:language>en</dc:language> <dc:created>2004-08-19</dc:created> <dc:rights rdf:resource="&lic;" /> </rdf:Description>

        Then all you need is something trivial like this to create a file ready for a MySQL 'load data local infile .....'

        #!/usr/bin/perl local $/ = "\n\n"; open RDF, $ARGV[0] or die $!; while(<RDF>){ next unless m/<rdf:Description rdf:ID="etext(\d+)"/; my $id = $1; next unless m/<dc:title[^>]+>([^<\n]+)</; my $title = $1; next unless m/<dc:creator>([^<\n]+)</; my $author = $1; $title =~ s/\s+/ /g; $author =~ s/\s+/ /g; print "$id\t$title\t$author\n"; }



      Looking a little closer and also in older index files i found this.
      *****A "C" Following a Project Gutenberg eBook Number Indicates Copyri +ght**** *****A "*" Following a Project Gutenberg eBook Number Indicates Reserv +ed **** [snip] The Life of John Ruskin, by W. G. Collingwood + 13076 A Hero and a Great Man, by Francis Kruckvich + 13075C [Illustrator: Fritz] Punch, Vol. 100, February 7, 1891, Ed. by Sir Francis Burnand + 13074 [snip] Feb 1995 Moon and Sixpence by Somerset Maugham [Maugham #1][moonaxxx.x +xx] 222 Feb 1995 The Return of Sherlock Holmes [Magazine Edition] [rholmxxb.x +xx] 221B Feb 1995 The Secret Sharer, by Joseph Conrad [Conrad #2] [ssharxxx.x +xx] 220
      I did not found the meaning of the 'B' though.

        221B - as in 221B Baker Street. It could mean something else, but I think it's somebody's attempt at humor.

        Wonder how this project turned out?

        But God demonstrates His own love toward us, in that while we were yet sinners, Christ died for us. Romans 5:8 (NASB)

Re: Parsing Gutenberg Catalog Index
by PodMaster (Abbot) on Aug 30, 2004 at 05:46 UTC
    Not perfect, but its a good start
    use strict; use warnings; @ARGV = 'GUTINDEX-2004.txt' unless @ARGV; my $file = shift or die "Usage: $0 GUTINDEX-2001.txt"; my @data; { open IN, '<', $file or die "Error: Can't open $file : $!"; my $start_parsing; local $_; while(<IN>){ next if /^\s*$/; last if /^End\s+of\s+\Q$file\E/; chomp; if( $start_parsing ){ if(/^(\w+.*?) by (.+?)\s+(\d{5}.?)$/){ my( $title, $author, $id ) = ( $1, $2, $3 ); $data[$#data+1]->{id} = $id; $data[$#data]->{title} = $title; $data[$#data]->{author} = $author; } else { for( /\[(.+?)(\])?/g ){ my $foo = $1; unless( $2 ){ while(<IN>){ if( /(.+?)\]/ ){ $foo .= $1; last; } else { $foo .= <IN> } } } if( $foo =~ /\[?(\S+):(.+)\]?/s ){ $data[$#data]->{$1} = $2; } elsif( $foo =~ /,\s\d+$/s ){ $data[$#data]->{Date} = $foo; } } } } # elsif( /^Title\s+and\s+Author/ ){ elsif( /^\Q~ ~ ~ ~ Posting Dates for the below eBooks\E/ ){ $start_parsing++; } } } use Data::Dumper; print Dumper( \@data );
    What I noticed is that the first entry in GUTINDEX-2004.txt doesn't have a closing [ for [Subtitle:, which is a bug in the file.

    MJD says "you can't just make shit up and expect the computer to know what you mean, retardo!"
    I run a Win32 PPM repository for perl 5.6.x and 5.8.x -- I take requests (README).
    ** The third rule of perl club is a statement of fact: pod is sexy.

Re: Parsing Gutenberg Catalog Index
by DrHyde (Prior) on Aug 31, 2004 at 08:00 UTC
    I have some code which makes an attempt at this and which (mostly) works, as part of my ongoing project to convert all of PG to Palmdoc. Email me, and I'll send you a copy.
      Dear Sir, I am doing a project to convert Gutenberg books into webpages. If you don't mind, could you please share your codes with me? Thanks, Mark my email:
        All that I've got now is at

Log In?

What's my password?
Create A New User
Domain Nodelet?
Node Status?
node history
Node Type: perlquestion [id://386850]
Approved by blokhead
Front-paged by Old_Gray_Bear
and the web crawler heard nothing...

How do I use this?Last hourOther CB clients
Other Users?
Others goofing around in the Monastery: (4)
As of 2024-06-22 13:47 GMT
Find Nodes?
    Voting Booth?

    No recent polls found

    erzuuli‥ 🛈The London Perl and Raku Workshop takes place on 26th Oct 2024. If your company depends on Perl, please consider sponsoring and/or attending.