Beefy Boxes and Bandwidth Generously Provided by pair Networks
There's more than one way to do things

Parsing Gutenberg Catalog Index

by hacker (Priest)
on Aug 30, 2004 at 05:04 UTC ( #386850=perlquestion: print w/replies, xml ) Need Help??
hacker has asked for the wisdom of the Perl Monks concerning the following question:

I've been thrown a curveball in a community project I signed myself up for.. a linguist contacted me with the intent of rolling through all of Project Gutenberg, importing every ebook found, and doing some analysis of the contents of the texts, for his paper.

He found me, because I suggested a project to convert all of Project Gutenberg to Plucker format, in a professional, scalable, automated fashion.

The combined talents of a "Professional Screen Scraper" and a Linguist, would be ideal here, which is how the two of us managed to hook up on this project.

But we ran into a snag.. there is ZERO consistency in the Project Gutenberg etexts, after editing them by hundreds of volunteers, each with their own ideas. Project Gutenberg's Distributed Proofreaders is a good first step, but it isn't quite there yet.

How does this relate to Perl? Well, that would be our engine, to roll through each book, unzip it, catalog it, store it in MySQL, and output it into a format we can grok with other tools.

Our first step is to take the Gutenberg Master Index, and parse it for the title, filename, document number, and so on, to populate the initial tables in the database. From there, we'll query the db and fetch each ebook in succession to perform our tests, analysis, import, and conversion of the documents.

But we're stuck on the index. The only thing that seems to remain constant in there, is the document number, a 5-digit number in the right-most column. It is always at the end of the very first line of text describing a new book entry.

What is the best way to approach parsing this? I don't even know where to begin to scan this file for the right textual members in a way that doesn't overlap another previous or following entry, during our import.

I've seen the Glossarymaker and this entry from SpritusMaximus, but neither of them seem to do what we need.

Pseudo-examples or real examples we can use as a starting point would be great. Thanks again, everyone.

Replies are listed 'Best First'.
Re: Parsing Gutenberg Catalog Index
by kvale (Monsignor) on Aug 30, 2004 at 05:30 UTC
    For the other readers, here is a snippet of the index:
    Anna Karenina, by Lev Nikolaevica Tolstoi + 13214 [Language: Dutch] Night Before Christmas & Other Popular Stories For Children, by Variou +s 13213 The Wild Olive, by Basil King 13212 The Pearl, by Sophie Jewett 13211 El Comendador Mendoza, by Juan Valera 13210 [Subtitle: Obras Completas Tomo VII] [Language: Spanish]
    It seems that there is a good bit of structure here. Each new entry starts on a new line. The title and author are separated by /, by/. The ID is at the end of the first line of the entry. Combining these, a first stab at a regexp would be
    $line =~ /^(\w.*?), by (.*?)\s+(\d+)$/; $author = $1; $title = $2; $id = $3;


      Woudlnt it just make sense to use their XML formatted catalog? ...

        I would second that, given that the RDF looks like:

        <rdf:Description rdf:ID="etext13218"> <dc:publisher>&pg;</dc:publisher> <dc:title rdf:parseType="Literal">Don Orsino</dc:title> <dc:creator>Crawford, F. Marion (Francis Marion) (1854-1909)</dc:cre +ator> <dc:language>en</dc:language> <dc:created>2004-08-19</dc:created> <dc:rights rdf:resource="&lic;" /> </rdf:Description>

        Then all you need is something trivial like this to create a file ready for a MySQL 'load data local infile .....'

        #!/usr/bin/perl local $/ = "\n\n"; open RDF, $ARGV[0] or die $!; while(<RDF>){ next unless m/<rdf:Description rdf:ID="etext(\d+)"/; my $id = $1; next unless m/<dc:title[^>]+>([^<\n]+)</; my $title = $1; next unless m/<dc:creator>([^<\n]+)</; my $author = $1; $title =~ s/\s+/ /g; $author =~ s/\s+/ /g; print "$id\t$title\t$author\n"; }



      Looking a little closer and also in older index files i found this.
      *****A "C" Following a Project Gutenberg eBook Number Indicates Copyri +ght**** *****A "*" Following a Project Gutenberg eBook Number Indicates Reserv +ed **** [snip] The Life of John Ruskin, by W. G. Collingwood + 13076 A Hero and a Great Man, by Francis Kruckvich + 13075C [Illustrator: Fritz] Punch, Vol. 100, February 7, 1891, Ed. by Sir Francis Burnand + 13074 [snip] Feb 1995 Moon and Sixpence by Somerset Maugham [Maugham #1][moonaxxx.x +xx] 222 Feb 1995 The Return of Sherlock Holmes [Magazine Edition] [rholmxxb.x +xx] 221B Feb 1995 The Secret Sharer, by Joseph Conrad [Conrad #2] [ssharxxx.x +xx] 220
      I did not found the meaning of the 'B' though.

        221B - as in 221B Baker Street. It could mean something else, but I think it's somebody's attempt at humor.

        Wonder how this project turned out?

        But God demonstrates His own love toward us, in that while we were yet sinners, Christ died for us. Romans 5:8 (NASB)

Re: Parsing Gutenberg Catalog Index
by PodMaster (Abbot) on Aug 30, 2004 at 05:46 UTC
    Not perfect, but its a good start
    use strict; use warnings; @ARGV = 'GUTINDEX-2004.txt' unless @ARGV; my $file = shift or die "Usage: $0 GUTINDEX-2001.txt"; my @data; { open IN, '<', $file or die "Error: Can't open $file : $!"; my $start_parsing; local $_; while(<IN>){ next if /^\s*$/; last if /^End\s+of\s+\Q$file\E/; chomp; if( $start_parsing ){ if(/^(\w+.*?) by (.+?)\s+(\d{5}.?)$/){ my( $title, $author, $id ) = ( $1, $2, $3 ); $data[$#data+1]->{id} = $id; $data[$#data]->{title} = $title; $data[$#data]->{author} = $author; } else { for( /\[(.+?)(\])?/g ){ my $foo = $1; unless( $2 ){ while(<IN>){ if( /(.+?)\]/ ){ $foo .= $1; last; } else { $foo .= <IN> } } } if( $foo =~ /\[?(\S+):(.+)\]?/s ){ $data[$#data]->{$1} = $2; } elsif( $foo =~ /,\s\d+$/s ){ $data[$#data]->{Date} = $foo; } } } } # elsif( /^Title\s+and\s+Author/ ){ elsif( /^\Q~ ~ ~ ~ Posting Dates for the below eBooks\E/ ){ $start_parsing++; } } } use Data::Dumper; print Dumper( \@data );
    What I noticed is that the first entry in GUTINDEX-2004.txt doesn't have a closing [ for [Subtitle:, which is a bug in the file.

    MJD says "you can't just make shit up and expect the computer to know what you mean, retardo!"
    I run a Win32 PPM repository for perl 5.6.x and 5.8.x -- I take requests (README).
    ** The third rule of perl club is a statement of fact: pod is sexy.

Re: Parsing Gutenberg Catalog Index
by DrHyde (Prior) on Aug 31, 2004 at 08:00 UTC
    I have some code which makes an attempt at this and which (mostly) works, as part of my ongoing project to convert all of PG to Palmdoc. Email me, and I'll send you a copy.
      Dear Sir, I am doing a project to convert Gutenberg books into webpages. If you don't mind, could you please share your codes with me? Thanks, Mark my email:
        All that I've got now is at

Log In?

What's my password?
Create A New User
Node Status?
node history
Node Type: perlquestion [id://386850]
Approved by blokhead
Front-paged by Old_Gray_Bear
Jar. Jar!...

How do I use this? | Other CB clients
Other Users?
Others examining the Monastery: (5)
As of 2017-02-20 12:21 GMT
Find Nodes?
    Voting Booth?
    Before electricity was invented, what was the Electric Eel called?

    Results (295 votes). Check out past polls.