BioGeek has asked for the wisdom of the Perl Monks concerning the following question:

Hey Monks, I have a page, which has the following format:
It is a list with (biological) "candidates".
For each candidate, a bunch of information is given, like protein name (in square brackets), a R-score and a GO-score.
For each candidate, I'm interested in it's R-score and protein name. The tricky part is the variable number of occurences of the protein name.

The information can have one of the following formats.

1) R-score and protein name.
CANDIDATE 2
DNA (Note) PROTEIN: <A HREF = http://www.ncbi.nlm.nih.gov:80/entrez/query.fcgi?cmd=Retrieve&db=Protein&dopt=GenPept&list_uids=5729915>NP_006601</A>
R-score = 0.002033; GO-score = 0.000618

So I want to fetch "NP_006601" and "0.002033".

2) R-score and more than one protein name.
CANDIDATE 4
DNA (Note) PROTEIN: <A HREF = http://www.ncbi.nlm.nih.gov:80/entrez/query.fcgi?cmd=Retrieve&db=Protein&dopt=GenPept&list_uids=12738831>NP_075389</A> <A HREF = http://www.ncbi.nlm.nih.gov:80/entrez/query.fcgi?cmd=Retrieve&db=Protein&dopt=GenPept&list_uids=12738834>NP_075390</A>
R-score = 0.006971; GO-score = 0.000458


So I want to fetch "NP_075389", "NP_075390" and "0.006971".

3) R-score and no protein name.
CANDIDATE 1
DNA (Note)
R-score = 0.001743; GO-score = 0.000618

So I want to fetch "0.001743".

The code I wrote fetches the R-scores, but I can't come up with a code that matches all possible cases of the protein name.
use strict; use warnings; use LWP; my $R_score = (); my @protein = (); my @R_score_protein = (); $browser = LWP::UserAgent->new(); my $resp = $browser->get(http://www.bork.embl-heidelberg.de +/g2d/list_hits_disease.pl?U57042:Inflammatory_bowel_disease_7); my $content_all = $resp->content(); while ( $content_all =~ m{R\-score<\/A>\s=\s(\d\.\d+)\;}g ) { $R_score = $1; push( @R_score_protein, $R_score );
Update:To improve readability, I've removed the relevant HTML snipets, and placed them on my BioGeek's scratchpad.

Replies are listed 'Best First'.
Re: regex with variable input
by graff (Chancellor) on Aug 10, 2004 at 04:47 UTC
    I'm interested in it's R-score and protein name. The tricky part is the variable number of occurences of the protein name. ... So I want to fetch ..."

    And on fetching the various chunks, what do you need to do with them? How do you want to organize, manipulate or summarize them? Is each "candidate" uniquely identified by the "candidate number"? If so, do you want a simple listing that shows, e.g.

    candidate-number R-score [protein-name ...]
    What you intend to do with the data is an important issue, because it should determine how you set up the appropriate data structure for the task. For example, if the candidate numbers are unique and sequential, you probably want an array of hashes:
    my @candidates; # should use a hash instead, if my $c_id; # the numbering has lots of big gaps while (<>) { if ( /CANDIDATE\s+(\d+)/ ) { $c_id = $1; } elsif ( /DNA\s+.Note.\s*(\S*.*)/ ) { my $proteins = $1; my @p_set = (); while ( $proteins =~ /\[(NP\s+\d+)\]/g ) { push @p_set, $1; } $candidates[$c_id]{proteins} = [ @pset ]; } elsif ( /R-score\s*=\s*([.\d]+)/ ) { $candidates[$c_id]{rscore} = $1; } }
    If you're still learning about data structures and references, I'm sure the relevant man pages will supply any other details you may need (perlreftut, perldsc). Use Data::Dumper as well, of course.
Re: regex with variable input
by bradcathey (Prior) on Aug 10, 2004 at 00:10 UTC

    BioGeek, I don't mean to complain, but can simplify your question, a bit, get it down to the basics (I don't want to try to figure out all that HTML). Also, consider <readmore> tags if posting long questions. Thanks, and I look forward to seeing a distilled version.

    Update: Escaped my angle brackets per hv

    —Brad
    "Don't ever take a fence down until you know the reason it was put up. " G. K. Chesterton
Re: regex with variable input
by spq (Friar) on Aug 10, 2004 at 01:23 UTC

    First of all, does bork.embl-heidelberg.de offer any sort of batch download, or text based save that could be used?

    I'm guessing your looking to do this in a (semi) automated manner, and starting from saving as text from your web browser wont help in the long run, since your scripting the fetch? I also might suggest that you may want to consider contacting the web site maintainers. They may not be thrilled about getting hammered by your script, and may be very open to helping find an easier way to provide this information to you instead.

    That being said, I used wget to download the page and wrote this code which seems to capture what you want (this is a quick translation from a command line test, and in no way should be considered production ready or tested).

    while (<>) { chomp; if(/CANDIDATE (\d+)/){ $cand = $1; @acc = () } # if candidate line elsif (m{http://www.ncbi.nlm.nih.gov:80/entrez/query.fcgi} && m{(\w+)</A>}) { push(@acc,$1); } # else if ncbi accession # elsif (m{R-score</A>\s*=\s*(\S+);}) { print "$cand\t", join(",",@acc), "\t$1\n"; } # else if R-score, print this candidates info } # while reading web page
Re: regex with variable input
by TomDLux (Vicar) on Aug 10, 2004 at 02:12 UTC
    • Collapse the three lines of declaration into one line:

      my ( $R_score, @protein, @R_score_protein );
      Arrays are by default initialized as empty arrays, so you don't need to set them to (). <em$R_score is a scalar, not an array, so by default it is undef, which acts like the empty string if you try to concatenate to it; but you should initialize it to "" if it might be read before it is assigned.

    • In any case, you don't need $R_score, since you only use it to transfer a value from $1 into @R_score_protein. Delete the variable and assign $1</em, into the array.

      my ( @protein, @R_score_protein ); ... push( @R_score_protein, $1 );

    • Does 'strict' not complain about that unquoted string? In any case, get that URL out of the code and into a configuration variable at the beggining of the file. Better yet, provide for the user to override the URl at run time:

    • if you can process proteins and R scores separately:

      while ( $content_all =~ m{R\-score<\/A>\s=\s(\d\.\d+)\;}g ) { push( @R_score_protein, $1 ); } while ( $content_all =~ m{\[(NP_\d+)\]}g ) { push @protein, $1; }
      Better yet, you could move the regular expressions into 'local' variables or 'configuration' variables:
      my $protein_re = qr|\[(NP_\d+)\]|; my $score_re = qr|R-score</A>\s=\s(\d\.\d+)|; while ( $content_all =~ m{$protein_re} ) ..... while ( $content_all =~ m{$score_re} ) ....
      Tidier, more readable, more maintainable.

    • I suspect you want to associate the protein codes and the R scores, which the version above doesn't provide. You need to break the input into distinct lines, which I'll arrange to have happen magically in the break_into_lines() routine: