http://www.perlmonks.org?node_id=1100999

Anonymous Monk has asked for the wisdom of the Perl Monks concerning the following question:

We recently upgraded perl from 5.8 to 5.10

we discovered after the upgrade that the fetchrow_array loop ends abruptly

code snippet:
my $searchSQL = $dbh->prepare($sql1);
my $insertSQL = $dbh->prepare($sql2);


$searchSQL->execute();
while (my ($id,$name) = $searchSQL->fetchrow_array()) {
$insertSQL->execute($id,$name);
$insertSQL->finish;
}


if I comment out the "$insertSQL->execute($id,$name);" it loops thru all the records.

the code perfectly works in 5.8 but not in 5.10

any ideas?

Thanks

Replies are listed 'Best First'.
Re: fetchrow_array loop in perl 5.10
by Corion (Pope) on Sep 18, 2014 at 08:08 UTC

    Depending on (the version of) your database driver, it might be that your database driver does not like more than one statement handle being active at the same time. In your case, that would be $searchSQL during the loop and in that loop the $insertSQL.

    A workaround could be to first fetch all items into an array and then execute the insert for each element of that array.

    In my experience, at least SQLite had this behaviour of not really wanting more than one active statement handle.

      first fetch all items into an array and then execute the insert for each element of that array
      Another option might be to keep two database handles, and run the search in one, and the insert in the second one.
      لսႽ ᥲᥒ⚪⟊Ⴙᘓᖇ Ꮅᘓᖇ⎱ Ⴙᥲ𝇋ƙᘓᖇ
        yep, we already found a workaround for it, but we wanted to find a centralized fix for it, this is an existing code (thousand lines of codes are coded this way) and was working in perl 5.8
      Depending on (the version of) your database driver, it might be that your database driver does not like more than one statement handle being active at the same time.

      Yep, that's what happened to me. I had to declare $dbh2 to make my stuff work. $dbh cannot handle both statements. I use DBD::Pg for Postgresql.

      Perl 5.8.8 on Redhat Linux RHEL 5.5.56 (64-bit)

        See perhaps, DBI, fork, and clone.

        I can't tell if applicable to those problems (yours and that of the OP), but it's interesting anyway :)

Re: fetchrow_array loop in perl 5.10
by McA (Priest) on Sep 18, 2014 at 07:20 UTC

    Hi,

    I can't say much about that issue. The only thing I can see from the code is the line

    $insertSQL->finish;

    which makes me wonder. Why do you have this statement in the loop body? As soon as you have more than one record to fetch your insertSQL statement handle isn't valid anymore.

    At least I would expect to have this statement at the end of the script outside of the while loop when you've executed all insert statements.

    Probably a point to start from.

    Regards
    McA

      thanks for the reply,I tried commenting it out, issue is still there (didn't loop thru all the records).
Re: fetchrow_array loop in perl 5.10
by mje (Curate) on Sep 18, 2014 at 08:27 UTC

    You definitely do not want to call finish on that insert statement. However, I see you say commenting it out does not help. What DBD are you using and what are the versions of it and DBI?

      from
      DBD::DB2 :1.78
      DBI :1.611

      to:

      DBD::DB2 :1.85
      DBI :1.631

        Could you cut the code down to the smallest that fails then run with DBI_TRACE env var set to 15 like this:

        DBI_TRACE=15=x.log export DBI_TRACE run code show us the last few hundred lines of x.log
Re: fetchrow_array loop in perl 5.10
by sundialsvc4 (Abbot) on Sep 18, 2014 at 12:21 UTC

    Did you re-install the packages when you upgraded from 5.8 to 5.10, and are you certain that your PERL5LIB is pointing to the right location?   In this case, I trust the implementation of DBI implicitly, and your code looks okay, so I suspect that there was a problem introduced by the upgrade process that you used.   Some of this stuff uses “XS” (C-language code, libraries, etc.) very extensively, and upgrades can upset its delicate clockwork.   A clean-reinstall of the package will, for example, recompile those components.   Also, you could well have two packages out there (in two different Perl-version-specific directories), and be pointing to the wrong one (which might have the same name and version-code).

Re: fetchrow_array loop in perl 5.10
by andal (Hermit) on Sep 18, 2014 at 14:12 UTC

    Potentially, this is result of DB upgrade. For example SQLite databases have changed the way how they manipulate locks on files. Depending on which table is read and which table is updated you may get different locking problems. One can play with BEGIN TRANSACTION or autocommit settings to avoid dead-locking.

    Anyway, most likely you should provide information about DB that is being used and SQL statements that are executed.

      Judging from the trace-outputs listed above, I would hazard a guess that the target is DB/2.

      However, your mention of MySQL and locking brings up once again a point that I found to be hugely important:   that you should always use transactions with SQLite, especially when writing, because otherwise SQLite will physically verify every write.   There’s no buffering, no “lazy writing,” and this is by design (in the SQLite engine, AFAIK).   I haven’t had need to keep up with driver-changes and so on for quite some time, so I don’t know what recently might have changed.   I simply, as a blanket rule, use transactions for everything SQLite, including reads.

      If the target is DB/2, then I am “entirely convinced” that the actual situation that the OP is facing right now is ... bogosity.   The present behavior appears to be nonsensical, and most likely it is, because the root cause will likely to be “this version’s apples trying their best to talk to last version’s oranges, or vice-versa.”   There is probably nothing fundamentally wrong with this source-code:   the problem is in the particulars of the (not-quite right ...) environment in which the code is trying its best to execute.   “Red herrings.”   Oooh, I hate it when that happens.

        You are making factually incorrect statements.

        You have been disputed on these exact same points several times before. You seem to like to rely on the saying that "the best way to get the correct answer on the Internet is to post the wrong one", but this time I'm not giving you the benefit of that effect. Go through your own posting history, you have been shown to be wrong plenty of times, without correcting yourself, without acknowledging the corrections, or even learning from them! I would love it if you would stop posting if you don't know what you're talking about, but I'm betting you won't do that.

        Therefore, this is merely a warning to others that the parent node contains incorrect statements.
Re: fetchrow_array loop in perl 5.10
by TomDLux (Vicar) on Sep 19, 2014 at 00:41 UTC

    You expect people to determine from deep understanding of the code why your program doesn't work.

    I view programming the other way around ..... it doesn't work? What is the return value of execute()? What does DBI::errstr() return? If you run the program in the debugger, what happens in the first ( and only successful ) execute()? Is there a second execute() attempted but it fails? Or does the loop terminate after the first execute?

    As Occam said: Entia non sunt multiplicanda praeter necessitatem.

      errstr has no value for every after line, the issue here is that the loop (fetchrow_array) ends after 2 rows (out of 4) always. again the code perfectly works before the perl upgrade.

        If there are no errors but the loop ends unexpectedly, have you checked to see that the loop conditional is what you expect?

Re: fetchrow_array loop in perl 5.10
by Limbic~Region (Chancellor) on Nov 17, 2016 at 19:02 UTC
    Anonymous Monk,
    I had the same problem and was able to further isolate it - it only breaks for me if one of the columns you are selecting is a LOB. I'm not sure if that helps or not.

    Cheers - L~R