Re^6: Paging with REST::Client?

[Argel]

The only documentation is an example in an appendix. I've used it with with Perl code shelling out to Curl, so I can go back to it if necessary. But that was really ugly code and I don't have time to write something prettier. I was really hoping this came up enough that an existing module supported it, or e.g. that LWP::UserAgent had the hooks in it to add support for it. Sort of like how XML::Twig has an option to iterate over large XML files.

It seems like the easiest approach may be to see if I can tell that there was an error because of too much data, and if so, I can assume a restart is needed. Thanks for the help!

[Anonymous Monk]

The only documentation is an example in an appendix.

Where can one read that documentation?

I was really hoping this came up enough that an existing module supported it, or e.g. that LWP::UserAgent had the hooks in it to add support for it.

Sorry but you still haven't explained the problem, what are you talking about?

"Paging" is just looping. https://metacpan.org/release/REST-Client/source/lib/REST/Client.pm is a thin wrapper around LWP, and LWP has plenty of hooks.

Its like a delivery driver asking a car mechanic, does my car have paging? Can you add a paging module?

Dear delivery driver, you are the pager :) You drive for pickup, you drive for delivery, repeat, ...

[Argel]

Paging in this context refers to pulling in a subset of data, then the next subset (page) of data, etc. Here's some documentation (not hosted by the vendor or the company I work for):
• https://ipam.illinois.edu/wapidoc/additional/sample.html#executing-a-paging-request
• https://ipam.illinois.edu/wapidoc/#get

[Anonymous Monk]

Paging in this context refers to pulling in a subset of data, then the next subset (page) of data, etc.

Yeah, thats a loop

Here's some documentation (not hosted by the vendor or the company I work for):

Ok, ask yourself, where is "Infoblox" mentioned in REST, REST::Client or LWP?

Or why it took take until Re^8 to link Infoblox ?

Vacation time or something

#!/usr/bin/perl --
use strict;
use warnings;
our @PAIRS;
our %JSON;
Main( @ARGV );
exit( 0 );
sub Main {
    %JSON = fromPairs();
    my  $startUrl = $PAIRS[0][0];
    Paginate(
        $startUrl,
        sub {
            my( $json, $url ) = @_;
## real work here
            print "## callback  @_ \n";
            print $json->{result}[0]{name}, "\n\n";
        },
    );
    return;
}
sub getAll {
    my @jsons;
    Paginate(
        $startUrl,
        sub {
            my( $json, $url ) = @_;
            push @jsons, $json;
        },
    );
    return \@jsons;
}
sub Paginate {
    my( $url, $callback )= @_;
    my @gets = $url;
    while( @gets ){
        my $url = shift @gets;
        my $json = GET( $url );
        if( my $id = $json->{next_page_id} ){
            push @gets, urlNext( $url, $id, $json );
        }
        $callback->( $json, $url );
    }
}
sub urlNext {
    my( $url, $id, $json ) = @_;
## logic here
    return "https://127.0.0.1/wapi/v2.7.3/record:a?_page_id=$id";
}
sub GET {
## real network request, REST::Client or whatever
    return $JSON{ $_[0] };
}
sub fromPairs {
    @PAIRS = (
  [
    "https://127.0.0.1/wapi/v2.7.3/record:a?_max_results=2&_paging=1&_
+return_as_object=1&_return_fields=name&name~=test1.com",
    {
      next_page_id => "789c5590...4efc1732",
      result => [
        {
          _ref => "record:a/ZG5zLmJpbmRfYSQuXY29tLn3DEwLjEuMC4x:a1.tes
+t1.com/default",
          name => "a1.test1.com",
        },
        {
          _ref => "record:a/ZG5zLmJpbmRLnRlc3QxLGE1LDEwLjEuMC41:a5.tes
+t1.com/default",
          name => "a5.test1.com",
        },
      ],
    },
  ],
  [
    "https://127.0.0.1/wapi/v2.7.3/record:a?_page_id=789c5590...4efc17
+32",
    {
      next_page_id => "789c5590...3e113c3d4d",
      result => [
        {
          _ref => "record:a/ZG5zLmJpbmRfYSQc3QxLGE0LDEwLjEuMC40:a4.tes
+t1.com/default",
          name => "a4.test1.com",
        },
        {
          _ref => "record:a/ZG5zLmuY29tLnRlc3QxLGEzLDEwLjEuMC4z:a3.tes
+t1.com/default",
          name => "a3.test1.com",
        },
      ],
    },
  ],
  [
    "https://127.0.0.1/wapi/v2.7.3/record:a?_page_id=789c5590...3e113c
+3d4d",
    {
      result => [
        {
          _ref => "record:a/ZG5zLmJpbmRfYSQuX2RLGEyLDEwLjEuMC4y:a2.tes
+t1.com/default",
          name => "a2.test1.com",
        },
      ],
    },
  ],
);
    return map { @$_ } @PAIRS;
}