http://www.perlmonks.org?node_id=1212636
thanos1983 has asked for the wisdom of the Perl Monks concerning the following question:

Hello fellow Monks,

I got again stack on something possibly (minor) problem that I can not over come and I need to ask you guys in case that someone know more about it.

My question is extension of How to pass credentials through REST::Client. This time I need to POST a file through the module REST::Client.

The scenario is more or less the same, I have a server listening on the background and I use a small module to send a file. Sample of code:

restClient.pl

#!/usr/bin/perl use strict; use warnings; use Data::Dumper; use MyClientRest::ClientRest; my $host = "http://127.0.0.1:8000"; # instatiate class my $object = new ClientRest( $host ); my $username = "user"; my $password = "password"; my $file = 'test.txt'; my $upload = '/upload/'; my %optionsFile = ( "url" => $upload, "file" => $file, "username" => $username, "password" => $password ); my $postFile = $object->postSnippetsFile( %optionsFile ); print Dumper $postFile; __END__ $ perl restClient.pl $VAR1 = 'Not a SCALAR reference at /usr/local/share/perl/5.22.1/LWP/Pr +otocol/http.pm line 260. ';

Code in the main module.

ClientRest.pm

package ClientRest; use JSON; use Carp; use strict; use warnings; use MIME::Base64; use Data::Dumper; use version; our $VERSION; $VERSION = qv('0.0.1'); use REST::Client; sub new { my $class = shift; my $self = { _host => shift, }; _parameterValidation($self); # instatiate Rest::Client and create constructor my $client = REST::Client->new({ host => $self->{_host}, timeout => 10, }); $self->{_client} = $client; bless $self, $class; return $self; } sub _parameterValidation { my( $self ) = @_; croak "Invalid host syntax: sample 'http://<host>:<port>' " unless ( $self->{_host} =~ /^http:\/\/\d{1,3}\.\d{1,3}\.\d{1,3}\.\ +d{1,3}\:\d{1,5}$/ ); } sub postSnippetsFile { my ( $self, %options ) = @_; my $headers = { "Content" => {'file' => $options{file}}, "Content-type" => 'multipart/form-data', "Authorization" => 'Basic ' . encode_base64($options{username} . ':' . $options{password +}) }; $self->{_client}->POST( $options{url}, $headers ); return $self->{_client}->responseContent(); } 1;

If I use curl POST and it works as expected. Sample bellow:

$ curl -u user:password -H "Content-Type: multipart/form-data" -H "Acc +ept: application/json" -H "Expect:" -F file=@test.txt -X POST http:// +127.0.0.1:8000/upload/ | jq '.' % Total % Received % Xferd Average Speed Time Time Time + Current Dload Upload Total Spent Left + Speed 100 455 100 249 100 206 2547 2107 --:--:-- --:--:-- --:--: +-- 2567 { "url": "http://127.0.0.1:8000/snippets/2/", "id": 2, "highlight": "http://127.0.0.1:8000/snippets/2/highlight/", "owner": "user", "title": "Default Title", "code": "Test POST file Perl", "linenos": false, "language": "perl", "style": "emacs" }

I was reading online on how to post a file with this module and I found How to POST attachment to Confluence page using REST API with perl?, HTTP::Request::Common and multipart/form-data, PUT a Multipart request in PERL and POST a Multipart request in Perl via LWP etc etc. All links point to the same solution as the one that I have implemented.

From the error that I am getting:

$ perl restClient.pl $VAR1 = 'Not a SCALAR reference at /usr/local/share/perl/5.22.1/LWP/Pr +otocol/http.pm line 260. ';

I checked the module following snippet is from line (257-274) where the error is coming:

else { # Set (or override) Content-Length header my $clen = $request_headers->header('Content-Length'); if (defined($$content_ref) && length($$content_ref)) { $has_content = length($$content_ref); if (!defined($clen) || $clen ne $has_content) { if (defined $clen) { warn "Content-Length header value was wrong, fixed"; hlist_remove(\@h, 'Content-Length'); } push(@h, 'Content-Length' => $has_content); } } elsif ($clen) { warn "Content-Length set when there is no content, fixed"; hlist_remove(\@h, 'Content-Length'); } }

I can understand that the header is producing the problem but I am not sure where I am going wrong. Can anyone else spot the problem?

Update: I tried also to define in the file in the header as HoA (Hash Of Arrays) e.g.:

my $headers = { "Content" => ['file' => [$options{file}]], "Content-type" => 'multipart/form-data', "Authorization" => 'Basic ' . encode_base64($options{username} . ':' . $options{password +}) };

Minor note also here, normal post with dictionary (data) it works as expected. Sample of post (important parts of code):

# module part sub postSnippets { my ( $self, %options ) = @_; my $headers = { "Content-type" => 'application/json; charset=UTF-8 +', "Authorization" => 'Basic ' . encode_base64($options{username} . ':' . $options{password +}) }; $self->{_client}->POST( $options{url}, encode_json($options{hashRef}), $headers ); return decode_json $self->{_client}->responseContent(); } # script part my $hashRef = { "title" => "Test Title", "code" => "print \"Test POST Request\"", "linenos" => "false", "language" => "perl", "style" => "emacs" }; my %options = ( "url" => $url, "hashRef" => $hashRef, "username" => $username, "password" => $password ); my $post = $object->postSnippets( %options ); print Dumper $post; # output $ perl restClient.pl $VAR1 = { 'linenos' => bless( do{\(my $o = 0)}, 'JSON::PP::Boolean' ), 'id' => 2, 'language' => 'perl', 'code' => 'print "Test POST Request"', 'title' => 'Test Title', 'style' => 'emacs', 'highlight' => 'http://127.0.0.1:8000/snippets/2/highlight/' +, 'url' => 'http://127.0.0.1:8000/snippets/2/', 'owner' => 'user' };

The models that process the file upload and the simple code post are different but they should work with the a bit different header as the sample of code posted above.

Update2: After a lot of experimentation and research on the web I found that I a missing the body, I have added the body but now something else is missing. I get the following error:

$VAR1 = '{"detail":"Multipart form parse error - Invalid boundary in m +ultipart: None"}';

Sample of code with the body:

sub postSnippetsFile { my ( $self, %options ) = @_; my $headers = { "Content-type" => 'multipart/form-data', "Content" => [ file => [$options{file}]], "Authorization" => 'Basic '. encode_base64($options{username} . ':' . $options{password +}), }; $self->{_client}->POST( $options{url}, encode_json({file => $options{file}}), $headers ); return $self->{_client}->responseContent(); }

Update3: Solution provided further down with sample of code. Hope this helps someone else also in future.

Thank you in advance for everyone time and effort.

Seeking for Perl wisdom...on the process of learning...not there...yet!

Replies are listed 'Best First'.
Re: POST file through REST::Client
by NetWallah (Canon) on Apr 10, 2018 at 18:35 UTC
    Here is something to try:
    my $headers = { "Content-type" => 'multipart/form-data; boundary=--- +-MyRestAPIClientDataDelimiter',

                    Memory fault   --   brain fried

      Hello NetWallah,

      Apologies for the late reply but I got busy with something else and I did not get the time to test it until now.

      After your proposed modification I have updated the method to:

      sub postSnippetsFile { my ( $self, %options ) = @_; my $headers = { "Content-type" => 'multipart/form-data; boundary=- +---MyRestAPIClientDataDelimiter', # "Content" => [ 'file' => [$options{file}]], "Authorization" => 'Basic '. encode_base64($options{username}.':'. $options{password}), }; $self->{_client}->POST( $options{url}, encode_json({file => $options{file}}), $headers ); return $self->{_client}->responseContent(); }

      Now I am getting the following error (from server):

      $VAR1 = 'AttributeError at /upload/ \'NoneType\' object has no attribute \'name\' . . .

      Hmmm this I need to debug it, but from a quick point of view it looks like the file is not attached through the request. But this is a good point forward. I appreciate your effort to help me.

      BR / Thanos

      Seeking for Perl wisdom...on the process of learning...not there...yet!

      Hi,

      thats a job for "LWP" (HTTP::Request::Common)

        Hello Anonymous Monk,

        Maybe I should change to another module, it gets very complicated for no reason.

        BR / Thanos

        Seeking for Perl wisdom...on the process of learning...not there...yet!
Re: POST file through REST::Client
by thanos1983 (Vicar) on Apr 11, 2018 at 11:06 UTC

    Solution to the POST file with REST

    First of all I want to say thank you all for the effort and time and

    I finally found the only solution that worked for me is a combination of LWP::UserAgent and HTTP::Request::Common.

    Sample of working code bellow (for future reference):

    I know that most people will say why not use the LWP::UserAgent::credentials, I tried to use it but it did not worked for me so I used the old traditional manual adding credentials to header. Works like a charm. :)

    Seeking for Perl wisdom...on the process of learning...not there...yet!

      You can use that solution in your module

      sub postSnippetsFile { my ( $self, %options ) = @_; my $request = HTTP::Request::Common::POST( '', 'Content_Type' => 'form-data', 'Content' => [ file => [ $options{file} ] ], ); $request->authorization_basic($options{username},$options{password +}); #print Dumper $request; my $headers = { 'Content-Type' => $request->header('Content-Type'), "Authorization" => $request->header('Authorization'), }; my $body_content = $request->content; #print Dumper $headers,$body_content; $self->{_client}->POST( $options{url}, $body_content ,$headers ); #print $self->{_client}->responseCode(); return $self->{_client}->responseContent(); }
      poj

        Hello poj,

        Thanks a lot for your time and effort. It works as expected.

        BR / Thanos

        Seeking for Perl wisdom...on the process of learning...not there...yet!
Re: POST file through REST::Client
by Anonymous Monk on Apr 10, 2018 at 18:39 UTC

    Hi

    "Authorization" => 'Basic ' .encode_base64($options{username} . ':' . $options{password}) };

    You should be using the credentials method. Why aren't you using the credentials method?

    Update2: After a lot of experimentation and research on the web I found that I a missing the body, I have added the body but now something else is missing. I get the following error: Multipart form parse error - Invalid boundary in m +ultipart: None"

    Where does the error come from?

    I dont understand what you're trying to do with that code snippet. You dont seem to be doing the same thing as Re: PUT a Multipart request in PERL ( lwp-mechanize-put-multipart-form.pl ). Have you printed out the resulting request to see what it looks like?

      Hello Anonymous Monk,

      You should be using the credentials method. Why aren't you using the credentials method?

      To be honest I thought this is better passing the credentials to the header as I see the same thing happening with a CURL request. It is not the credentials that are producing the problem because on other POST / PUT / DELETE requests that I have already format are working just fine. The problem appears when I am trying to send the file.

      Where does the error come from?

      This error comes from the client side, complaining that it can not find the file to post it.

      I dont understand what you're trying to do with that code snippet.

      I am trying to post a file to server for further processing. The proposed solution that you send I will take a look, and try to apply it on my case.

      Have you printed out the resulting request to see what it looks like?

      Yes I am printing the return of the request and this is the error output:

      $VAR1 = '{"detail":"Multipart form parse error - Invalid boundary in m +ultipart: None"}';

      I will give it a try on your proposed solution and I will update my question. Thank you for your time and effort. BR / Thanos

      Seeking for Perl wisdom...on the process of learning...not there...yet!

        Hi,

        I realize you've marked this solved, but stratigery important :)

        Yes I am printing the return of the request and this is the error output:

        That is not a request. That is information you've already provided. Its like you pay for a drink, bartender says not enough, you still owe me 3 dollars, and you respond with, yes, I already gave you a napkin.

        I dont understand what you're trying to do with that code snippet...I am trying to post a file to server for further processing.

        I honestly thought you were trying to bake cookies, which is why I asked how much water you've added, because I wanted to make sure you were serious when you said I am trying to bake cookies and I added this much water, because that makes total cents. Total cents!

        You should be using the credentials method...To be honest I thought this is better ... CURL...working just fine .. except upload ... I tried to use it but it did not worked :)

        Credentials is convenient, shorter and easier to remember and use, than hand-enconding basic auth.

        As a bonus, credentials doesn't send username/password unless the remote/target/destination server asks for authentication, and the domain/realm matches.

        If credentials "isnt working", your domain/real doesn't match, or your server is broken by not asking for authentication.

        Brute force is a terrible way to learn, sourcediving to copy/paste instead of taking 10min to understand 20 lines of code and the corresponding docs.

        Anyway , if reject convenience, go big and reject the entire enchilada

        #!/usr/bin/perl -- use strict; use warnings; use LWP; my $ua = LWP::UserAgent->new; ## watch "actual" http transactions on stdout $ua->add_handler("request_send", sub { shift->dump; return }); $ua->add_handler("response_done", sub { shift->dump; return }); $ua->show_progress(1); ## on stderr my $urljig = 'http://jigsaw.w3.org/HTTP/Basic/'; $ua->credentials( URI->new( $urljig )->host_port, 'test', 'guest', 'gu +est' ); $ua->get( $urljig ); ## first login (get challenged, authenticate) $ua->post( # then upload "big" file $urljig, { # %form file => [ # form input field name undef, # or '/real/path/to/file/to/upload.txt' "filename to be send that is different from real/path/to/f +ile/to/upload", -content_type => 'text/json', -content => "file contents cause undef doesnt read file", ], }, content_type => 'multipart/form-data', # for the %form ); __END__

        When run this produces fruits

        GET http://jigsaw.w3.org/HTTP/Basic/
        User-Agent: libwww-perl/6.15
        
        (no content)
        HTTP/1.1 401 Unauthorized
        Connection: close
        Date: Thu, 12 Apr 2018 21:49:56 GMT
        Server: Jigsaw/2.3.0-beta4
        WWW-Authenticate: Basic realm="test"
        Content-Length: 261
        Content-Type: text/html;charset=ISO-8859-1
        Client-Date: Thu, 12 Apr 2018 21:53:28 GMT
        Client-Peer: 128.30.52.21:80
        Client-Response-Num: 1
        Title: Unauthorized
        ...
        
        GET http://jigsaw.w3.org/HTTP/Basic/
        Authorization: Basic Z3Vlc3Q6Z3Vlc3Q=
        User-Agent: libwww-perl/6.15
        
        (no content)
        HTTP/1.1 200 OK
        Connection: close
        Date: Thu, 12 Apr 2018 21:49:56 GMT
        ETag: "1lkdfte:qoguo8q8"
        Server: Jigsaw/2.3.0-beta2
        Content-Length: 458
        Content-Location: http://jigsaw.w3.org/HTTP/Basic/ok.html
        Content-Type: text/html
        Last-Modified: Sun, 25 Jun 2000 17:08:58 GMT
        Client-Date: Thu, 12 Apr 2018 21:53:28 GMT
        Client-Peer: 128.30.52.21:80
        Client-Response-Num: 1
        Title: Basic Authentication test page
        ...
        
        POST http://jigsaw.w3.org/HTTP/Basic/
        Authorization: Basic Z3Vlc3Q6Z3Vlc3Q=
        User-Agent: libwww-perl/6.15
        Content-Length: 232
        Content-Type: multipart/form-data; boundary=xYzZY
        
        --xYzZY\r
        Content-Disposition: form-data; name="file"; filename="filename to be send that is different from real/path/to/file/to/upload"\r
        -Content: file contents cause under doesnt read file\r
        -Content-Type: text/json\r
        \r
        \r
        --xYzZY--\r\n
        HTTP/1.1 405 Method Not Allowed
        Connection: close
        Date: Thu, 12 Apr 2018 21:49:57 GMT
        Server: Jigsaw/2.3.0-beta4
        Allow: HEAD,GET,OPTIONS,TRACE
        Content-Length: 41
        Content-Type: text/html
        Client-Date: Thu, 12 Apr 2018 21:53:28 GMT
        Client-Peer: 128.30.52.21:80
        Client-Response-Num: 1
        
        Method POST not allowed on this resource.