Beefy Boxes and Bandwidth Generously Provided by pair Networks
laziness, impatience, and hubris

Net::Amazon::EC2 loses pound-shebang when base64'd

by hesco (Deacon)
on Oct 04, 2009 at 05:59 UTC ( #799081=perlquestion: print w/replies, xml ) Need Help??
hesco has asked for the wisdom of the Perl Monks concerning the following question:


TIMTOWTDI, I suppose. After opening and reading the user_data file, and encode_base64() to the scalar collected by iterating over it, I still got nowhere. The Anonymous monk below points me back to the source documentation, where I find nothing on this particular question, but did discover a whole new library of perl code to adapt from, using a different perl module than the one I've been working with. The one I'm working with was updated less than two weeks ago, but its perldoc gives me no further guidance.

So, alas, I tried this, instead and it seems to be doing the trick:

if($build_config_instance){ my $host = 'root@' . $instance->dns_name(); my $scp = "scp -i $pvt_key $base_path/$build_script $host:/roo +t/"; print STDERR "\nssh -i $pvt_key $host chmod +x /root/$build_sc +ript \n"; print STDERR "ssh -i $pvt_key $host /root/$build_script \n"; `$scp`; `ssh -i $pvt_key $host chmod +x /root/$build_script `; `ssh -i $pvt_key $host /root/$build_script `; }
Well my shell script bombed out after 16m47.233s, but only a few lines from the end. Walked around that issue, I guess. Now to debug my bash script which builds this server.

Original post . . .

I sure would appreciate some clues on how to move past this one.

For those unaware, the Amazon Cloud API permits passing 'UserData' across the invocation. There is a size limit, but even so, it provides a useful way to pre-configure an instance as it is brought online. At least that is the theory. Some AMI Machine Images are set up to look at that UserData and feed it to an appropriate interpreter as the root user to help that process along.

The $ec2->run_instances() method takes a UserData key, whose value the perldoc says: 'Needs to be base64 encoded.' I used MIME::Base64::encode_base64() to give it its favorite food. And to show me its gratitude, it throws the following error at me:

Oct 4 02:15:51 svrname S70ec2-get-credentials: New ssh key added to / +root/.ssh/authorized_keys from +ub lic-keys/0/openssh-key Oct 4 02:15:52 svrname S71ec2-run-user-data: Retrieving user-data Oct 4 02:15:52 svrname S71ec2-run-user-data: Skipping user-data as it + does not begin with #!
If I try it without the encode_base64() I get Invalid BASE64 encoding of user data instead.

Here is a piece of the code which I think will reproduce this error; as well as a snip from the shell script its being fed, sporting a fresh #!/bin/bash interpreter line.

Any ideas on how to make my interpreter line more apparent than it already is?

#!/usr/bin/perl use strict; use warnings; use Data::Dumper; use Net::Amazon::EC2; use MIME::Base64; my $ec2 = Net::Amazon::EC2->new( AWSAccessKeyId => 'not-telling', SecretAccessKey => 'not-telling-this-either' ); my $custom_ami_id = 'ami-ff46a796'; my $user_data = encode_base64('/home/hesco/sandbox/EC2/trick_out_my_se'); if($create_new_instance){ my $instance = $ec2->run_instances( 'Placement.AvailabilityZone' => 'us-east-1a', KeyName => 'aws_test', SecurityGroup => 'asterisk', ImageId => $custom_ami_id, UserData => $user_data, MinCount => 1, MaxCount => 1 ); print STDERR Dumper($instance); }
My starts out:

#!/bin/bash set -e -x apt-get --yes update apt-get --yes upgrade apt-get --yes install subversion build-essential etc . . .
if( $lal && $lol ) { $life++; }
if( $insurance->rationing() ) { $people->die(); }

Replies are listed 'Best First'.
Re: Net::Amazon::EC2 loses pound-shebang when base64'd
by Anonymous Monk on Oct 04, 2009 at 06:43 UTC
Re: Net::Amazon::EC2 loses pound-shebang when base64'd
by merlyn (Sage) on Oct 06, 2009 at 14:09 UTC
    Just a wild stab, but are you sure that the line-endings for /home/hesco/sandbox/EC2/ are correct? Not Windows line endings or something weird?

    And are you sure the interpreter mentioned after "#!" exists in the AMI?

    I'm asking this because something in the AMI startup is apparently being more picky than what your local box needs. Or, something is corrupting the base64 decode (are you exceeding the size? try something smaller).

    -- Randal L. Schwartz, Perl hacker

    The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in RFC 2119.

Re: Net::Amazon::EC2 loses pound-shebang when base64'd
by esh (Pilgrim) on Oct 06, 2009 at 05:40 UTC
    Try passing in the user-data script without base-64 encoding it. It works for me.

    I happen to maintain the user-data script running software and the Amazon EC2 Debian AMI you are running. I suppose I could enhance it to try to base-64 decode the user-data if it wasn't in the clear, but I'm not sure that there is any good reason to do so.

      I take back my comment about not base-64 encoding. I just looked at my code and it says:
      my $instances = $ec2->run_instances( ImageId => $ami_id, MinCount => 1, MaxCount => 1, KeyName => $key, (defined $user_data ? (UserData => MIME::Base64::encode_base64($user_data)) : ()), (defined $instance_type ? (InstanceType => $instance_type) : ()), #... ) or die "$ec2->{error}";
Re: Net::Amazon::EC2 loses pound-shebang when base64'd
by Anonymous Monk on Oct 06, 2009 at 07:02 UTC
    The Anonymous monk below points me back to the source documentation, where I find nothing on this particular question,

    Why? Searching for the error "Skipping user-data as it does not begin with #!" I found ec2-run-user-data, the program you appeared to be the trying to run. It does't mention base64.

    Net::Amazon::EC2 is just an interface to some amazon API, the important docs would be found on amazon.

Re: Net::Amazon::EC2 loses pound-shebang when base64'd
by esh (Pilgrim) on Oct 06, 2009 at 23:18 UTC
    I think the problem in your code is that you are passing user-data which is a base-64 encoded version of the file name instead of the file contents.
Re: Net::Amazon::EC2 loses pound-shebang when base64'd
by hesco (Deacon) on Oct 07, 2009 at 01:20 UTC
    Thanks, Eric for taking a look at this.

    I had earlier tried that, when I do this:

    my $trick_my_box_script; my $base_path = '/home/hesco/sandbox/EC2'; my $build_script = ''; open('USER_DATA','<',"$base_path/$build_script") or die "Unable to ope +n $base_path/$build_script: $! \n"; while(<USER_DATA>){ $trick_my_box_script .= $_; } close(USER_DATA); my $user_data = encode_base64($trick_my_box_script);
    Then I get errors which look like this:

    hesco@marcus8:~/sandbox/EC2$ time perl $VAR1 = bless( { 'errors' => [ bless( { 'message' => '500 read failed: Connection reset by peer', 'code' => 'HTTP POST FAILURE' }, 'Net::Amazon::EC2::Error' ) ], 'request_id' => 'N/A' }, 'Net::Amazon::EC2::Errors' );
    That shell script is only 7.2k, well within the 16k limit. Any idea why it would be rejected like this?

    -- Hugh

    if( $lal && $lol ) { $life++; }
    if( $insurance->rationing() ) { $people->die(); }
      I doubt this new problem is related to the user-data.

      AWS generally expects you to retry your API call when you get a 500 error.

      If you keep getting them, then something is generally more seriously wrong and you should wait a while before trying again.

      If you can consistently reproduce a 500 error when calling it with "X" while getting success calling it with "Y", then you may have found a bug.

      I'd recommend seeking help on the EC2 forum as this no longer looks like a Perl issue:

Log In?

What's my password?
Create A New User
Node Status?
node history
Node Type: perlquestion [id://799081]
Approved by broomduster
[Corion]: Meh. I need to find myself a better "programming" feed than r/programming (which is just HackerNews reposts and advertisements, very little code). Maybe I should select a list of links that I liked on r/programming and then google for ...
[Corion]: ... an aggregator site that also listed (ideally) all of these links once. And maybe also have an exclude list to blacklist some of the most spammy links that the site may have never mentioned at all
[marto]: I gave up reading HN at all, and feel much better for it :P
[Corion]: Now, how to best automate that Google search ... :-)
[Corion]: marto: Yeah, I'm also short of not reading it at all. I already stopped reading it in the morning because it gave me a foul mood.
[Corion]: The good posts on HN are the non-computing posts. Most of the other stuff is maybe relevant to you if you are 20 and live in Silicon Valley...
[Corion]: Two attributes that don't describe me.
[Corion]: Maybe I should also write a curator for HN or simply not read it anymore, like you do.

How do I use this? | Other CB clients
Other Users?
Others cooling their heels in the Monastery: (8)
As of 2017-07-24 13:24 GMT
Find Nodes?
    Voting Booth?
    I came, I saw, I ...

    Results (354 votes). Check out past polls.