Beefy Boxes and Bandwidth Generously Provided by pair Networks
There's more than one way to do things
 
PerlMonks  

Newbie question

by SpaceCowboy (Acolyte)
on Oct 19, 2021 at 00:27 UTC ( #11137709=perlquestion: print w/replies, xml ) Need Help??

SpaceCowboy has asked for the wisdom of the Perl Monks concerning the following question:

Hi. I am currently exploring Perl and wondering:

1. Is there a way to encrypt a Perl file containing password? I need to import database credentials from a file and would like to know more about encrypting passwords

2. Pandas equivalent in Perl -- can I import a million rows and transpose, rename, transform and do a bunch of joins -- is this possible in Perl? I know I can write SQL and call it using DBI but not sure If I can get the table output and start performing transformations in it..

3. Books/Resources recommendations -- any advice on books on perl deployment practices? docker?

Thank you for supporting Perl and thanks in advance for your help!

Replies are listed 'Best First'.
Re: database and deployment questions
by hippo (Bishop) on Oct 19, 2021 at 09:06 UTC

    Hello, SpaceCowboy. Welcome to the Monastery.

    Is there a way to encrypt a Perl file containing password?

    There are many ways to encrypt a file in Perl - the real question is how do you decrypt it again? That will require a password and it will need to be stored somewhere, say in a file which then needs to be encrypted, and so ad infinitum. See How can a script use a password without making the password visible? and Password encryption for discussion of some approaches to this.

    can I import a million rows and transpose, rename, transform and do a bunch of joins -- is this possible in Perl?

    Yes, but why would you? As you say SQL is likely the better tool for this, that's why we have databases. Your reason for avoiding the database seems vague to me. Why not try the SQL approach and if you need help re-framing the results that could be another question.

    any advice on books on perl deployment practices?

    I'm not aware of any books specifically about that but deployment is touched on in various other books and articles. You might find the "Managing Real Programs" chapter of Modern Perl to be useful, for example.


    🦛

      Thank you @hippo for your response. I have data from more than one database and needed to bring them over to drop, rename, perform calculations, joins and export it as flat file... does not seem possible with SQL alone, need some kind of "pandas" environment; I am willing to learn and try other approaches - storing millions of rows and hundreds of columns in 2d array if possible... any advice on books related to data tables processing?

        I would still be looking to do all that within an RDBMS.

        storing millions of rows and hundreds of columns in 2d array if possible

        Sure it's possible - if you have enough memory. I just don't see why you would do that when a database is the right tool for the job. There is nothing you have said here to suggest to me that it isn't.


        🦛

        Perl arrays and hashtables support nesting (e.g. hash-of-hash-of-array-of-hash etc.), mixed "data types" and slices (a convenient form of indexing). There is also PDL if your data is numeric. Additionally, there is Inline::Python if you want to inline python code you already have (untested by me).

        storing millions of rows and hundreds of columns in 2d array if possible

        well that's illegal in some states and some of us Perl-ers still work on Spectrums, :)

        Update: There's also DBIx::Array which offers this:

        foreach my $row ($dbx->sqlarrayhash($sql, @bind)) { do_something($row->{"id"}, $row->{"column"}); }

        And there's DBIx::Class as a long-term investment.

        bw, bliako

        I have data from more than one database and needed to bring them over to drop, rename, perform calculations, joins and export it as flat file

        That sounds like a use for a Temporary Table in the RDMS to me...

Re: Newbie question
by swl (Vicar) on Oct 19, 2021 at 21:47 UTC

    hippo has answered #2, but you could also have a look at Data::Frame for a general structure. It does not do things like joins and the API is marked as experimental, but it might still be interesting.

    Data::Frame uses PDL for the numeric data, which is something else you could look at if you do not have mixed strings and numbers (but then you would probably be asking about numpy equivalents).

    It also has a high number of fails on cpan testers, but those seem to have been cleared up with recent releases of PDL.

      Thanks for your response. I am getting data from multiple databases. I needed to extract them, store them in a variable, drop a few columns, rename a few, calculate bunch of fields and prepare a final dataset. IS this still possible with Data::Frame and PDL as a whole?

        Possibly, but have a look at the documentation. It's probably better in the long run to work directly with the database, as per hippo's suggestion in 11137712.

        One place to start would be DBI, but stated with the caveat that I've never used it or interacted with databases via perl. However, many of the other monks have and could give you good advice.

        If PDL processing is still desired, there is PDL::IO::DBI to bring in data from RDBMS.
Re: Newbie question
by Perlbotics (Bishop) on Oct 19, 2021 at 19:14 UTC
    1. Is there a way to encrypt a Perl file containing password? I need to import database credentials from a file and would like to know more about encrypting passwords

    Since you mentioned Docker... If you need something that scales in a large environment, i.e. multiple nodes requiring credentials and that also works for unattended (re-)boots, have a look at HashiCorps Vault (client: i.e. WebService::HashiCorp::Vault, server: https://github.com/hashicorp/vault). There is still a secret to be kept (i.e. token), however, management of credentials becomes much more feasible at the cost of running a dedicated centralised (hopefully redundant) vault service.

Re: Newbie question
by fishy (Friar) on Oct 20, 2021 at 17:46 UTC
Re: Newbie question
by perlfan (Vicar) on Oct 19, 2021 at 17:36 UTC
    1. Is there a way to encrypt a Perl file containing password? I need to import database credentials from a file and would like to know more about encrypting passwords

    Is it sufficient to just rely on your system's file permissions? Tall order, I know. That said, maybe you want to look at gpg if security is actually important (can't vouch for Crypt::GPG, but it's there). If not, maybe file perms (0600).

    3. Books/Resources recommendations -- any advice on books on perl deployment practices? docker?

    DDG that - lots of talks from past conferences; look on dockerhub for perl environments, lots of neat stuff. And perl books in generally, there are many lists in PMs and I've even typed my list up a few times.

Log In?
Username:
Password:

What's my password?
Create A New User
Domain Nodelet?
Node Status?
node history
Node Type: perlquestion [id://11137709]
Approved by marto
Front-paged by Corion
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others making s'mores by the fire in the courtyard of the Monastery: (2)
As of 2022-05-28 03:50 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?
    Do you prefer to work remotely?



    Results (98 votes). Check out past polls.

    Notices?