Re^8: Scrape Yahoo Financial Historical- Process Dataset - format and create dynamic page

[tbone654]

AM,

I downloaded the modules from CPAN... Figured out how to debug from a browser... Uploaded the modules to Yahoo into a directory I created /lib... put a statement into my perl script...

use lib "./lib"

added additional module dependancies... and now it works, I can get historicals directly from yahoo, and print it right to the browser.

2012/10/01,144.5200,145.6900,144.0100,144.3500,135911200 
2012/10/02,144.9200,145.1500,143.8300,144.5000,113422200 
2012/10/03,144.8900,145.4300,144.1300,145.0900,121283100 
2012/10/04,145.6400,146.3400,145.4400,146.1300,124311600 
2012/10/05,146.9100,147.1600,145.7000,146.1400,124842100 
2012/10/08,145.6000,146.1200,145.3100,145.6400,78415400 
2012/10/09,145.5300,145.6500,144.1500,144.2000,148872900 
2012/10/10,144.1800,144.3200,143.0900,143.2800,123992700 

Awsome, the power of figuring this out. Thank you so much for the great hints.

My next task is to perform some math on the data and then print the original data plus calculated columns back to the browser. There are modules out there "Stockmonkey" Math::Business::xxx that does some common financial type things to the data. But it uses a database behind the calcs, and I think I must avoid that. The way this is going to look in the end is something like this:

Date Open High Low Close Volume    op rg cl rg 89 163 str Reaction up trend dn trend Reaction DPDH RPDL APDH BPDL 
10/10/2012 144.18 144.32 143.09 143.28 124,247,408  4 B .89 .15 -1  .29 .00 .00 .00 147.24 1.23 1.23 .00 .00 
2012/10/10 144.18 144.32 143.09 143.28 123,992,700  3 ss .89 .15 1  .31 .00 .00 .00 147.24 2.56 .17 .00 1.06 
2012/10/09 145.53 145.65 144.15 144.20 148,610,100  2 S .92 .03   .30 .00 .00 .00 147.24 1.97 .34 .00 1.16 
2012/10/08 145.60 146.12 145.31 145.64 78,415,400 B 1 B .36 .41   .32 .00 .00 .00 147.24 1.85 .42 .00 .39 
2012/10/05 146.91 147.16 145.70 146.14 124,842,100  6 ss .83 .30   .30 .00 .00 .00 147.24 .64 1.72 .82 .00 
2012/10/04 145.64 146.34 145.44 146.13 124,311,600  5 S .22 .77   .28 .00 .00 .00 147.24 -.01 2.21 .91 .00 

So my question is something like this: I belive I need to use an array of arrays (sorted by first element - Date) and append my calcs to the end of each array. (one line/array is a list of elements that I have to store in memory) Then I print it sorted (reverse) to the browser. Am I on the right track in that thinking? I'm worried about building a large data structure in memory on a server I don't own. I'm wondering if a module already exists that may do some of this already? And I don't know, what I don't know, so I was hoping to run this bye you first. Any thoughts?

I can see it in c code like this...

for(i=0;i<250;i++){
    for(j=0;j<20;j++) {
        array\i\\j\ = {do some math};
    }; //end of for
}; //end of for

see what I mean?  make sense?

[Anonymous Monk]

use lib "./lib"

Relative paths are fragile, use a full path :) it can be dynamic ( see File::FindLib, FindBin, use lib $lib_dir not working )

But it uses a database behind the calcs, and I think I must avoid that.

The FAQ says to use database to store results, so you don't need it, but if you need one, you can always use DBD::SQLite

Am I on the right track in that thinking? I'm worried about building a large data structure in memory on a server I don't own. I'm wondering if a module already exists that may do some of this already?

I feet not :)

A casual look at Math::Business::HMA/ Math::Business::StockMonkey::FAQ ... shoes hints it doesn't use a lot of memory (it doesn't build a giant array), so I'd just use stockmonkey, but I would do local testing to check for memory growth ( Devel::NYTProf, Devel::Leak Devel::LeakTrace, WeakRef )

*hint* the memory question would make a good addition to Math::Business::StockMonkey::FAQ

I feel my shoes with my feet , wanna take good care , remain elite