The reason for not going to ftp is that, best I can tell, I can only download the data based on what was uploaded on a given day. Everything for every day would have to be filtered through and deleted to get just the files I need (well under 5% of the universe). I don't have the hardware for it.

Using the website, it seemed like I could put various firm "CIK" codes into a data file and then access and parse the table information. Then I just download just the firm-form combinations I needed, possibly renaming in the process to make more descriptive.

For those giving me the great links to things that could help, is there any way I could trouble you for some "for dummies" level sample code or even for the site in question? Otherwise I suspect I'm just going to be staring at cpan.org with a very confused look on my face for a very long time to come (it's happened before).

I am not an expert on SEC filings or the Edgar database, but after a few minutes of poking around the README file and docs directory it looks like everything would still be simpler with ftp unless you are a screen-scraping wizard.

As I have just learned, each company that files with the SEC has a CIK number. In your example, the CIK number for IBM is 51143. All of IBM's filings live in that directory ftp://ftp.sec.gov/edgar/data/51143. From the explanation in the README it seems that prior to Edgar 7.0 (starting in the year 2000) all of each company's filings were stored in one directory, but because of problems with overwriting documents when ammendments were submitted, everything is now stored in sub directories based on the accession number of the documents. This might be why it appears that information is per-day. Fortunately, it looks like the SEC provides an index of filings for each quarter by company name or type of filing so you don't have to mess around with slogging through every sub-directory to find the information you need. The only benefit I see from the http interface in your example is that your search focused on filings related to change of ownership. However, I gather that these are related to a known subset of forms (4, K-8) and this information is available in the index, so you could subset the information yourself.

So, if it were me I would probably move forward in two stages using Net::ftp to 1) grab indices and subsetting the records I need based on company and filing-type to create a list of the files I want to get, and then 2) and grab those files with Net::ftp again.

For accessing the web, WWW::Mechanize is pretty nice.

And HTML::TableExtract in between (have it parse $mech->content) should making getting the content into tables pretty easy ..

I planned to go to 100 actually. It will be more efficient. But the need to hit that next link for more data will still be necessary.

You can use WWW::Mechanize's follow_link method.

It might look like this depending upon the size of the query you are doing. I think it says Next 40, but you might want to do Next 100 if that's the max they allow.


$mech->follow_link( text => 'Next 40');
[download]


http://search.cpan.org/~petdance/WWW-Mechanize-1.20/lib/WWW/Mechanize.pm#%24mech-%3Efollow_link(...)