Beefy Boxes and Bandwidth Generously Provided by pair Networks
Clear questions and runnable code
get the best and fastest answer
 
PerlMonks  

Time for Some Movie Magick

by abitkin (Monk)
on Oct 14, 2004 at 14:12 UTC ( #399213=CUFP: print w/replies, xml ) Need Help??

Background:
So, let's say there's a webcam out there, with pictures taken at regular intervals. No, not that type of webcam, I'm talking about one of outdoors, so you can see what's happening outside your little cube. Now staring and refreshing at that little image 24/7/365 isn't really your cup of tea. I mean, shouldn't you use wget to grab the images for you?

So suddenly, you have hundreds of images, but what good are they? I mean yeah, it's cool to say you have a record of them, but there must be a better use. After scratching your head for a while it dawns on you, what about an animated gif? I mean wouldn't that show you what was happening better than that little jpg you have? So after some digging you find image magick, and you start to conver these jpgs into an animated gif, when it hits you, animated gifs are bad. Too few colors, and compress jpgs to gif sucks.

What's a developer to do? After a little more thought it dawns on you, that perhaps a movie would be a better format. After playing with image magick some more you find that it can convert jpgs into mpgs. Wonderful!

So for a few days you convert everything into the same movie. Each time your cron job runs, it gets progressively longer. Finally, when it takes 72 minutes to run, you say enough. It's time to start searching for a better way. Perhaps if you just did smaller time increments.

So with this new goal you start off on trying to make your scripts understand that an image only belongs to a day, and that you want movies of each day. Enter this cool use for perl.

FYI: in this little trip down my train of thought, all the references to you are actually thoughts I had.

This is the calling script that the cron job kicks off. It should download the images, and rename them to epoch seconds. Fifth, yes I know I shouldn't use system as I do, but in this particular case, I figured the chances of actually having this hole exploited was low, especially since I know the names of all the images before hand.

#!/bin/bash #you need to do a wget here. image_path=some_path image_dest=some_other_path for i in `ls ${image_path}*.jpg` do mv -f $i ${image_dest}/image-`stat -c "%Y" $i`.jpg done touch .rename.config.txt ./convert_day.pl ${image_dest}/*

So now we get to the perl (finally.) Things you should note, first you should have write access to . Second, this uses a persistent data structure. Third, becuase of it's persistent data structure, it would be bad to have more than 1 copy running at the same time in the same path. Fourth, yes I know there can be a lot of time improvements, but those are minimal compared with the time it takes to run convert on 10+ images.

#!/usr/bin/perl use strict; use Time::localtime; use Data::Dumper; my %pics; my %exec_hash; my @elements; my $FH; my $final_path="some_path"; # the user should ensure that this script is only run serially open(FH, "< .rename.config.txt"); my $line; my $lines = ""; while (defined ($line = <FH>)){ $lines .= $line; } %exec_hash=%{eval $lines} unless $lines eq ""; close(FH); # create the movie names and add images to the movies for(@ARGV){ my $temp = $_; my $tm = localtime((stat($temp))[8]); my $nm = sprintf("%02d%02d%04d", $tm->mon+1, $tm->mday, $tm->year+ +1900); push @elements, $nm unless defined $pics{$nm}; $pics{$nm} = [] unless defined $pics{$nm}; push @{$pics{$nm}}, $temp; } # convert the images to movies for(@elements){ my $t = $_; my $line = "convert -delay 10 " . join(" ", @{$pics{$t}}) . " ".$f +inal_path."/" . $t . ".mpg 2>/dev/null 1>/dev/null" ; # As processing a lot of images takes a long time # only update those images which have changed if(!defined($exec_hash{$t}) || $exec_hash{$t} ne $line){ system $line; $exec_hash{$t} = $line; } } my $save = pop @elements; system "cat ".$final_path."/". (pop @elements) .".mpg ".$final_path."/ +". $save . ".mpg > ".$final_path."/current.mpg" ; open($FH, ">.rename.config.txt"); $Data::Dumper::Terse = 1; $Data::Dumper::Indent = 1; print $FH Data::Dumper->Dump([\%exec_hash],['exec_hash']); close($FH);

Feel free to comment away, suggest improvements, ect. One thing to note, mpgs can be catted together, thus reducing the time taken even more; however, I did discover a couple cases where the images were updated as wget was running, thus it missed an image, (which it filled in later using the kickoff script.) Also note, rsync commands can be added to the kickoff script to upload the movies automatically.

The results of all this are here.


==
Kwyjibo. A big, dumb, balding North American ape. With no chin.

Replies are listed 'Best First'.
•Re: Time for Some Movie Magick
by merlyn (Sage) on Oct 19, 2004 at 21:28 UTC
    for i in `ls ${image_path}*.jpg`
    I wonder what the thinking is that gets people into this twisted level of code. Maybe it's just seeing it somewhere else, then cargoculting it without thinking.

    So, given that this is likely the case, let me say, Don't do it this way.

    Here's what you're asking your poor system to do:

    1. Shell, please fork
    2. Child shell, figure out what matches ${image_path}*.jpg
    3. Child shell, now pass that list to an ls that you forked
    4. ls, please go look up this list of names that I've already handed you. (And, by the way, if any of those are directories, expand them, even though that isn't what I want, so it'll be a bug I have to fix later.) And spit those names out one by one, with a newline.
    5. (Parent) shell, take the output of that child process, and create elements of a list, newline delimited (which breaks if any of the names contain whitespace such as newline, but that's another bug to be fixed later).
    6. Shell, start looping through that list, doing the following...
    Now, let's see how we could have written it, properly:
    for i in ${image_path}*.jpg
    Yeah, that's it. Simple. Now look what we told the system to do:
    1. Shell, figure out what matches ${image_path}*.jpg
    2. Shell, start looping through that list, doing the following...
    Not only is it faster and shorter to type, it's now also whitespace safe, newline safe, and directory safe!

    So, why does the other form get cargo-culted so much? It's like the useless use of cat that I keep seeing all the time. My guess is that it's like that in a bad book or a bad FAQ, and then it gets copied and pasted from one person to the next, like a magical formula that nobody understands and yet repeats, as if the magic words will open up the door if uttered in the proper moonlight.

    Just curious, I guess. Remember, this isn't personal... I'm just trying to figure out your thinking so that I can work out a counter-meme to it to help everyone at once, instead of one at a time like I keep ending up doing. Call me lazy. {grin}

    -- Randal L. Schwartz, Perl hacker
    Be sure to read my standard disclaimer if this is a reply.

      Merlyn,
      I thank you. This is actually very constructive critisim. Hopefully, it's obvious that I don't do a lot of bash programming, and I picked the mentioned code up from other bash scripts I had laying around and guess and check. Unfortunately what I've found online very rarely explains in detail as you have, and is often times more like my code than yours.

      Once again, I thank you for keeping me honest, and for attempting to understand my thinking. I know it's scary inside my head, heck I have to live there every day.


      ==
      Kwyjibo. A big, dumb, balding North American ape. With no chin.
Re: Time for Some Movie Magick
by elwarren (Curate) on Oct 19, 2004 at 20:22 UTC
    Very nice. The movie is cool too, better than anything I've seen on tv so far :-)

    I have been working on a similar project and just have a few thoughts to share regarding some problems I've run into.

    I grab an image once every 30 seconds via wget called from cron. A nightly job moves these images out of the download directory and into directories sorted by date. With that many images I started exceeding the command line arg limits. Now I sort images into subdirs broken down to /cam/year/month/day/hour/

    The cool part about this is that I can easily generate a movie of, say, every sunrise in the month of July with a command like: jpegtoavi -f 6 320 240 /cam/2004/07/*/05/*.jpg I'm an oracle guy but this works so well I haven't gotten around to building the db portion.

    I build my movies as MotionJPEG videos using the jpegtoavi tool. It is much faster than building mpegs with convert and there is no loss of detail or artifacting like you sometimes get with mpegs. Your video is actually very clear, I'm going to have to revisit the way I build my mpegs for dvd burning.

    I use a tool called fdupes to get rid of duplicate images. The cam I monitor starts at 7:00am and shuts off at 10:00pm, but I grab images for 2 hours before and after, to accomodate for the server clocks being out of sync and clock changes in spring/fall. This was faster than the solution I had written in perl :-(

    Again, cool project. Glad I'm not the only crazy guy watching some cam that doesn't do much unless you look at it a day at a time :-0
      :) yeah, rather than the cron every 30 seconds, I found the directory where they store the image, (along with the last 250 of them.) And wget the index (as they have an apache dir listing as their index) gets me all of them. Anyways, here's the link to my webserver. It won't handle a lot of traffic, but a little it may do. Shuts down at 10 pm, starts at 5 am, movies are done on another machine and uses rsync to upload.

      ==
      Kwyjibo. A big, dumb, balding North American ape. With no chin.

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: CUFP [id://399213]
Approved by Joost
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others exploiting the Monastery: (5)
As of 2020-06-01 00:26 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?
    If programming languages were movie genres, Perl would be:















    Results (177 votes). Check out past polls.

    Notices?