Beefy Boxes and Bandwidth Generously Provided by pair Networks
Welcome to the Monastery
 
PerlMonks  

Help improving image recognition application to track store purchases.

by zli034 (Monk)
on Jan 08, 2007 at 07:16 UTC ( #593488=perlquestion: print w/ replies, xml ) Need Help??
zli034 has asked for the wisdom of the Perl Monks concerning the following question:

Dear monks:

I planning to do some product researches in my little clothing store. My problem is that the products in my store all have no barcodes on them. I cannot use barcodes to keep record on each item I sold. I have more than couple thousands items in the store; every day we will renew hundrads sold items, print individual barcodes become a hard job and needs training.

My solution to the problem, I don't need barcodes, I just take a photo for each item when it is sold. Then I can apply AI technics to recognize and classify the photos. Then I would be able to know, say, how many pants, tops or skirts I sold each day.

I developed a technic to calculate some indicator values of each photo. I tested it on photos from only three types of items. It worked quite good. However there are less than a hundred types item in the store, and many of them are very similar on colors, I consider the technic will not be powerful enough. I have some codes after you guys reading this; it was modified from a monk merlyns code.

Can anyone of you have specific image recognition knowledges to introduce me a professional way to do this? By the way, I thought I can use perl to control the PC camera to make taking photo easy. Fanally I figured out that's impossible under at least windows OS. So just processing photos, can we do it?

#!/usr/bin/perl #by monk zli034 use strict; use Image::Magick; use Cache::FileCache; my @files = (); my $folder = "c:\\photos"; my $FUZZ = 5; # permitted average deviation in the vector elements my $CORRUPT = "CORRUPT"; # if defined, rename corrupt images into this + dir my $cache = Cache::FileCache->new; unless (opendir(FOLDER, $folder)){ print "Cannot opne folder $folder!\n"; exit; } @files = grep (!/^\.\.?$/, readdir(FOLDER)); closedir(FOLDER); #print join("\n", @files), "\n"; sub warnif; my $write="ID,mean,stdv,class\n"; foreach my $file (@files){ #the first number of the photo file gives the #information of the type of the item. my $class; if ($file =~ /^1/){ $class = "first";} if ($file =~ /^2/){ $class = "second";} if ($file =~ /^3/){ $class = "third";} #print $file; my $image = Image::Magick->new; my $x = $image->Read($folder."\\".$file); #print $file; my @vector; #print "... skipping ($file)\n"; #print "\n"; my (@stat) = stat(_) or die "should not happen: $!"; # retur +n 13 elements of file status. my $key = "@stat[0, 1, 9]"; # dev/ino/mtime print "$file is ", join("x",$image->Get('width', 'height')), + "\n"; warnif $image->Normalize(); warnif $image->Resize(geometry => '4x4!'); warnif $image->Set(magick => 'rgb'); @vector = unpack "C*", $image->ImageToBlob(); print "vector ", @vector, "\n"; $cache->set($key, [@vector]); my $mean = mean(@vector); my $stdv = std_dev($mean, @vector); print "mean $mean \n"; print "standard deviation $stdv \n"; #$write=$write.$file.",".$mean.",".$stdv.",".$class."\n"; foreach my $element (@vector){ $write=$write.$element.","; } $write = $write.$class."\n"; } #Output result-------------------------------------------------- open(OUTFILE, ">testdata.csv") or die "Can't open testdata.csv: $!"; print OUTFILE $write; close OUTFILE; use Carp qw(carp); sub warnif { my $value = shift; carp $value if $value; } sub mean { my $result; foreach (@_) { $result += $_ } return $result / @_; } sub std_dev { my ($mean, @vector) = @_; my @elem_squared; foreach my $element(@vector) { push (@elem_squared, ($element **2)); } return sqrt( mean(@elem_squared) - ($mean ** 2)); }

Comment on Help improving image recognition application to track store purchases.
Download Code
Re: Help improving image recognition application to track store purchases.
by arkturuz (Curate) on Jan 08, 2007 at 10:13 UTC
    I believe that barcodes are invented to avoid the solutions you propose. Why don't you buy some empty stickers, print serial numbers on them, and stick them on items. There are also bar code generators, RFIDs, etc. Use the computer to solve the problems, not to invent them.
Re: Help improving image recognition application to track store purchases.
by zentara (Archbishop) on Jan 08, 2007 at 12:24 UTC
    I agree with arkturez. In inventory management, you need certainty. AI photo-recognition will be tricky, unless you make some sort of new-fangled pseudo-barcode, like a series of color dots in the upper right corner. So you are still printing.

    I can just see the hours it took to trace down the bug, "why you bought 500 bikini bottoms instead of jogging shorts" , due to the sunlight angle at 3PM on cash register #3.


    I'm not really a human, but I play one on earth. Cogito ergo sum a bum
Re: Help improving image recognition application to track store purchases.
by Joost (Canon) on Jan 08, 2007 at 13:13 UTC
    My solution to the problem, I don't need barcodes, I just take a photo for each item when it is sold. Then I can apply AI technics to recognize and classify the photos. Then I would be able to know, say, how many pants, tops or skirts I sold each day.
    Well, if you get that to work, you should be able to retire quite comfortably on the profits. More likely though, is that you won't ever get it even remotely as reliable as barcodes/rfid which are made to do just that and even then they're not 100% reliable.

    As an example of an image recognition system: this link describes a traffic-sign recognition system. Note that traffic signs are designed to be easily differentiated. This thing will recognize 3 signs a second and has 98% "recognition rate" (which in my estimates is much worse than the usual barcode scanner at the super market)

Re: Help improving image recognition application to track store purchases.
by shmem (Canon) on Jan 08, 2007 at 13:23 UTC
    every day we will renew hundrads sold items

    how many hundreds? 200, 300, 400 or 500? at what photo quality? 640x480, 24bit makes roughly 100kB. That's a lot of data to pass around, when 256 Bytes for each item would be enough to provide a unique identifier to each item you sell in the entire lifetime of your shop.

    Sum up the time for taking and postprocessing the photos: you'll miss that at some point.

    If you delay processing, you have ~200MB of raw photo data per week, selling 400 items a day, 5 days a week - your hardware shop will thank you for your approach...

    --shmem

    _($_=" "x(1<<5)."?\n".q·/)Oo.  G°\        /
                                  /\_¯/(q    /
    ----------------------------  \__(m.====·.(_("always off the crowd"))."·
    ");sub _{s./.($e="'Itrs `mnsgdq Gdbj O`qkdq")=~y/"-y/#-z/;$e.e && print}
Re: Help improving image recognition application to track store purchases.
by lin0 (Curate) on Jan 08, 2007 at 13:58 UTC

    Hi zli034,

    Using AI techniques to recognize and classify the items of your store is a bad idea. There are so many challenges to be solved in your particular application that the computational cost and the training required of your personnel will be many times higher than using an alternative as putting bar codes on the items. Just to give you an idea of the challenges, lets focus only on the shape matching part (even though, you already mentioned that you also need to match for color). Here you will have a number of challenges: your system has to be good enough to be robust against changes in orientation, occlusions, and deformations. To minimize the problem, you have at least two options:

    1. train your cashiers to place the item in the same position every time, to adjust it to have a certain shape that will be easily recognized, and to avoid getting in the way of the camera. This will involved a lot of training. You could check the challenges associated with image guided surgery applications (for that, you would need to do a Google seacrh) to have an idea of all the problems of solving this type of problems in real time (you will need real time unless your customers are willing to spend a lot of time at the cashier)
    2. develop a state of the art algorithm that can do the shape matching in real time (for the sake of your customers), has a 100% of accuracy (for your own sake) and is robust against occlusions, changes in light conditions, deformations, and changes in orientation. If you succeed in making such an algorithm, you better patent it because you cannot imagine how valuable it would be.

    In short, you better go with the bar code solution or with any solution that involves identifying a small identifier that you put on every item.

    Please, let us know if you have another question.

    Cheers,

    lin0
Re: Help improving image recognition application to track store purchases.
by Melly (Hermit) on Jan 08, 2007 at 14:17 UTC

    Heh - well, if nothing else, your intended approach embraces the 3 virtues of a perl programmer...

    Beyond what others have said, it's worth mentioning that you are not required to bar code every item - a catalog of items with their bar codes at each till might be a better approach - particularly if your 2000 items are actually 100 items in 20 different colours (i.e. maintain bar codes for the 100 items with separate bar codes for the colours).

    map{$a=1-$_/10;map{$d=$a;$e=$b=$_/20-2;map{($d,$e)=(2*$d*$e+$a,$e**2 -$d**2+$b);$c=$d**2+$e**2>4?$d=8:_}1..50;print$c}0..59;print$/}0..20
    Tom Melly, pm@tomandlu.co.uk
Re: Help improving image recognition application to track store purchases.
by talexb (Canon) on Jan 08, 2007 at 15:17 UTC

    I studied Pattern Recognition under Ed Jernigan at the University of Waterloo, and I gotta tell you, it's tough slogging -- quite the fascinating subject -- but tough math.

    To me, this sounds like a solution looking for a problem -- I think you need to stop and think about what problem you're trying to solve. If the problem is inventory control, it's probably a piece of cake to make up a dozen or so types of tags, one type for each piece of clothing, a sort of apparel taxonomy. You don't care what colour the sweater is, but you care if it's a sweater or a parka. So just go from there.

    Or maybe it's some other problem you're trying to solve ..

    Alex / talexb / Toronto

    "Groklaw is the open-source mentality applied to legal research" ~ Linus Torvalds

Re: Help improving image recognition application to track store purchases.
by kyle (Abbot) on Jan 08, 2007 at 15:18 UTC

    First, I agree with the many others who have said that this is the wrong approach to the problem (and I have to wonder if this is what you were posting about last week).

    Second, it might have been helpful to link to the code you're cribbing, since there's a better description there of how this technique works.

    Third, I wonder about the utility of looking at the mean or standard deviation of an image's resulting vector. Imagine an image that is half black and half white and another that has the halves reversed. They're as different as night and day, but they have the same mean and standard deviation. If you had so many images that doing all the comparisons becomes a chore, narrowing your search using these values might be useful (or it might not), but you don't seem to have that many.

    Finally, you might want to look at earlier nodes dealing with image recognition, such as Comparing images.

Re: Help improving image recognition application to track store purchases.
by kwaping (Priest) on Jan 08, 2007 at 15:58 UTC
    If you end up changing your mind based on the (excellent) replies to your node, I'd like to point you towards GD::Barcode.

    ---
    It's all fine and dandy until someone has to look at the code.
Re: Help improving image recognition application to track store purchases.
by davido (Archbishop) on Jan 08, 2007 at 17:06 UTC

    I applaud your effort, and actually the fact that you've gotten as far with it as you have. I'm impressed.

    But your strategy is flawed. Point of Sale has to go as smoothly as possible. Any slowdown at POS will cost you sales. There is no way that you can take a controlled picture of sufficient quality to guarantee image recognition accuracy faster than you could jot down a six digit number, and even worse, your salespeople or clerks aren't going to be even half as careful as you. This is especially true when they've got a line of frustrated customers at the counter, impatient over the notion that everything they buy has to be photographed carefully first. POS slow-downs are one of the biggest No No's in the retail environment.

    If barcoding is too expensive to implement, at least just generate sku's. Printing sku labels isn't terribly difficult, and that will make inventory tracking simple, efficient, and in keeping with what clients and clerks are accustomed to.

    That doesn't mean the project doesn't interest me. It really does. But the purpose for which you're applying the project is not ideal, imho.


    Dave

Re: Help improving image recognition application to track store purchases.
by qbxk (Friar) on Jan 08, 2007 at 17:38 UTC
    I have nothing useful to add, but I too applaud your hubris. I can't believe you have something buggy, which implies that you have functionality that only works *sometimes*. And that's a lot more than I'd expect from this idea. You're either crazy or stupid or just ahead of your time. (likely the latter)

    If you'd used Perl6 though this would be a snap, isn't that right monks?

    fine. i can't help but think when i write. here's something useful: I bet all the labels or the tags on your merchandise have more intricate patterns that maybe you'll be able to more reliably "parse". Say, a shirt, snap a picture of just the label up on the back of the neck, grab a little background of the shirt too (in case you have a few shirts with the same labels). That image should result in some "hash", which you'd be more likely to get back when you snapped the shirt's label again later, since there's less "complications" in that picture. In other words, the label changes less as you fling the shirt around in real life (shirts get wrinkled and have different colored tables or objects underneath them, etc). I think the "value" in this idea boils down to: don't snap a picture of the whole object when just it's most unique tiny patch of the object will do. I'd imagine that from picture to picture, there are fewer differences when you deal with smaller areas. Of course that's almost exactly what barcodes are.. a whole bunch of unique 'areas' to affix to your, otherwise identical, crap.

    And let us know how this turns out, you crazy merchant


    It's not what you look like, when you're doin' what you’re doin'.
    It's what you’re doin' when you’re doin' what you look like you’re doin'!
         - Charles Wright & the Watts 103rd Street Rhythm Band, Express yourself
Re: Help improving image recognition application to track store purchases.
by brusimm (Pilgrim) on Jan 08, 2007 at 18:24 UTC
    Awesome effort,.. BUT

    barcodes - I know the initial time expenditure does not look good to your immediate needs, in as much as the training and potential equipment additions to your business. But the long term benefits should far outweigh any time spent upfront in the process, while digital photos will more than likely continually create new and unique opportunities to have to work on process modifications as products change in any number of unanticipated ways down the road.

    If you take a week or so now for development / training, you can cut your handling time over the long haul immeasurably, saving you time and money. And besides, I have never bought something and had to hold it up for a photograph! I don't think you want that kind of marketing word of mouth, do you?

    Good luck

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: perlquestion [id://593488]
Approved by chargrill
Front-paged by andyford
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others making s'mores by the fire in the courtyard of the Monastery: (7)
As of 2014-11-28 18:23 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    My preferred Perl binaries come from:














    Results (199 votes), past polls