I am not a computer vision expert. Here's how I would do it:
in reply to BMP: Drawing or photograph?
Load some representative target images into Mathematica or similar program and try out various filters. Start with transformations related to edge-detection or finding continous areas of colour and brightness. Visit http://mathematica.stackexchange.com/questions/tagged/image-processing if you need help.
When you have found a filter or combination of filters that correctly recognise to a high confidence whether an image is suitable for JPEG compression or not, recreate them in Perl. Binding to various imaging toolkits written in C are available, so this shouldn't be too complicated. The filter (pipeline) simply returns a boolean, use PNG or JPEG output format accordingly.