Beefy Boxes and Bandwidth Generously Provided by pair Networks
The stupid question is the question not asked
 
PerlMonks  

Re: How do I find peaks in noisy data?

by spiritway (Vicar)
on Apr 27, 2006 at 02:32 UTC ( #545934=note: print w/ replies, xml ) Need Help??


in reply to How do I find peaks in noisy data?

OK, so then my question is, how do you know when you have valid data, instead of just noise? That is, you might have input with a fairly small variation, where the peaks aren't all that much higher than the valleys. Would you be able to distinguish whether you had any good data among the noise? If so, how? What makes your values good or bad?

There needs to be some means of determining what your good input is. It could be the absolute magnitude of your values (a threshold value, say 500 in this case). However, you said that this wouldn't work - the noise levels, and presumable the levels of the good data, would be variable, so 500 might be within the noise range sometimes.

So another means might be if you knew how many good points you're supposed to have. In that case, simply order the points by value, and grab the top X% of them as "good", and ignore the rest. However, this requires that you know how many good points you're supposed to have.

If this is something that is somewhat periodic, you could use an FFT to get a set of frequencies out of the data; then apply the Kronecker delta function ($kron= $val>$threshold? $val: 0). That basically means, get rid of the frequencies that are not strongly represented in the data. Then do the inverse FFT to get your data back, minus the noise caused by unwanted frequencies. This would only work if you knew your data should be hovering around some definite set of frequencies, and not be all over the spectrum. Check out MathWorld for much more information on FFT, the Kroneker delta, and all that good stuff.

Finally, you can only do so much with bad data. You may need to take a look at the data source itself, if possible, and modify it to try to eliminate the noise at the very beginning.


Comment on Re: How do I find peaks in noisy data?
Download Code

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://545934]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others scrutinizing the Monastery: (20)
As of 2014-07-24 14:24 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    My favorite superfluous repetitious redundant duplicative phrase is:









    Results (160 votes), past polls