Before we can suggest an improvement to your algorithm, we need to see what you are doing now. I imagine you are just brute-force calculating the distance from each atom to each other atom in your array and comparing that to your cut-off distance? The math there is pretty simple .. i don't think there's going to be a "trick" to calculating the distance between two points.
Any algorithmic trick is most likely going to involve doing FEWER calculations, not speeding up the calculations you already do.
Let's say you have your points A(1), A(2), ... A(n), each represented by their coordinates. I assume that at some point you calculate the distance from a(1) to a(2). Once you've done that, is your algorithm smart enough to not calculate the distance from a(2) to a(1) later? I would think so, but short of seeing your code, I make no assumptions.
