To understand the performance of algorithms, you need to count how many steps the algorithm will take to come up with an answer. People tend to think of that kind of counting operation as math.
There is no way to convey the concept of how to calculate the efficiency of an algorithm without actually going through how to count operations. In that sense the concepts are math.
That said, you certainly do not need to master all kinds of advanced math to be able to handle this kind of counting. For instance expertise in differential equations will not prepare you to understand why a quicksort is on average better than a bubblesort, and why a mergesort has better performance guarantees than quicksort, even though its average performance is worse (if all memory is equally fast to access).