It is difficult to learn if you're only used to programming in an imperative style, but it's worth persevering. I sometimes found that some bits only really made sense to me if I skipped over the chapter, saw how it was used in practice, and then came back for the explanation.
As for not using it - I rarely programme in an entirely functional style, but I learned a huge amount from HOP that I find incredibly useful every day, both in my fun-time programming and at work. There are some problems where it's really useful to be able to construct functions on the fly which are variations on a template, and to throw functions around as data. Sure, you can always find other ways of doing it, but the functional way is easier to read and debug.