You started me on an interesting trail of discovery, and I can share a little of it with you.
I dug out an article by Frederick L Bauer who invented the stack, and even tried to read his original paper, written in 1950. He talked there about 'pushdown' data storage and it was known as the Kellerprinzip or cellar principle. So I reckon that's where 'push' comes from.
I was not surprised to find that Bauer's work was based on original papers by a certain Alan Turing, written in 1934. He is most famous for his work with breaking the Enigma code in the second world war. But go look him up in your resource of choice, and you will find his ideas in many other areas, including AI and neural networks.