Your shop has created such an edge-case ... such an enormous volume of both variables and data that a single program is expected to process ... that I think your only realistic alternative (as a shop ...) is to: (a) in the short run, “throw silicon at it.” Then, (b) in some way, start fundamentally re-defining the problem and therefore its approach ... splitting or sharing the file such that multiple blade processors (not cores, not threads, not processes...) can tackle it in parallel, reducing the number of variables from 30-god-thousand to only what is required, and so on.
The trouble with “removing my” is that it really is a fundamental logic-change to the program. No matter how many years this piece of source-code has been buying your groceries, errors will be introduced, and when they do, how-the-hell will you know? Data structures that used to be known-empty no longer are, and so on. The business consequences could be disastrous indeed. (And I don’t mean to under-state the business risk of “redefining the problem,” but it is a Hobson’s Choice by now.)
Basically ... and this needs to be raised right-now as a senior management issue ... “we are running out of track. We have been putting this thing off and trying different languages and so forth, but our data volume is catching up with us.”
