It is always hard to know when to retire legacy software. It is almost always faster (cheaper) to shoe-horn in one new feature than it is to start over. However, we often see with hindsight that if we had paid the start-over price three versions ago, we would already have recovered that cost and every future version would have been cheaper.
We only see a cost benefit in a redesign if we really do make a significant improvement and if enough future changes benefit from it that we recover the investment. On the first point, our new designs are seldom as good as we expect. Customers seem to find new requirements that do not quite fit with our brilliant designs. On the other hand, the need for any program usually lasts much longer than we expect. In summary, you must predict the number and nature of future of changes. This seems impossible, but remember that your best guess is almost certainly better than not even trying.