For the last few hours I was trying different ways of storing (serializing) a large data structure to disk.
Just to make sure you understand perfectly clear because you seem to be ignoring the advice: the tool that a competent engineer reaches for when presented with this task is a database.
Otherwise, all you are doing is (poorly) reinventing a database. Even if you pull off this particular task, the very next thing you are going to be asked to do with the data is something that is going to be very simple with a database, but extremely difficult or impossible to implement in your custom data format.