CSV = Comma Separated Values, is nothing more or less than a data-exchange format ... mostly concerned with what to do when the content of a particular data field contains either a comma or a newline. It has no notion of formatting.
I guess from your post that what you’re really trying to do is to get data into an Excel spreadsheet. In this case, a tool such as Excel::Writer::XSLX might be more to your liking. The CSV file-format does not have (and was never intended to have) the expressiveness that your present requirement appears to demand.
You should also consider whether it would be appropriate (in a Windows environment) to use OLE. Launch “Excel, itself,” tell it to create a Spreadsheet object, then feed data into it ... formatting and all. Tell Excel to save the spreadsheet and you are done.