I agree, the intermediate format should be plain text that could optionally be converted to any other format. I suppose that Markdown, however, is used as intermediate format here. It is close to plain text while it can preserve simple layout information.
In practice, I would use the Markdown output and plug it into any tool that converts that into the desired final output format.
That sounds reasonable. I might explore pdf -> markdown -> epub.
I wonder if this could somehow be used directly by calibre. I think calibre's pdf->epub conversion isn't amazing. In particular, tables often end up broken.
In practice, I would use the Markdown output and plug it into any tool that converts that into the desired final output format.