That's a reasonable point. So long as the panel's output voltage is higher than the controller's minimum required input it would be stable.

I still wouldn't want to run a Pi directly off solar. I'd want enough warning to shut it down rather than killing the power.