It absolutely is, if it makes compile times unreasonable for reasonable code. Compilers have to make trade-offs like this all the time, they can't use overly excessive optimizations.
Because it can still result in compile times I find excessive.
For example breaking up a function that takes 5 seconds to compile into a bunch of functions that take 1 to 2 seconds in total.