Many good suggestions already, but I’ll throw my hat into the ring.

You could say something like “you and I have very different tastes in travel, but I still like traveling when it’s on my terms.”