Probably less of an issue than you think. It's pretty common in DC. There are recruiters who specialize in placing people who work in "Clearance" jobs.
If you want a job at another cleared contractor. Getting out of this very siloed world is really hard, though. For a number of reasons, actually; nit being able to provide detail is just one of them.