Indeed. In order to even try to cross the blood brain barrier, an organism would have to be in the blood already. If that's unexpected, the immune system certainly won't like it. Large compounds and even immune cells aren't supposed to cross it. Usually brain infections are from tiny viruses because of that, unless they found another way (like ear, nose or eye infections, all of them effectively brain 'appendages').

Now, it could be the case that in a person with a compromised immune system AND a compromised blood brain barrier, that organisms would be able to live long enough to reach the brain. Once there, they would be mostly shielded from the immune system, except for brain glial cells (and I guess antibodies; usually the blood brain barrier stops most of them)