> I've been told that it's irresponsible, that I'll be endangering him

With all due respect, whoever is telling you that is should mind their own fucking business.