In addition to Kamshak's note about parallel inference accumulating float errors differently due to order of operations, which makes LLMs theoretically non-deterministic at temperature 0, there is the issue of them being practically non-deterministic as-deployed, not just via temperature but because of inclusion of prior "turns" in context, variations in phrasing of prompts, etc.
It's also "non-deterministic" in the sense that if you removed all sources of non-determinism and asked "What is 1+1?" and received the answer "2" deterministically, that doesn't guarantee a correct answer for "What is 1+2?". Ie a variation in the input isn't correlated in a logical way with a variation in the output, which is somewhat fatal for computer programs, where the goal is to generalize a problem across a range of inputs.
It's also "non-deterministic" in the sense that if you removed all sources of non-determinism and asked "What is 1+1?" and received the answer "2" deterministically, that doesn't guarantee a correct answer for "What is 1+2?". Ie a variation in the input isn't correlated in a logical way with a variation in the output, which is somewhat fatal for computer programs, where the goal is to generalize a problem across a range of inputs.