But you don't, in general, pick a prior. You pick a procedure that has an expected loss under various conditions. It's one player game theory.
If you happen to have a prior, then you can use it to choose a unique procedure that has minimal expected risk for that prior given the loss function, but even so that may not be what you want. For example, you may want a minimax procedure, which may be quite different from the Bayes procedure.
Minimax still requires a probability distribution, which means you need a prior.
Edit: Based on the downvotes, I see my audience is not convinced. I'll repeat an explanation I posted a while ago. Probably should make this a blog post because I see this claim quite often. I'd love to know what book you read it in.
--
In minimax regret, you have a set of available decisions D, and a set of possible states of nature N, and a utility U(D,N). Each state of nature also has a probability P(N) (which can be influenced by the decision too in some problems).
States of nature include "interest rates rise 1%", "interest rates fall 1%", and "interest rates stay the same". Decisions include "invest in stocks" and "invest in bonds".
Minimax regret proposes to ignore the probabilities P(N), instead suggesting a way to make a decision purely based on the utilities of the outcomes. But that is actually an illusion.
Outside of math class word problems, we don't have N or U(D,N) handed to us on a silver platter. There is always an infinite range of possible states of nature, many of which have a probability approaching but never reaching zero, including states such as "win the lottery", "communist revolution", and "unexpected intergalactic nuclear war".
In commonsense decision-making we don't include those states of nature in our decision matrix, because our common sense rules them out as being implausible before we even think about our options. You wouldn't choose to invest in bonds just because stocks have the most regret in the event of a communist takeover.
So what actually happens is we intuitively apply some probability threshold that rules out states of nature falling below it from our consideration. Then we minimize max regret on the remaining "plausibly realistic" states of nature.
Humans are so good at doing probability mentally that this step happens before we even realize it. But if you are writing code that makes decisions, you'll need to do it, and so you'll need to have at least a rough stab at the probability distributions.
Conceptually you are right: all mathematical models have assumptions, including assumptions about their scope of applicability.
But you are redefining "prior" to refer to all the assumptions of the model, and not its usual meaning as the prior distribution used in Bayes calculations.
Prior distribution is P(N|I), where I is the background information you have such as "historical interest rates in the USA looked like this", and "communist revolutions occurred 6 times in the 20th century" (made-up number). I is not itself the prior.
For this investing example, it's also the only information we have, unless we're trying to update on something like a central bank announcement. So our probability distribution over N is just the prior distribution.
When you're actually trying to make a decision, and not just solving a problem handed to you in math class, you can't avoid using P(N). You can either say "The minimax procedure requires knowing P(N) as an input, so that it isn't dominated by extremely improbable N", or you can say equivalently that "Minimax doesn't require P(N), but as an assumption of my model I'm ignoring all states of nature N with P(N) < y, then applying minimax regret over the remaining N".
I think we must be coming from two different communities of practice where the words don't quite line up. All the operational things you are saying I agree with. I just put them under different verbal categories.
"There is always an infinite range of possible states of nature"
Well, I think that is definitely and unambiguously false. The universe is not infinite, nor infinitely divisible, as far as we know, and the number of future states of any particular person (or humanity) are even smaller than those of the universe. Limits in time mean limits in space, and limits in space mean limits in particles and possibilities.
I'm not sure I can make a case that it matters, but if it doesn't matter, why say infinite?
I guess this is a tangent. First of all my point really doesn't hinge on the infinity; it can be finite (but really big) but regardless, whenever you apply minmax you must first crop your decision space to a probability threshold, or else you'll make nonsensical decisions based on what gives the best outcome if the sun should happen to explode.
But secondly, I think (although I'd happily concede if convinced otherwise) that the space of possible scenarios really is infinite, even if the observable universe is not. The space I'm talking about is not the actual state space of the universe, which in some interpretations of physics might be finite or even unitary. It spans the space of hypothetical universes that are all consistent with your information with nonzero probability, which I think is probably infinite, but again, if it's not infinite that's a technicality. If you include the states that have with zero probability (because why not? GGP was advocating that the probability is irrelevant to minmax decisions) then the space is definitely infinite, because even physically impossible states of nature will impact our decision making.
Another way to conceptualize the "cropping" is to get rid of all future states where planning would have been meaningless anyway.
We are momentary Boltzmann brains? We'll assume not, because if so, nothing really matters.
Trivial difference, but that avoids potentially difficult threshold problems and cousins of the St. Petersburg paradox or even Pascal's mugger, at the risk of being slightly more hand wavy.
Arguably an aesthetic distinction at this point, I generally think your description and approach are right.
I think it's sophistry to pretend we haven't any more idea of that since pre-Democritus. Thousands of years of science has shown that infinities are always a problem in our heads, with our theories. Does that prove they don't exist? No more than it's proven that the sun will come up tomorrow, I guess.
Just to be clear why this whole conversation thread is a TypeError, let's say I assign a probability of 99.9% to the hypothesis that the state space of the universe is finite, and 0.1% to the state space of the universe being infinite...
... In that case, how big is my hypothesis space about possible states of the universe?
There are plenty of examples of inifinities that are not problematic. Infinitely small wavelengths make our current understanding of physics break down, indeed. Or maybe infinitely divisible solids that lead to paradoxes like Banach-Tarski's. On the other hand, infinitely dimensional configuration spaces or continuous parameterization (e.g. coordinates, field strengths, phases) are trivial unoffensive parts of classical and quantum mechanics.
> Do you agree with Aquinas’ corollary, that absent an actual infinity, there must be some First Cause, which we call God?
Note that even if you agree with Aristotle’s position, which is essentially an arbitrary assumption, and the corollary that there must then be a first cause, there's nothing except the boat of being stepped in a particular religious tradition to suggest that the first cause should have any of the other traits of any particular concept of God. It works just as well to take the earliest known thing on the sequence of causes and say “this cause is uncaused”.
Aquinas' argument is that there can't be an actual infinity, so even though it appears that everything has a prior cause, it must be that there is something which is self-causing. "The Big Bang" qua event clearly didn't cause itself (events only cause events that are later in time), so the typical way to cash this out is "the Big Bang" qua set-of-physical-laws is self-causing.
This leads to a new problems (why this set of laws vs. some other), unless you posit that the laws are somehow perfect or necessary (which is essentially Deism), but the laws of our universe seem to be contingent (lots of unexplained physical constants).
You can make a metaphysically plausible case of a Big Bang-Big Crunch cycle that goes on forever, but then you're back to believing in an actual infinity.
> This leads to a new problems (why this set of laws vs. some other), unless you posit that the laws are somehow perfect or necessary
Those problems are only problems with the aesthetic preference that the universe be perfect or necessary. Once you accept that the universe can be without adhering to any such aesthetic preference, they cease to be problems.
So then the universe existing is a brute fact with no cause (as opposed to being self-causing).
You can do that but once you say there are facts without causes it's hard to know what you're signing up for. Why is this the brute fact and not something else? By definition, there is no answer (no cause) for that question. Uh okay, but if brute facts are possible, how can we do science at all? For all we know, we're just surrounded by brute facts and attempts to systemize facts into theories is just a waste of time because a new brute fact can just come along and bite you in the ass tomorrow. But I thought we only posited brute facts because science was pushing us in that direction by show us that there was a Big Bang, but now suddenly we're told "science is only contingently possible and sometimes just fails entirely due to the existence of brute facts".
It's not a satisfying intellectual stance, and if you really poke at it, it just feels like motivated reasoning in which the conclusion (there is no God) is leading the premises (some facts have no causes), not the other way around.
Heh, this conversation sounds like one I once had in university.
Here's the secret: Causality itself, that is, the notion that things have causes and A-->B (A causes B), is a concept that only makes sense within a system that has causal laws of physics and, in particular, a notion of ordering such as time. It applies to "event"-type objects such as A and B.
Since we humans live within a universe governed by causal physics, with a sense of order given by time and entropy, we observe events always having causes. But this is a property of events within our universe.
To ask whether the universe itself has a cause is a reasonable question, but to assert that it must have one, due to causality, is another TypeError. Universes, as a class of objects, are not governed by the same laws of physics as things within a universe. Time itself is in a sense a member variable of our particular universe, remember. So universes are not subject to causality any more than they're subject to gravity. Causality and gravity both apply to things within universes.
It might be that there are other laws that govern the formation and structure of universes. But we won't be able to infer very much about them by performing experiments within our universe.
To help you visualize this concept, think about a cellular automaton like Conway's game of life. That game has particular laws of physics, and can run on a PC. The evolution of the game state, though, is not closely coupled with the PC's environment. The PC can pause the game, or run it at 100x speed, or run it backward (if it has reversible laws; Conway's doesn't), but from the in-game perspective, it wouldn't be noticeable. Within the game, it would perhaps be possible to perform experiments to discover the governing rules of the cellular automaton, but there's not really many experiments that a one could do within the game to learn about how the PC works.
You can even build a Turing machine within the game, and have it run another kind of program. There wouldn't be a way for an AI within that program to distinguish that the Turing machine running it exists as a cellular automaton, as opposed to any other Turing machine, let alone to discover the PC at the upper level.
There's no reason to think that the same laws of physics apply at higher abstraction layers, and so it's entirely possible that our universe has no cause, because causality itself is an in-universe concept.
In my analogy, God (if used a meaningful term with all the cultural baggage that it carries) maps much more closely to John Horton Conway than to either the universe or the PC. I'm not asserting that Conway does or doesn't exist, merely that a Game of Life does not necessarily imply a Conway.
> You can do that but once you say there are facts without causes it's hard to know what you're signing up for.
Once you start inventing unjustified entities to satisfy an aesthetic preference for things (except the invented entities!) to have causes, you know what you are signing up for—a perception of reality driven by your desires rather than justified belief.
That the universe exists is a fact. The question of whether the existence of the universe has a cause may not be answerable, and there's a pretty good argument that it asking what the cause of the universe existing is itself is as incoherent as asking what the color of 1+1 is. To assert anything as a prior cause of the universe is to assert an entity outside of the universe, which is just equivocation because the “universe” in the question is the sum total of all existence.
> Uh okay, but if brute facts are possible, how can we do science at all?
Quite easily.
> For all we know, we're just surrounded by brute facts and attempts to systemize facts into theories is just a waste of time because a new brute fact can just come along and bite you in the ass tomorrow.
Of course, that's the fundamental nature of science. It's always contingent, but we build on what has observed predictive utility, because if there are any systematic rules, that's the only even loosely objective way to discern them. Accepting that scientific knowledge is inherently contingent doesn't prevent doing science.
> It's not a satisfying intellectual stance
Satisfaction is subjective; clearly, it doesn't appeal to your aesthetic preferences.
> and if you really poke at it, it just feels like motivated reasoning in which the conclusion (there is no God)
That's not the conclusion. Rejecting a particular argument for the necessity of a First Cause (which, while it gets abused as one, wouldn't be an argument for the necessity of anything much like the image of God it is used to justify even if it was valid on its own terms) isn't the same as denying the existence of God (I'm, as it turns out, a Catholic who quite firmly believes in God, so the distinction is not merely theoretical.)
> is leading the premises (some facts have no causes),
That's not the premise, either. Rejecting as unwarranted the assertion that all facts must have causes isn't asserting the existence of uncaused facts. (Though the argument from First Cause is asserting the existence of uncaused facts, so it's kind of odd for someone defending that particular God-as-brute-fact argument to mock the—imagined, but not actually real—premise of others that brute facts exist.)
Of course, St. Thomas Aquinas’ argument (and the similar though different previous effort at proving the logical necessity of God by St. Anselm) are the actual motivated reasoning in the debate, not the rejection of those arguments.
We already believe from quantum mechanics that there are random events that have no prior cause going on all around us. Even creation ex nihil seems to be going on constantly at the smallest scale, with virtual particles popping in and out of existence governed only by laws which ensure the conservation of energy.
So, whether correct or not, our scientific understanding already posits effects without causes. Ascribing no cause to the big bang itself is then not an extra assumption, so no contorted logic is necessary to get rid of the First Causer.
And related to your description about brute facts - while perhaps unsatisfying, I believe that it is how many scientists do perceive the world. We have a set of observations and we try to come up with the simplest set of laws that describe these observations, and test their predictive power on new facts as they come along. Sometimes, we discover that our set of laws had some hidden assumption that we were not aware of, such as the surprising facts about the speed of light being constant when measured from moving vehicles leading to the realization that Newton's laws of motions only hold for small enough speeds, and the need for general relativity to accurately describe what happens at a larger scale.
Even today, we have clear, well known gaps in our scientific understanding: the standard model only applies for matter at certain energy levels; quantum mechanics only applies at certain scales and can't take into account gravitational effects; we have mathematical singularities that come up when trying to describe black holes, which are unlikely to be physically correct; and we don't know what most of the matter in the universe is made of; and I could go on.
Scientific understanding is known to be contingent and any day a new unexpected fact could be observed, toppling our understanding of the laws of physics. That is a well known and inescapable fact.
All of this is not to mention that it still seems disingenuous to call the posited First Cause 'God', bringing to mind YHWH. I would be more inclined to accept the idea that there is some kind of transcendental First Cause (though, again, I don't think that is a particularly necessary concept), but you would need many more arguments to go from that to any particular conception of god.
Also, all of this discussion relies on certain assumptions about the real world and our ability to perceive it. For example, physics is also entirely compatible with the Hindu notion of Maya, that the world is an elaborate illusion, with everything we perceive actually being like drops of water in the ocean that is God, Brahman. Not to mention that even our understanding of logic is contingent. Perhaps there is some limitation of our biological brains that prevents us from seeing some fundamental flaws in our arguments.
All in all, my point is that it is not possible to obtain true certainty beyond any possible doubt on any topic. The best we can do is choose some base assumptions we believe in, and try to see what we can understand of the world starting from those. And if you chose God as one of those base assumptions, that is perfectly legitimate, and you can get a coherent model of the world that includes it. But you can't convince someone who has chosen pure empiricismaas their base assumption of the necessity of this base assumption.
But you don't, in general, pick a prior. You pick a procedure that has an expected loss under various conditions. It's one player game theory.
If you happen to have a prior, then you can use it to choose a unique procedure that has minimal expected risk for that prior given the loss function, but even so that may not be what you want. For example, you may want a minimax procedure, which may be quite different from the Bayes procedure.