This is connected to the equivalence relationship between optimal indexing and optimal AGI. The "best" way is optimal for the entire universe of possible queries but has the downside of being profoundly computationally intractable.
Requiring perfect knowledge of how information will be used is brittle. It has the major benefit of making the algorithm design problem tractable, which is why we do it.
An alternative approach is to exclude large subsets of queries from the universe of answerable queries without enumerating the queries that the system can answer. The goal is to qualitatively reduce the computational intractability of the universal case by pruning it without over-specifying the queries it can answer such as in the traditional indexing case. This is approximately what "learned indexing" attempts to do.
Requiring perfect knowledge of how information will be used is brittle. It has the major benefit of making the algorithm design problem tractable, which is why we do it.
An alternative approach is to exclude large subsets of queries from the universe of answerable queries without enumerating the queries that the system can answer. The goal is to qualitatively reduce the computational intractability of the universal case by pruning it without over-specifying the queries it can answer such as in the traditional indexing case. This is approximately what "learned indexing" attempts to do.