To clarify, the use of gloves depends a lot on the item. Books are made to be handled by people and are usually resilient to contact with clean human hands. However there are other museum pieces that are far more sensitive to oil which still merit the use of gloves. For example, here is a video [0] of Adam Savage discussing a piece from the Met Museum which is handled using gloves because they want to preserve it.

[0] https://youtu.be/u_-oUvv28dE