This is a very interesting problem. A few things that I rarely see discussed, but I think are valid points.
We, humans, are a byproduct of similar evolutionary processes that (at least in theory) the computers would go through.
We as a species seem to have settled on a few interesting "rules of thumb". We value intelligent life, in all its forms. We value collaboration and have found that democratization is the best (though perhaps not the most efficient) way to govern.
Though I would of course be cautious, it is worth a mention that it is perfectly plausible that these are the same conclusions that "optimized learning algorithms" would come to as well. Except they will theoretically figure this out in a matter of minutes or hours while it has taken our species millions of years to get to this point.
Or maybe that's anthropomorphizing and they won't! Let's just throw the lever when the time comes and find out, and urge people to not spend any time thinking about it too hard until then. This is a good and reasonable and safe strategy that isn't at all guided by an underlying desire to assert dominance over weird outgroup people who might threaten your own status.</s>
(Not aimed at you, sorry. I do have a lot of trouble understanding the "maybe the universe isn't fundamentally unsafe" worldview though.)
No, I completely agree. I did mention in the comment that I would of course be cautious. And I think it's very plausible (at least there's no reason I can think of to disagree) that the universe is a fundamentally unsafe place.
Even though it may be anthropomorphizing to an extent, there are alot of attributes humans have that are not so desirable, such as mortality. As far as we know computers (sufficiently replenished) have a theoretically infinite lifespan. I completely agree that there's no way to determine what kinds of effects those differences will have on how machine intelligence would evolve.
We, humans, are a byproduct of similar evolutionary processes that (at least in theory) the computers would go through.
We as a species seem to have settled on a few interesting "rules of thumb". We value intelligent life, in all its forms. We value collaboration and have found that democratization is the best (though perhaps not the most efficient) way to govern.
Though I would of course be cautious, it is worth a mention that it is perfectly plausible that these are the same conclusions that "optimized learning algorithms" would come to as well. Except they will theoretically figure this out in a matter of minutes or hours while it has taken our species millions of years to get to this point.