[Iain Banks' Culture]
Someone must initially create the AI. Those are the programmers. Do you expect they will all agree on how these beings will operate? We already have a disagreement. How can I know they are "conscious" or sentient, having subjective states of mind and qualia? Are they supposed to keep all the humans alive forever, who exist when everyone switches over to the matrix, or do they make new humans? Are we supposed to "upload" our mind into a machine?