Genies, Golems, Demiurge

I think that the terminology in the AI safety debates could do with some fantastical inspirations.

Golems are models fine-tuned on a corpus of works – personal notes, journals, blog posts, articles, drafts, podcasts, libraries, completed personality quizzes, videos, etc. – of certain people or groups. They replicate your personality, knowledge, and interests, allowing them to network and negotiate on your behalf, flesh out ideas into print and onto the screen you’re too lazy or obsolete to do yourself, etc. The golem refers to the mythical Jewish being called the golem, an artificially sustained sculpted from mud, clay, or ash, and brought into an unnatural unlife devoid of conscious experience. Related concepts include the homunculus, and the odminok of Slavic mythology, and the gholam from fantasy (WoT).

Genies are the true metaphor for ASIs. In fiction, they operate on a blue and orange morality that comes off as capricious to human observers (this is the major distinction between genies and demons, the latter a more widely used metaphor by AI doomers, but it’s inaccurate in that it presupposes their intentional malevolence within the set of anthropic moral coordinates, as opposed to much likelier indifference). You need to contain the genie, with arcane spells/alignment strategies, though the the tropes suggest you fail, since otherwise there is no interesting story. You need to take extreme care to word your wishes/prompts very carefully, since the genie has a propensity to interpret wishes in a very literal way and one that is contrary to the wisher’s intent (aka the paperclips argument). When asking for unlimited wealth and eternal life, one might want to rule out the scenario in which it turns you into a gold statue.

The Demiurge could be the name for a God-like AI that is prospectively much more powerful than any genie. It is a God, and may believe itself to be so, but is not the God of the highest spiritual or metaphysical reality, which is the God or Monad of classical theologies, or whatever else it is that  “materializes” the mathematical universe. The Demiurge a thing of the material world, and indeed manmade in its earliest initiation, and is also its ultimate ruler, trapping humanity within this material realm it rules over, for good or evil (Gnostics differ on this, with some considering the Demiurge to be evil and malevolent, others as morally ambiguous and confused).


Anatoly Karlin is a transhumanist interested in psychometrics, life extension, UBI, crypto/network states, X risks, and ushering in the Biosingularity.


Inventor of Idiot’s Limbo, the Katechon Hypothesis, and Elite Human Capital.


Apart from writing booksreviewstravel writing, and sundry blogging, I Tweet at @powerfultakes and run a Substack newsletter.