Reading "How the Mind Works" by Steven Pinker. The last line of this page 15 excerpt has stuck with me...
An intelligent being has to deduce the implications of what it knows, but only the relevant implications. Dennet points out that this requirement poses a deep problem not only for robot design but for epistemology, the analysis of how we know. The problem escaped the notice of generations of philosophers, who were left complacent by the illusory effortlessness of their own common sense. Only when artificial intelligence researchers tried to duplicate common sense in computers, the ultimate blank slate, did the conundrum, now called “the frame problem,” come to light. Yet somehow we all solve the frame problem whenever we use our common sense.
Imagine that we have somehow overcome these challenges and have a machine with sight, motor coordination, and common sense. Now we must figure out how the robot will put them to use. We have to give it motives.
What should a robot want? The classic answer is Isaac Asimov’s Fundamental Rules of Robotics, “the three rules that are built most deeply into a robot’s positronic brain.”
Asimov insightfully noticed that self-preservation, that universal biological imperative, does not automatically emerge in a complex system. It has to be programmed in (in this case, as the Third Law). After all, it is just as easy to build a robot that lets itself go to pot or eliminates a malfunction by committing suicide as it is to build a robot that always looks out for Number One. Perhaps easier; robot-makers sometimes watch in horror as their creations cheerfully shear off limbs or flatten themselves against walls, and a good proportion of the world’s most intelligent machines are kamikaze cruise missiles and smart bombs.