Here's a wonderful little post by James Gleick about the meaning of the word "information," according to the OED. A palace indeed. This reminds of of one of my favorite books, Jeremy Campbell's Grammatical Man: Information, Entropy, Language, and Life — probably the first book I read that suggested serious connections among my own work (the interpretation of texts), cognitive science, and computers. It was this book that told me who Claude Shannon is and why he matters so very, very much to the world we now inhabit.
Grammatical Man is almost thirty years old now and much of the science is clearly outdated, but it's still fascinating, and I wish someone brilliant would tell the same story again, in light of present knowledge. Maybe a job for Steven Johnson?