During summer, I had time to read several books. One of them was “A Mind at Play”. A biography about Claude Shannon, which was great to read.

This book tells the story of a fascinating scientist. His main research topic was to understand what information is in a mathematical way, which led him to become the father of information theory. This immediately found application in telecommunications and data transfer.

Another fascinating aspect was that he also asked how much information contains natural languages. Long story short, natural languages contain a significant amount of redundancy. This is easy to see, as characters and words do not appear in random order, and you can actually guess which word will come home. [And you will also notice if a word does not fit into a sentence.]

A funny footnote is the story of the Müllabfuhrwortmaschine. This is a hypothetical machine that knows the probability of words in sentences and can synthesize readable text. Shannon wrote down this idea in the 1930s, at a time when computers were still mechanical.

For me, Müllabfuhrwortmaschine sounds like Shannon already 100 years ago accurately predicted the concepts that are nowadays known as LLMs and AI slop.