There's a pithy slogan that sums it up nicely: information is a correlation between the state of two systems.
For example: if you measure a voltage using a volt-meter then, if the volt-meter is working properly, there will be a correlation between the readout on the meter and the actual voltage, and so the meter contains information about the voltage.
Note that information is symmetric: the voltage in this case also contains information about the meter.
Your comment and the one above it are awesome summaries of some of Shannon's concepts. Re:your statement on information being symmetrical, how does zero knowledge encryption, Homomorphic encryption and the like fit into this when there is asymmetrical knowledge and validation without full access to the information?
The best way to approach that question is to start with the one-time pad (OTP). The OTP has a random key K which is the same size as the message M. The ciphertext C is M XOR K.
So... if information resides in correlations, where does the information reside in this system? Because K is random, it has zero correlation to M (that's what "random" means). It also has zero correlation to C, though this is not so obvious. (Proof is left as an exercise.) In fact it turns out that C and K are both completely random, and so it's not even possible to tell by looking at C and K which is which. They are interchangeable.
So we have the following situation:
1. Information about M obviously has to reside somewhere in the composite system comprising C and K because we can completely recover M simply by computing C XOR K.
2. The information is not in C and it is not in K.
So where is it?
[Spoiler alert. It's worth thinking about this for a while to see if you can figure it out on your own. Hint: think about the case where C, M and K are all just a single bit, because the multi-bit case is just a straightforward extension of the single-bit case.]
Answer: the information about M is contained in the correlations (or anti-correlations) between C and K. In the one-bit case, if C and K correspond (i.e. are both 0 or are both 1) then M is 1. If they don't correspond, then M is 0.
The more complicated cryptographic cases are very similar, but of course, more complicated. What cryptography is really about in some sense is trying to get the same results but with a smaller key, i.e. it's about trying to take a fixed-size K that is (much!) smaller than M and "distribute" the entropy in such a way that the information is hidden in correspondences between bits in such a way that an attacker can't figure out where those correspondences are.
This is hands down the best Hacker news interaction I've had! I took a screenshot of your explanation and did not scroll past the spoiler. Will give this some thought before proceeding... Thanks!
For example: if you measure a voltage using a volt-meter then, if the volt-meter is working properly, there will be a correlation between the readout on the meter and the actual voltage, and so the meter contains information about the voltage.
Note that information is symmetric: the voltage in this case also contains information about the meter.