Talk:Shannon-Hartley theorem
From Academic Kids
I don't think this is Shannon's theorem; it is simply his definition of informational entropy (= expected amount of information).
Shannon's theorem is a formula for the maximal rate at which you can send information down a pipe if you know the bandwidth and the signal-to-noise ratio. AxelBoldt
- Fair enough. I was just replacing garbage, and took planetmath.org's word for it. One undergraduate course aside, I know no real communication theory -- User:GWO
I would like to see a reference on this page to the means used to obtain the maximum rate: Shannon's reasoning was based on assuming an equivalence between signal space and N-dimensional euclidean space; noise power determined the size of the spheres, the number of dimensions being the number of possible signal tuples, and the result being how to pack the maximum number of spheres into that space, for which the limit was well known. Bukowski
Wikipedia just needs more general information about the relationship between digital and analog bandwidth. What other articles/subjects are related to this? - Omegatron 16:30, Apr 24, 2005 (UTC)
| Contents |
Modem comparison needs peer review
Original: The last paragraph (especially "V.90 claims a rate of 56 kbit/s, apparently in excess of the Shannon capacity") is wrong. The V.90 data rate does not account for compression.
Edit by Mark Rejhon: This modem information has now been clarified from Mark Rejhon's edit about a month ago. This needs some peer review. Also, 56 kbit/s is achieved without compression, or rather, 53 kbit/s due to FCC regulations on signal level. But this is done, without compression! It's rather, done, via avoiding a digital-to-analog conversion step, at the telco end. This information is widely available on the Internet, but needs to be confirmed with accurate sources.
- From my study of the Shannon channel capacity theorem, compression doesn't matter. Information is information, compressed or not. To the best of my knowledge, Shannon made no assumptions about the coding of information. This is his real genius. In general, his analysis is based on "nats", not bits (base e as opposed to base 2). Binary is a convenient coding, so most people talk about bits. As far as I know, Telcos typically sample at 8kHz with 8 bit samples. This means a Nyquist limit of 4 kHz an ideal quantization noise of about 48 dB (maybe even 54 dB if you use half of 256). If you lived 10 feet away from your central office and there was no other interference, you might get that kind of S/N. Reality is that you are probably thousands of feet from the CO and there are hundreds of other lines next to yours adding noise. I am told that 30 dB S/N is possible with BW of 3.5 kHz. This works out to about 35 kb/s. If you assume 48 dB, the channel capacity is 55.8 kb/s. Close to that mystical figure of 56 kb/s, I would think. "Principles of Communication Systems" by Taub and Shilling is a great book. Madhu
- If you have like, encyclopedia.txt, a 100 MB file, and you compress it down to 1 MB, send it in one second, and them decompress it later, you could say that you had sent 100 MB/sec. I think that's all they meant. They are talking about the actual transmission speed (in this case 1 MB/sec). Apparently modems can also do on-the-fly compression, which would give effective rates greater than the actual channel capacity for easily compressed data. - Omegatron 16:38, Feb 17, 2005 (UTC)
- In my understanding, anyway, the theorem only applies to the actual data being sent, regardless of whether it is compressed data, binary, or all zeroes. In other words, compression of the data does not count towards getting closer to the theoretical max. - Omegatron 17:02, Feb 17, 2005 (UTC)
- I think we're saying the same thing. The important detail is the terminology. Shannon talks about information rate as opposed to bit rate (he doesn't really talk about bits). In your above example, the information rate is 1 MB/sec. If I have a 100 MB file with truly random data that cannot be compressed (try compressing a compressed file sometime ;-), then the bit rate drops to 1 MB/sec instead of 100 MB/sec. Another way to say this is that a highly compressible file contains less information than an incompressible file. Madhu
Several versions
There used to be five different versions of this article. Now there are three. Over the last while, several people have merged these articles (myself included, for one of these merges, but it appears I am not the only one). However, there are still two more articles that still needs to be merged into this article.
These are now the same article:
- Shannon-Hartley theorem (main article)
- Shannon limit (now merged into above)
- Shannon's theorem (now merged into above)
- Shannon-Hartley law (now merged. seems to be a different but closely related concept.)
- Shannon's law (now a disambig page)
We need to clarify the differences or merge the similarities between these articles. From what I understand, they are all the same thing, and should be merged and redirected to Shannon-Hartley theorem. Also clarify the relationship to Nyquist-Shannon sampling theorem. (One deals with digital data and one deals with sampled analog data, correct?). Then clarify in bandwidth the relationship between regular analog bandwidth, sampled data bandwidth, maximum digital bandwidth, etc. - Omegatron 20:21, Aug 9, 2004 (UTC)
- There are two different equations, and two different concepts, but I feel they are related closely enough and have similar enough names that they should all be combined into one article, and the differences and similarities explained within. This way people searching for one will not get confused, not realizing that they are looking at the wrong article. - Omegatron 20:02, Sep 18, 2004 (UTC)
Difference between laws
According to this site http://www.cs.nmsu.edu/~jcook/Classes/DE-CS484/Physical-1.html the "law" section is actually part of the nyquist theorem.
According to this site http://www.fas.org/man/dod-101/navy/docs/es310/DigiComs/digicoms.htm the laws are:
R = W Log2 M
and
C = R Log (1 + S/N)
where:
R = the rate at which data can be transferred, given in bits per second (also known as the baud rate)
W = the minimum bandwidth required to create this pulse
C = capacity in bits per second (bps)
S/N = signal-to-noise ratio (depends of modulation type and noise)
I'm confused. - Omegatron
- This "cheat sheet" describes it better:
- The capacity related to signal levels is only for noiseless signals. The other formula is a better one and takes noise into account. It also says that the noiseless one was derived by nyquist, so i think the one labeled "shannon-hartley law" is actually the shannon-nyquist formula. - Omegatron
I think you've combined the topics correctly except there isn't much connection (at least in my mind) between Shannon's Law and the Nyquist sampling rule. On an unrelated note, I found the D = 2B log2M noiseless formula to be confusing, as it's not part of the theorem it's just part of the thinking leading up to the theorem. So I've replaced it with a written version of the same concept which uses a "thought experiment" model instead of a formula to describe how to transmit infinite information over a fixed-bandwidth link (and how noise makes that impossible in practice). technopilgrim 20:36, 19 Sep 2004 (UTC)
- Alright. the first equation was labelled as shannon's law in the article of the same name. i was just merging. they seem to be seen together a lot in articles online though... i'm skeptical that it should be removed. maybe added to the nyquist-shannon article instead? - Omegatron
My problem with R = 2*B*log_2(M) is it seems to be neither fish nor fowl. In we were giving the full derivation of the S-H theorem this is an important and non-trivial milestone on the way to the full proof and we would definitely want to include it (as do professors when giving an extended talks on the topic). But we are trying to write a concise encyclopedia article and we can't assume we are addressing an audience of engineering students. Where does the 2*B factor come from? Shouldn't it be simply B? These are advanced questions in information theory (which is why radio communication was decades old before anyone understood this). Not to say that a formula can't be a great way of showing things in some situations, but here it is perplexing (part of the problem is that the explanation accompanying the 2*B*log(M) formula is not quite correct). technopilgrim 19:27, 20 Sep 2004 (UTC)
- I like your description. It is clearer than the formula. Thanks. - Omegatron 18:22, Sep 21, 2004 (UTC)
"on each cycle" should be changed to each clock or each transmission or each pulse or something - Omegatron 18:30, Sep 21, 2004 (UTC)
Steganography
Example two concludes: "This shows that it is possible to transmit using signals which are actually much weaker than the background noise level, as in spread-spectrum communications."
Would it be fair to say that that is the principle behind steganography? --Elijah 23:48, 2005 Jan 3 (UTC)
- No, at least, not for cryptographic steganography. I should say first that steganography is a much slipperier subject than cryptography - it's harder to specify and analyze. But a spread-spectrum signal need not be any harder to detect than a narrow-spectrum. A signal that is transmitted at a very low bitrate may be difficult to detect, but even this is not necessarily true. In steganography, the message is normally hidden inside another message. The study of analog, technical ways to transmit signals secretly is not really within the purview of steganography - it includes tricks like line-of-sight infrared lasers, transmissions as short, high data rate bursts, and other esoteric tricks. --Andrew 05:04, Jan 4, 2005 (UTC)
