User:Sterile/ASKinfo

Shannon information is NOT correct

The definition of Shannon information is indeed Kolmogorov complexity and not Shannon information. It is blatantly incorrect, and there is a distinct difference between the two. Sterile 12:26, 8 May 2009 (UTC)

While I might have conflated the two ideas to some extent, I believe that you've gone too far the other way.

You changed "Shannon was concerned with measuring information for the purposes of compressing and transmitting data to "Kolmogorov was concerned with measuring information for the purposes of compressing and transmitting data" (my emphases), yet Shannon was concerned with transmitting data. And according to Wikipedia (okay, not a reliable source, but...) "Shannon's entropy represents an absolute limit on the best possible lossless compression of any communication...".

You changed "whereas the same-length string "The quick brown fox jumps over a lazy dog." can not be so readily compressed." to "whereas the compression of the same-length string "The quick brown fox jumps over a lazy dog" is not compressible if the algorithm is to reproduce letters one at a time.". You removed the full stop from the example sentence, thus making it not the same length, and I have no idea what "one at a time" has to do with anything. Your reference for this doesn't seem to discuss this, at least in anything like the same words.

You changed "However, although Shannon's research has proved to be very useful in many areas of data storage and communication, it does not take into account the intended meaning of the information." to "However, although Kolmogorov's research has proved to be very useful in many areas of data storage and communication, the meaning of the information depends on the algorithm and individual processes to compress and decompress it.". The original sentence was correct, as Shannon theories do not take the meaning into account. Also, the new version is completely ambiguous regarding the "meaning". It seems to be saying that the meaning of a sentence depends on how it is compressed and decompressed, which is nonsense, as the sentence has (or doesn't have) meaning before it is compressed. In any case, Kolmogorov's ideas are looking at the statistical complexity, and also ignore the meaning of the information.

You changed "A completely random sequence of letters conveys no information, but takes more space to store than a proper sentence that means something.", which was quite correct, to "A completely random sequence of letters cannot be compressed and takes more space to store if the algorithm is to reproduce the letters individually. A proper sentence that means something may be more compressible if and only if the linguistic algorithm used to compress the string takes into account the meaning of words." Again, what does "reproduce the letters individually" have to do with it? And it is wrong to claim that compression algorithms need to take into account the meaning. Compression algorithms take into account the repetitiveness of strings of characters (replacing all occurrences of "the" with a single byte, for instance), and have no understanding of the meaning at all.

Philip J. Rayment^discuss 15:14, 9 May 2009 (UTC)

• • Eh, OK--I'll give you that Shannon's information does tell you something about compressibility, but the concept of repeat 123's is misleading. That more accurately describes KC, and only tangentially describes Shannon information.
  • Meaning is still as yet undefined on this wiki. Meaning requires a context--linguistic, gene function, etc. It is not standardized word in the sense of information theory that I am aware of.
  • The KC is entirely algorithm-dependent. You cannot look at a string of letters, symbols, binary bits and determine the KC. You must understand this if you are to understand KC. I was patching up the lazy dog example to at least have an conceptual algorithm present, which was not in the original text. Hence, all the qualifiers. (The length of the string does not necessarily tell you how compressible it is. A longer string may be more compressible than a shorter string. In KC, the number of steps that your computer program takes to compress it is the KC. Let's put it this way: "123123123123123123123123123123123123123123" and "The quick brown fox jumps over a lazy dog." both increase the size of this wiki by 42 bytes. Why? Because the wiki uses up one byte per character; it doesn't have a "repeat" function available to its algorithm. If it did, it could the compress the numerical string more. But it could also have an instructions for "produce 'The quick brown fox jumps '", "produce 'over a lazy dog.'", "produce '123'". In which case the sentence would be 2 algorithmic steps and the string would be 14 algorithmic steps. There the number string is less compressible than the sentence. Those are all concepts based on KC.
  • (This is part of the reason why mutations or even loss of a sequence can be an increase of information or new information when looking at the sequence of a string, when defined in this sense. It depends on the algorithm. Mutation as the corruption of information is really a Shannon concept, as one particular location that was specified before now has more options and hence a increase information entropy, "less Shannon information," if you will.)
  • Words and their meaning require algorithms of linguistic interpretation that your brain uses. An obscure computer language might make no sense (no "meaning") to you but it would to the computer programmer ("significant meaning"). If the computer programmer dies and the computer gets smashed, is there still "meaning"? (You could still look at how much space it takes on the hard drive, the "information content" as it is, if it could be read.) That is one reason why your word analogies are senseless--they require a common language. They also use different algorithms than genetics goes.
  • The Shannon information entropy would likely be the same if they are stored as individual bytes because each character is equiprobable. Information entropy is not described in terms of algorithmic steps, but in terms of the probability of each character being present. "Lossless" is important. Shannon used his information entropy concept to look at the transmission of information because signals become less specified as a noisy channel is produced. (I am admittedly conflating Shannon information in terms of number of bits and information entropy here. The latter is important when transmitting a signal.)
  • As for which is "statistical" or not, Kolmogorov is looking at a mathematical complexity based on the number of steps in an algorithm. The only thing that might be called statistical is a count of the number of steps required by an algorithm. Shannon is based on a probability distribution and hence can be more properly called statistical.
  • The former version in my opinion lacked the precision necessary to make it a useful encyclopedia entry. I'm not saying my version is perfect or even 100% accurate, but it's better than what was present before.
  • And, one last point, after thinking about this some more, as to whether I've "over-de-conflated" the two types of information: The two types of information theory are very different from each other, and the attempt to use both at the same time leads to confusion. The ways they are related is that they are both used in information theory as ways to think quantitatively and qualitatively about information and that they both have been used for the same systems. But they are different concepts. Shannon information can be thought of as the relative uncommonness of a string of digitized information. Its compressibility deals with the minimum number of bits to produce string. KC is the list of steps to reproduce a sequence using algorithmic steps. Its compressibility deals with the minimum number of steps to reproduce a strings. These are quite different and can lead to dramatically different results. Replacing a letter in a sequence with a randomized letter will decrease the amount of Shannon information, but it could increase, decrease or keep the same the KC. Adding a "B" to "ABABABA" will increase the Shannon information if A and B are options for the added letter, but it could decrease the KC (1/repeat AB 3 times, 2/add A to 1/repeat AB 4 times). I would expect an encyclopedia to distinguish between the two, not mix them up. Sterile 21:04, 9 May 2009 (UTC)

Meaning is still as yet undefined on this wiki. "Meaning" has the meaning that it means something. Yes, that definition is self-referential, but the point is that you understood it because you know what meaning means. That's not to say that it can't ever be defined in a more rigorous scientific manner, but we don't need to define every word we use here. Until and unless it is defined differently, it has the meaning that it normally has.

It is not standardized word in the sense of information theory that I am aware of. However, "information theory" is a particular term with a particular meaning, which relates to the study of information in a statistical manner, a manner that precludes the information's meaning. Therefore it's a non-issue for information theory, which therefore doesn't need to define it.

The KC is entirely algorithm-dependent. You cannot look at a string of letters, symbols, binary bits and determine the KC. That may be true of KC (I don't know), but it's pointless to compress a string with KC if the algorithm for doing so is no smaller than the string. ...it could also have an instructions for "produce 'The quick brown fox jumps '", "produce 'over a lazy dog.'" Yes, but in so doing, the entire original string has been reproduced in the algorithm (albeit in two parts), so no real compression has taken place. That is why I mentioned that compression is only possible if there is repetition, and there is very little repetition in this string. You apparently agree that "random strings allow for little or no compression" (from the article), and this is because of the lack of repetition. Yet to use your argument, a random string could be split into two and reproduced in two steps also.

The length of the string does not necessarily tell you how compressible it is. A longer string may be more compressible than a shorter string. Quite correct, because a longer string is likely to have more repetition. Incidentally, I once zipped several large (c 1 MB) CAD files into separate .zip files, but the total size of all files was still too large to e-mail. I tried zipping the several .zip files into one .zip files and was surprised at how much further they were able to compress. Then I realised that it was because each individual CAD file had a large portion that wasn't really compressible, but the same part in each case, and putting them together allowed those identical incompressible parts to be compressed into one.

There the number string is less compressible than the sentence. Only because of the compression algorithm. Because the number string has more repetition, it should be more compressible, all else being equal.

This is part of the reason why mutations or even loss of a sequence can be an increase of information or new information when looking at the sequence of a string, when defined in this sense. It depends on the algorithm. That doesn't make sense.

Mutation as the corruption of information is really a Shannon concept... It's also a concept when considering the meaning, which Shannon and KC don't do.

If the computer programmer dies and the computer gets smashed, is there still "meaning"? Yes, although there is nobody left to understand the meaning.

That is one reason why your word analogies are senseless--they require a common language. How does that make them senseless?

As for which is "statistical" or not... By "statistical", I'm referring to methods that count (whether that be letters of algorithmic steps) and which ignore the meaning.

The former version in my opinion lacked the precision necessary to make it a useful encyclopedia entry. I'm not saying my version is perfect or even 100% accurate, but it's better than what was present before. I disagree, and you've not addressed a number of my points.

I would expect an encyclopedia to distinguish between the two, not mix them up. Fair enough, but at the same time you've ignored the meaning aspect of information.

Philip J. Rayment^discuss 03:22, 12 May 2009 (UTC)

As to your last, quoting your first, ""Meaning" has the meaning that it means something." Phil, I don't think you've done much to advance the understanding there. Really, "it's obvious to me" is not an explanation. As I dig through the book you quote-mined (Kupper) I realize more and more, sadly, that you actually have no idea what you are talking about. ħuman 05:40, 12 May 2009 (UTC)

I can't say that your comment did much to advance the understanding either. Did you understand my sentence that you quoted? If so, then it's not a case of "it's obvious to me", but it's also obvious to you. That is, you also understand the meaning of "meaning". Philip J. Rayment^discuss 09:37, 12 May 2009 (UTC)

I'm looking forward to responding to this when I am less busy. Sterile 13:25, 12 May 2009 (UTC)

Meaning. Meaning in italics. Meaning in quotes. You admit that your definition of meaning is "self-referential" meaning that we all understand what meaning, and that "we don't need to define every word we use here" and it has the "meaning that it normally has." And you say that information theory precludes our collective sense of the word meaning, and that we can't use the formalized system of information theory—with its mathematical and "statistical" rigor.

It's a well known rhetorical device to redefine terms to fit your needs. I never knew that it was a rhetorical device to hide behind a lack of a definition. The traditional concept of meaning is a language concept, an association between a word and a concept. It is the basis of your word analogies.

It is very convenient for you to leave the word meaning undefined. After all, it came rather late in our lengthy conversation on my talk page, after it became clear that "information" wasn't going to be enough. It is certainly convenient for you to keep it, because it places all the rhetorical power in your court. You get to control the definition, what is and isn't meaningful, and change it at will. You simply can just type a few sentences and say, no, that's not what Philip means.

And I'm not going to stand for it.

Let's start by dispelling the language analogy myths right off the bat. Apart from the dog example, your only arguments have been on the basis of palindromes, changes in letters in sentences, and other supposed word analogies to genetic and developmental changes. There are two contexts to these supposed changes to information: first, the change in a genetic sequence (generally a point mutation), and second, a change in the development of an organism after such a mutation. It's also pretty clear that this is where the only definition of meaning can really lie.

In terms of a genetic sequence, it's pretty clear that the language analogy is poor. Whether you are looking at the DNA bases or the proteins that are transcribed from them, the language analogy breaks down, and rapidly. There are 20 amino acids that can result from a codon and they all have a letter associated with them. If you were to take a series of those protein letters, any words that you got out of them would be pure coincidences, and certainly would never make it to an extended thought. And if you redefined a part of a protein sequence to make a word, later downstream you would get gibberish.

In reality, there are sequences of letters that get transcribed into protein structural features, such as alpha-helices and beta sheets. These structures are favored by thermodynamics and kinetics, not by any mysterious meaning, and certainly not by meaning that you imply. In reality a specific sequence is important because the information gets transcribed in the the structure of a protein, such that the protein can then catalyze a reaction. One possible definition of "meaning" in terms of a sequence is the ability to transcribe into a protein that catalyzes reaction. No word analogy necessary at all.

The other way that the sequences can be described as having meaning as in development, as with the developmental genes like the Hox genes. The Hox genes are highly conserved across organisms, and control the development of animals—like all of them—and they are expressed in order, allowing for development from one end of the organism to the order. You can take the Hox genes from one organism and put them in another, and still get a functioning organism. And there even is a correlation between the number of Hox genes and types of organisms:

“”It also becomes clear why we have up to thirteen Hox genes in each cluster rather than eight, as flies do. Vertebrates have post-anal tails, that is spines which go on well past their anuses. Insects do not. The extra Hox genes that mice and people have, which flies do not, are needed for programming the development of the lower back and tail.

—Matt Ridley, Genome, the Autobiography of Species in 23 Chapters Harper Collins: 2006 ed., p. 181

In fact, this is the type of decompression that Kolmogorov complexity describes: How the algorithms and processes of development are compressed into the genes of an organism, and the genes have a corresponding genetic sequence. Genes and their expression are the second type of meaning. Again, no word analogies necessary.

If you have a better, more clear sense of "meaning" that any creationist can provide I'd love to hear it. But if it's meaning that's not tied up in the ability to express genes, particularly developmental genes, and if it's meaning not in the ability to generate a protein sequence and structure, what good is it? And why bother with word analogies if they have no relevance to either situation?

It is also clear the Kolmogorov complexity and Shannon information tell us a lot about these types of meanings. Shannon information tells us about how a sequence changes and how unlikely it is, and KC can potentially tell us about how DNA sequences can be decompressed. What other types of changes in information or meaning or whatever you want to call it do we need to consider? If it's not how a sequence changes and it's not how that "meaningful" information is transcribed into a protein or morphology, what good is it? And what good is your supposed insight about the "virtually never adding meaningful information" if no one but the creationists know what it is?

What it comes down to is that information theorists and biologists have rigorous, well defined tools at their disposal. You have an assertion supported by word analogies about your undefined hunch about how the world works. The only types of evidence you have are word analogies that are largely irrelevant and an unclear dog example, certainly not one with rigorous tested data to support it. (Do you really think that any biological evidence supports leading to single genotypes? And what natural selection device are you using? Does it occur in antibiotic resistance, for which 99% of the bacteria are killed? How would you or I or anyone know when it does it occur? If not, then when? And how does it relate to the connection to mutations and the lack of new information?) And yet, you say others on this wiki "provide no evidence of [his] claim" and that his "sort of fact-free assertion doesn't count for much." And yet, you have an evidenceless, fact-free, definition-free, correlation-free and utterly unsupported hunch and assertion about an undefined concept that everyone is suppose to believe. And it's no wonder that they don't. Sterile 03:10, 13 May 2009 (UTC)

"Because the Bible tells me so..." Nice post Sterile, you frustrate me in that I am not actively involved any more in the pursuit of understanding language (philosophy of), that I am just some guy who tries to understand things. PS, you made two or three typos in your brilliang comment. I was tempted to fix them. I think we should copy most of your comment to the RW information article somehow? ħuman 03:55, 13 May 2009 (UTC)

(Human--Go ahead and fix them. This kind of just came out after looking back a Ridley's book and Philip's comments. I didn't proofread all that much. Sterile 12:42, 13 May 2009 (UTC))

Meaning. Meaning in italics. Meaning in quotes. You admit that your definition of meaning is "self-referential" meaning that we all understand what meaning, and that "we don't need to define every word we use here" and it has the "meaning that it normally has." ... It is very convenient for you to leave the word meaning undefined." And yet I didn't leave it undefined. Although I didn't quote, say, a dictinoary definition, I pointed out that you already know it. Do you know it? Did you understand my self-referential definition? You didn't say. If you understood it, then you obviously understand the meaning of the word. But are complaining that I haven't defined it? Why do I need to define it if you already know it? If you didn't understand, why not say so?

Actually, I now wonder if the problem is that you don't recognise (acknowledge) the existence of meaning. I came across this yesterday:

The third major component of naturalism is a theory about reality in which physical entities are all there is. God and angels are just imaginary fictions. The mind is really just the physical brain, free decisions are merely the results of prior events plus the laws of nature, and there is no teleology or purpose in the world — i.e., life is ultimately meaningless. —

Is it that you simply don't recognise the existence of purpose or meaning?

In reality, there are sequences of letters that get transcribed into protein structural features, such as alpha-helices and beta sheets. These structures are favored by thermodynamics and kinetics, not by any mysterious meaning, and certainly not by meaning that you imply. Really? That may be your assertion, but that is what I reject (and hence why I find English-language analogies useful). Why do such sequences get favoured? Because they use less energy to maintain? Then surely a vastly-shorter genome would be better? But then of course you would lose the instructions for hair, ears, butterfly wings, or whatever. That is, you would lose meaningful information.

In reality a specific sequence is important because the information gets transcribed in the the structure of a protein, such that the protein can then catalyze a reaction. One possible definition of "meaning" in terms of a sequence is the ability to transcribe into a protein that catalyzes reaction. No word analogy necessary at all. I never said that word analogies were necessary. No analogy is ever "necessary". Rather, analogies are useful to aid understanding. The lack of their necessity doesn't mean that they are inapplicable or invalid. But what you've just described is something meaningful; i.e. something that is there because it makes sense and does something useful, rather than being random.

I've not made this point so far (but never denied either), that Shannon and KC views of the genome can be useful. My point is not that they are inapplicable at all, but that they are inadequate, because they are only looking at one or a few aspects of the genetic information.

I'll give you another analogy, which you'll probably hate, even though it doesn't involve human language. Suppose the SETI program succeeds in detecting a radio signal from an alien race. We can use analysis of the signal, possibly even using Shannon or KC information theory, to determine that it really is an alien signal and not random noise or some such. However, having used information theory to determine this, we still have no idea what the signal actually says. That is, we are still oblivious to it's meaning. Yet do you really think that if we changed some random "letters" in the signal, and sent it back to the aliens, the message they receive from us would make sense? What if I said that repeating part of the signal (duplication) and changing some random letters in the duplicated part (mutation) constituted "new information"? Would the aliens agree? That is, would they agree that you've told them something new? Because that is the sort of argument that you and other here are making in claiming that mutations contitute new information. Yes, if you are looking at it from a Shannon or KC point of view, perhaps there is "new information". But if you are looking at what the message actually means, there's no new information at all. No matter what "well defined tools" information theorists have at their disposal.

As for your claims about my argument being a "hunch" and word analogies being the only types of evidence, etc., that is nonsense, as I have shown that it is more than that.

Does it occur in antibiotic resistance, for which 99% of the bacteria are killed? Sorry, does what occur? Natural selection? Yes, natural selection occurs, but not due to any new information.

And yet, you have an evidenceless, fact-free, definition-free, correlation-free and utterly unsupported hunch and assertion about an undefined concept... Completely incorrect, given that I've provided evidence, facts, definitions, and support.

From Human: "Because the Bible tells me so..." As I've said before, bibliosceptics accuse Christians of using this argument far more often than Christians actually do use it, and in this case is completely bogus because I didn't make any reference to the Bible in my post. If you have to make up straw-man arguments to make your point, you obviously have no argument.

...you made two or three typos in your brilliang (sic) comment. :-)

Apparently, "Meaning" is meaningless. What does this mean for Science? --Gulik 07:01, 14 May 2009 (UTC)

Philip, please don't add more confusion to the meaning of meaning by meaning that meaning means purpose. That is another meaning of the word meaning, but has no meaning in this context, as meaning of information doesn't mean meaning as in purpose. The word meaningless means without purpose, not without (information) meaning. Editor at CP 09:38, 14 May 2009 (UTC)

The above comment does not constitute a gain in information.--CPalmer 10:23, 14 May 2009 (UTC)

Obviously, I'm still displeased with the definition situation. It's a little hard to take seriously a statement that you've provided a definition after your last post. It's even harder to believe you are coming to the table to have a fair academic discussion when you take all the rhetorical advantage by controlling the central term of discussion, by not defining it. You claim that we both just know what meaning is, that it doesn't require a definition. Yet, I don't know if a biological example that I'm citing is relevant, or even what I'm looking for. And I've provided definitions, which you feel free to convolute.

At the least, I need to know "what has meaning" or "more or less meaning" so at least I can try to have a conversation with you and to help develop this wiki article. Since it is so obvious, the following experiment should be easy for you to respond to. Suppose we have a population of organisms. I tend to think of them as one hundred cute, fuzzy rabbits, but it really could be any organism. We take the initial rabbits and we sequence all of their genes. We also measure whatever other data we would like: ear length, eye color, resistance to disease, hair length, etc. Then we let them breed for quite a few generations (maybe 100 or 1000). After breeding, we collect the same data, genetic and other. Then we give you the data, but we don't tell you which one is the ancestor population and which is the descendent population, and instead we label them as population "X" and "Y." How will you and your creationist meaning-knowers tell which population has more meaningful information associated with it?

It can't be related to the passage of time solely, because that would mean a tautological definition and that wouldn't work (and we didn't tell you that anyway!). I can't see how the genetic sequence would help: A mutating to G is as likely as G mutating to A. Natural selection pressures that might favor a characteristic probably would be erased in a similar reverse experiment in which the pressures disappeared. Perhaps we have to wait more generations until an irreducibly complex feature disappears, although you are not a ID'er, so I'm not sure if it's relevant. I can't figure out what characteristic you would look at, but since you know and it's obvious, you should be able to tell us. Sterile 15:48, 15 May 2009 (UTC)