A degenerate (in its derogatory use) is a person who, from the perspective of the speaker, is "less than" or has "fallen from grace" — i.e., has degenerated from some pure or true form, into what they are now. In the US, the term has often been applied by the "rich, white, morally correct" folks to such unwanted groups as hippies, homosexuals, drunks and dope fiends.
The term was also frequently use by explorers, missionaries, and "scholars" from the so-called "Old world", to describe the so called primitive inhabitants of the Americas and Africa. The term stuck with such people long after they were part of the world's "modern society" when trying to make a case that some races were inferior to others. This is why most people imagine a toff voice in their head when they think of the word.
Renewed usage: neo-Nazi and neo-reaction
The term "Degeneracy" has come back into favor among the worst people the Internet has to offer. In context, it tends to be used to refer to people who are either not white or white but not racist (very much like the word "cuckservative"). 4chan's/8chan's /pol/ is particularly fond of using the word in a "ha ha only serious" kind of way.