"Nobody seems sure why so much data has the Benford distribution. " : Have a look at http://terrytao.wordpress.com/2009/07/03/benfords-law-zipfs-... (cited in the previous discussion on HN) for an excellent exposition of the theory, and an explanation of the 'power' of the Benford Law.
That Terry Tao article is excellent, as usual (well, at least for math dorks - what's unusual about this one is that you don't need a very high level of dorkery to grok it 100%) and the "executive summary" reason that Benford's law applies so broadly is easy to find:
"More generally, it is not hard to show that if X obeys the continuous Benford’s law, and one multiplies X by some positive multiplier Y which is independent of the first digit of X (and, a fortiori, is independent of the fractional part of log_10(X)), one obtains another quantity X' = XY which also obeys the continuous Benford’s law."
In other words, multiplicative combinations of (independent) quantities will inherit Benford's law as long as any one of the quantities obeys it on its own, so it just takes one sub-factor that grows exponentially in order for an entire distribution to follow the law.
That line especially bothered me, it makes it seem like it's some divine or magical rule. If that were the case, I would hate for it to be used as if it were infallible with no actual understanding.
I'm impressed that Madoff was smart enough to mimic Benford distribution in his monthly returns (that's what the article led me to infer).
I actually made an r/math post about Benford's Law that I thought was pretty interesting, although it didn't get much attention or many comments (admittedly bad signs). Any peer review (or su-peer-ior review) is appreciated.
Here's a snippet:
"...generation of the geometric layout for Benford first digits is a similar process to creating a Sierpinski Triangle by drawing a triangle, and randomly plotting midpoint dots between vertices and sides recursively. In the Benford case the plotted dots are instead individual numbers, which share a geometric connection (e.g. two galaxies colliding will on average be a doubling, so galaxy size should possess Benford distribution)."
I don't think the issue at hand is that fraud detections are unsophisticated. The issue at hand is that the fraud detections are in cahoot with the fraud themselves. For example: Enron, Worldcom and Accenture (past) Greece, Italy, Spain and Goldman Sachs, Fitch, Moody, S&P (present) Chinese local banks, Chinese companies and SEC (future)
The problem with hackers is that they always try to find mathematical solutions to moral problems. What the world needs right now is a moral solution. A beheading for those that have fouled.
Unfortunately, the hackers, the workers - ones that actually have the power to demand changes - are willingly allowing themselves to be exploited by those that lie, and not demanding any moral judgements. They're not interested in changing the politics, they claim. They just want to keep producing interesting things, they exclaim. They don't realize they're the ones keeping the lie going. They're the ones propping up this decrepit shell of a society.
Until one day the shell collapses on their kids, anyways.
> Unfortunately, the hackers, the workers - ones that actually have the power to demand changes - are willingly allowing themselves to be exploited by those that lie, and not demanding any moral judgements.
(Emphasis added)
I think this point of yours is frankly quite wrong. The productive part of society is not the part of society with political power. Gaining political power is in and of itself a non-productive activity, and accomplishing it takes so much time that none is left over for productive activities. So the non-productive part of society naturally has power over the productive part of society, just as teenagers who focus on being popular end up being more popular than teenagers who focus on programming computers and learning stuff.
Spontaneously? No, someone is going to have to organize all of them and get all of them on the same page. In other words, someone is going to have to join the non-productive political class to organize all the programmers. Then it's the organizer who has the power, and then you have a central point that's vulnerable to corruption, which is exactly what has happened with labor unions in the past.
That's definitely not a given. Revolutions are a time of instability and the outcome can be worse or better. It mostly depends on what precipitated the revolution in the first place, how educated the people in the country where the revolution takes place are and how involved they plan on being once the violence dies down. A revolution is a time of transition and instability without any guarantees towards the outcome.
How about women's suffrage? I understand that there were many factors playing into the move toward equality of the sexes, such as wars that required society to use women's labour, but by-and-large it was an incredibly large shift accomplished through moral suasion rather than violence. And although it seemed very gradual - a couple of generations, and some would argue still not done - it was pretty rapid as societal shifts go.
> What the world needs right now is a moral solution. A beheading for those that have fouled.
I understand you're using a metaphor here - or at least, I hope you are - but you ought to choose your words more carefully. The literal interpretation of what you're advocating led to mass genocides, slavery, destroyed nations and cultures, and many places haven't recovered from it. Yeah, there's crooks in finance. Khmer Rouge, pogroms, "reeducation" slavery camps, and the like aren't the answer.
If you were just trying use a colorful metaphor, you can just use "prosecute" instead of behead without losing your impact.
There was a story a little while ago, about a corrupt official in China who allowed baby milk to be polluted by taking bribes to look the other way during safety inspections. The Chinese executed him.
I'm confused; which of those ills (which one could compare to the corruption that this is about) do you not think had a "moral campaign" (similar to what the grandparent suggests) fighting against it?
I'm not saying it's the best framing of the situation, but going from "we should fight this because it is wrong" to "slavery, genocide, and reeducation campus" is a massive jump.
Have a look at 'the French revolution' for one instance in time where it worked reasonably well. The French royalty and its associated entourage had become such a drain on the country that the French decided to get rid of them in the most drastic fashion possible.
Yes it was gruesome and bloody. But it did move France forward in time ahead of some other European countries that even today have not managed to rid themselves of their God anointed overlords.
Of course there are plenty of counter examples but it doesn't always have to be bad.
As I wrote elsewhere in this thread, the period of a revolution itself is extremely messy and it could go either way. But I think that it is impossible to argue that France was eventually worse off because of the revolution.
