This is a huge problem for the intelligence services, and is an excuse frequently made for why the CIA failed to stop 9/11 and MI5 7/7. With the sheer amount of data logged and available to the services, how can one reliably wade through such huge amounts of data?
Being perfectly honest, I wouldn't be surprised if a Google Translate-style statistics package was in use right now. You, sir, fit the profile GeekWithGuitarAndCat72.
In recent years Oracle, IBM, Microsoft and SAP between them have spent more than $15 billion on buying software firms specialising in data management and analytics.
Why not do both? The major disadvantage with the seed fund approach is that you have no idea what you'll get out of it or how long it will take. If you but SPSS, it might cost you more, but you know exactly what you're getting and when you'll be getting it, which is a huge advantage if you feel you're in a hurry.
The one thing shows like 24 get perfectly right is that intelligence and warfare is now much more about computer scientists analyzing data than people on the ground.
After the money made processing the oil there will be plenty more to be made with the distribution networks and gas stations themselves. While processing data is an interesting problem for geeks, the more mainstream problem will be visualizing data. I don't go down to the refinery and pull my car up to the spigot. I buy my gasoline at a pump where I can also pick up the milk and get a cup of coffee at the same time.
It is very true. Intelligence Agencies are always wondering how to view the data better. I know for one example, we have 150 TB's of video data coming in every day from Iraq and Afghanistan and we are trying to find ways to view it.
One number my boss says a lot is the CIA wishes they had 14,300 Analysts to view through all this information at once. These of course would be 14,300 people looking at video 24/7... Its a huge problem that doesn't have a solution at all yet.
Being perfectly honest, I wouldn't be surprised if a Google Translate-style statistics package was in use right now. You, sir, fit the profile GeekWithGuitarAndCat72.