Informational advantage lies at the center of the defense and intelligence communities strategic objectives. Acquiring, structuring, storing and analyzing such things as government data, communications data, and census data are all subfunctions of this overall goal - seeing what others don’t.
Although the global security landscape is always evolving, one constant is the need to understand the plans and intentions of an entity’s adversaries—something that a new generation of analytical tools specialize in determining.
The key is to manage comprehensiveness and responsiveness.
In the pursuit of informational advantage, federal agencies have, and continue to acquire more data than they can possibly analyze. Even for the subsets of data that federal agencies evaluate, the size of the those data set often dictates the analytical approach.
Adopting technologies where super-massive statistics datasets can be analyzed in real-time enables agencies to adopt an approach with minimal tradeoffs, enabling the largest class of possible insight.
Most approaches to these problems are grounded in traditional CPU-era thinking. Those approaches no longer scale effectively - the size of the public datasets and the sophistication of the visualization having surpassed their capabilities to deliver timely insight. Further, new generations of technology can be deployed in edge-computing scenarios such as drones, mobile platforms and other forward deployed assets.
New classes of hardware and software are changing the scope of what is possible, including the gathering and analyzing of communications data. Now, social media or other communications channels that run to the billions of rows can be queried in real time. Patterns of Life can be rapidly assembled and managed, even when running to the billions of events. Social channels can be monitored - on a global basis, with geographic overlays, in real time, not on delay.
These new capabilities will define the chessboard for the foreseeable future as HUMINT and technology continue to evolve together.