Monday, March 4, 2013

Investigating the Efficacy of Terror Network Visualizations - SDM Pulse, Spring 2013


By Christopher W. Berardi, SDM ’11

Christopher W. Berardi is an active duty officer in the U.S. Air Force. He currently works as a program manager for multi-million dollar intelligence, surveillance, and reconnaissance weapon systems. Berardi holds an MIT M.S. in engineering and management, earned through SDM in 2013 and a B.S. from the United States Air Force Academy. 


An urgent challenge: Military intelligence analysts are increasingly tasked to sift through enormous volumes of data to identify the proverbial intelligence “needle in a haystack.” One specific domain exemplifying this new intelligence paradigm is network analysis of terrorist organizations. This area of intelligence analysis uses mostly commercially available software applications to leverage the powers of social network theory against large terrorism data sets.

An additional challenge is the fast paced development cycle for new sensors that are capable
of collecting data at unmanageable rates. Therefore, analysts are in dire need of new analytical techniques that give them the ability to effectively and efficiently transform the collected data into intelligible information and subsequently, intelligence.

Background:
Intelligence is only of value when it is available and contributes to, or shapes, a decision-making process by, “providing reasoned insight into future conditions or situations” (Joint Chiefs of Staff, 2012). However, this does not hold true for raw data. Therefore, the burden is on the intelligence analyst to transform raw data into intelligence. This transformative process begins with the collection of data from sensors.

The first step is to process the raw data into a form intelligible by an analyst. Depending on the type of raw data, this step is either automated as in the production of an image from a camera, or requires an analyst, in limited cases, to transform the raw data into information such as language translation. In the context of social network analysis, this stage typically involves transforming the tabular raw data into a visualization, or series of visualizations. This specific transformative process (data=information) is also known within the intelligence community as processing and exploitation1.

During the processing and exploitation phase, as shown in Figure 1, an analyst most commonly transforms the data into a node-link visualization. However, little to no emphasis is given to creating alternating modes of visualization that could result in a more effective transformation of data to information. Furthermore, there is little existing research into the effectiveness of one form of visualization over another in the domain of intelligence. 

After data is transformed into information, the subsequent information can be integrated and analyzed to produce intelligence. Once information is evaluated, it is ready for analysis. During analysis, assessments2 are made by comparing already integrated and evaluated information; these assessments are combined and used to discern patterns or links. Finally, the analysis and production process concludes with interpretation, which is a largely inductive reasoning process in which available information is evaluated.

From this sequence of integration, evaluation, analysis, and interpretation, intelligence is finally produced. Although, this is a generic process which applies to all forms of intelligence, within the context of social network analysis, analysis would be conducted by evaluating multiple visualizations of social networks and interpreting the information resident in each of those visualizations to create a prediction about the terror network, or networks, being analyzed (Figure 1).
Figure 2 - Node-link visualization
Research goal: Investigate various visualization methodologies for terror network analysis.

Research scope: Comparison of two visualization methods:
  • Node-link visualization (Figure 2) serves as control, as it is the most ubiquitous method of terror network visualization used within the intelligence domain today (Freeman, 2000; Wasserman & Faust, 1994)
  • Matrix network visualization (Figure 3), a promising method of social network visualization studied commonly within the academic community (Ghoniem, Fekete, & Castagliola, 2004; Henry & Fekete, 2006)
Figure 3 - Matrix network visualization
The experiment: To test the visualizations’ effectiveness, an experiment was conducted in which participants exploited matrix and node-link visualizations constructed from a surrogate terror data set (Zachary, 1977).

The 60 participants were all Air Force airmen who hold the Air Force specialty code of intelligence analyst. Each participant was given one of the forms of visualization and asked to accomplish two tasks:
  1. Identify leaders within the network, and
  2. Identify clusters or subgroups within the network.
These two tasks were chosen because of their reoccurring importance highlighted in both a literature review and a hybrid cognitive task analysis conducted prior to undergoing this research.

Discussion:

  • The node-link visualization resulted in statistically significantly better performance in all studied scenarios where the objective was identifying leaders.
  • Although node-link also returned a better performance than the matrix for identifying clusters, there was not a statistically significant difference.
  • In all cases, there was not enough difference between the times produced by the node-link and matrix to determine if either offers a statistically significant decrease in the time it takes to complete tasks using either visualization.
Conclusion:
At this time, the matrix should not be universally integrated into the current methodologies used by analysts to exploit terror network visualizations until more research is conducted into the respective strengths and weaknesses within the intelligence domain.

However, analysts should be independently encouraged to explore and adapt new methods of visualization into their current practices and identify new or improved versions of the visualizations identified within this thesis for future testing.

For Bibliography, please see sdm.mit.edu.


1: Defined as, the process by which raw data is transformed into information that can be readily disseminated, used, and transmitted by an analyst.
2: Assessment is defined as, a prediction of the future state of an organization, individual, or adversary.

No comments:

Post a Comment