Source Reliability, Grading, and Analysis
Grading source and data reliability is critical to the CTI Collection stage. From it, it’s possible to control the history of trustworthiness of the different sources used and estimate the veracity level of the information in the intelligence report. Grading and reliability metrics also help protect sources. Intelligence services often isolate sources from the CTI team personnel who don’t directly interact with them, avoiding source compromises. This part of the team needs to know how much they can trust such sources and the information they provide. Grading sources provide this metric while avoiding unnecessary exposure.
For the CREST CPTIA exam, the two primary standards for source reliability and grading are the UK’s National Intelligence Model (aka Police 5x5x5 system) and the North Atlantic Treaty Organization (NATO) Admiralty Code.
Before going in-depth into those codes, it’s important to note that their role extends beyond the intelligence life cycle collection phase. They also can, by themselves (like the Police 5x5x5) or in association with other codes (NATO Admiralty Code plus the Traffic Light Protocol [TLP]), help control the access to such products in the Dissemination phase while signaling the quality of intelligence and their sources on the report.
Police 5x5x5 has three main metrics: source evaluation, intelligence evaluation, and a handling code. As the name suggests, the first two metrics aim to estimate the confidence of each source and the information that such a source provides. Source evaluation is letter-graded (A to E), while intelligence evaluation is number-graded (1 to 5). The third metric guides the dissemination practice considering the intelligence product sensitivity. While useful in UK law enforcement, this handling code isn’t easily generalizable to other situations.
The Admiralty Code is more straightforward than the Police 5x5x5 since it has only two parameters: information credibility and source reliability. It’s used by NATO members and the intelligence agencies that compose the Five Eyes (Canada’s Communications Security Establishment [CSE], UK’s Government Communications Headquarters [GCHQ], New Zealand’s Government Communications Security Bureau [GCSB], and USA’s National Security Agency [NSA]).
Source Analysis
While both standards are essential for source grading operationalization, the CTI analyst must consider the sources implemented critically and in-depth. This means understanding the differences in reliability and biases inherent to each source type. It’s a critical thinking process and a core skill for the CTI analyst during the collection phase and even more so during the analysis phase (which will be discussed in another post).
This analysis has two dimensions: whether the source is primary or secondary and the continuum between objective and subjective sources.
A source is considered primary if it has a direct contact and a previous relationship with the CTI team. On the other hand, a secondary source transmits its information to a primary source, which then forwards it to the team. Using secondary sources is a widespread practice and isn’t inherently wrong. However, the analyst must consider the “Chinese whispers” effect, where the information reliability degrades as it passes through sources until it reaches the team.
There’s an important observation regarding the expression “validated by secondary sources” and the grading standards. This phrase doesn’t refer to secondary sources as defined above but to other primary resources at the CTI team’s disposal. One could assume that “secondary” is actually used as a reference to other sources aside from the one being categorized (i.e., the “primary”).
Objective sources provide direct access to technical data, such as domain information, reverse engineering, and DNS records. They may be right or wrong, given they provide data without human intermediaries.
Get Rodolfo Santos Flaborea’s stories in your inbox
Join Medium for free to get updates from this writer.
Subjective sources consist of humans (HUMINT) reporting on a specific topic, which can be technical data or, at the continuum extreme, CTI products. Since they rely on human analysis and production, the CTI team collecting data from it must control for bias, associated subject matter expertise, lower accuracy levels, report quality, and the history of relation with the source, if any exists.
An essential aspect of HUMINT analysis is the history of the relationship between sources and the intelligence team. In this sense, the Admiralty Code categorizes source quality following its performance across different intelligence cycles and reports. In CTI practice, though, the analyst chooses sources in an exceptionally dynamic and immediate context of cybersecurity. Nevertheless, the CTI team must be as critical of its sources as possible, ensuring a more solid and realistic judgment of the data obtained and processed.
Such careful posture is especially relevant when dealing with the most subjective information, e.g., agencies’ CTI reports and more qualitative works, like direct interviews with hackers or even interacting with them through social media or Internet Relay Chats (IRCs). Interview techniques, writing, subject postures, and general methodology must always be viewed critically and carefully.
This critical process puts bias (psychological or sociopolitical) in the spotlight, making it necessary to distinguish the bias’s assumptions and the rest of the logical elements, namely facts, premises, inferences, and assessments. It’s significantly valuable to dissect subjective works following those logical concepts, as it is an effective control measure against falling victim to the source’s bias or even to deliberate misdirection.
Deliberate Misdirection
Hoaxes, disinformation, and propaganda are all active measures or PSYOPS techniques aimed at confusing an adversary (e.g., the intelligence team, as viewed by threat actors). More broadly, those disciplines may have three different modes of operations:
- The source is explicitly attributed, and the information is valid (white).
- The source is partially attributed, and the truth is mixed with false information (grey).
- The source is deliberately misattributed, and the information is purposely misleading (black).
Critical source analysis, as discussed above, while complex, is an ideal solution against them. Additionally, the CTI team can complement this effort by relating the assumed intent to mislead and the truthfulness of information, illustrated in the graph below, derived from a Facebook video on combatting misinformation in social media.
It’s essential to note the differences between bias and deliberate misdirection. The first is often unintentional, while the latter, as the name implies, is purposeful. Regardless, critical source analysis is a fundamental countermeasure.
References
Irwin, Daniel, and David Mandel. “Standards for evaluating source reliability and information credibility in intelligence production.” Irwin, D. & Mandel, DR (accepted conditionally). Standards for evaluating source reliability and information credibility in intelligence production. In DR Mandel (Ed.), Assessment and Communication of Uncertainty in Intelligence to Support Decision Making: Final Report of Research Task Group S (2019).
