What the Heck is Threat Intelligence?

Josh Campbell|July 10, 2020

Introduction

Perhaps one of the more controversial subjects in the field of cyber security is the topic of threat intelligence (variously referred to as “cyber threat intelligence” (CTI) or more simply “intelligence”) and its function within the security operations cycle. The body of knowledge on threat intelligence is both vast and voluminous; strangely, however, outside of some key concepts which occur frequently in the study of intelligence, generally, much of this body of knowledge is either disconnected or even contradictory. As such, many are unsure of exactly how to perceive threat intelligence, let alone how to integrate it into their security operations. However, before the topic of intelligence can be discussed, it is important to define what intelligence isn’t.

What Threat Intelligence Isn’t

Recently, an article was published which began its discussion on the topic of intelligence by first defining what intelligence is not. Such a beginning can be valuable as it can dispel some of the conceptual clutter which threat intelligence has accumulated over the years. Therefore, it makes sense to begin the definition of threat intelligence through negation:

Threat intelligence is not a “magic bullet.” When an intelligence consumer requests additional information or context on a particular topic, there must be the expectation and understanding that the intelligence team may not have any additional information. There is often a perception that those who practice intelligence can draw upon secret stores of knowledge (often referred to as the “deep web” or the “dark web” by marketing personnel and product feature lists alike). The reality is that while these so-called deep and dark web sources exist, they are not fonts of secret knowledge, and much of intelligence is conducted using publicly available data and information.
Threat intelligence is not “brand monitoring,” or at least it isn’t usually. The aforementioned article identified that threat intelligence is not brand monitoring and any efforts such as that should be handled by a dedicated team, and that statement is true. However, there are absolutely cases where brand monitoring may contribute to intelligence analysis, and the intelligence team should absolutely consume and analyze the results of brand monitoring, including the creation of fake social media profiles and fraudulent websites masquerading as an organization.
No part of threat intelligence is a “secondary duty.” Some organizations attempt to assign threat intelligence duties to their line analysts, such as collection of reporting and other data, however threat intelligence is a specialty, much like incident response or digital forensics, and as such it requires specific training and knowledge. It is not a task that can be distributed among other teams, and when it is distributed it takes analysts away from their primary responsibility.
Threat intelligence is not monitoring for compromised credit cards or credentials, but the results of that monitoring can serve as another input for threat intelligence. Much like brand monitoring, monitoring for stolen credit cards and compromised credentials is often a task that can be offloaded to various third-party services (or indeed implemented in-house with many open source tools). However, a threat intelligence team should absolutely monitor the results of that third-party collection for wider analysis, especially during incident response investigations.
Threat intelligence is not simply blindly reading and ingesting polished reports released by various vendors. While these reports may serve as a further input, intelligence analysis requires that such sources and reporting be assessed, collated, correlated, and analyzedto determine if the reporting aligns with existing bodies of knowledge on the subject, and if not, why. Unfortunately, this practice remains very common in various security operations centers globally, and often plays a significant factor in analyst fatigue and often contributes to the decreased perceived value of threat intelligence. After all, if all an analyst does is collect, read, and ingest vendor reporting, what value are they adding?

Threat Intelligence Formats

Another challenging aspect to threat intelligence is that it is often marketed to consumers in a variety of different layouts, many of which have few, if any, overlaps. Generally, commercially available threat intelligence is offered in two primary formats:

Intelligence Reporting — these are typically glossy and prosaic reports which are designed for human consumption. They may be written for a variety of audiences (C-level, managerial, or technical) and may include a variety of details or may be relatively sparse. The challenge many organizations face with this format, however, is that in times of incident response and investigation, analysts’ time can be monopolized sorting through various reports in an effort supplement their investigation.
Threat Feeds — these are typically feeds which leverage automated distribution and will include various indicators of compromise and varying levels of tagging. This format can also challenge organizations, as the feeds often feature out-of-date information and lack any significant contextualization, resulting in analyst fatigue and frustration as well as a surge in false positive results.

An important point to consider with both intelligence reporting and threat feeds is that frequently neither is tailored to an organization. Indeed, they are typically designed to be applicable across multiple clients. As such, most will not be purveyors of actual threat intelligence(which implies analysis and validation within an environment), but rather threat data or information. This is not to discount the value these solutions provide, as many are very valuable, rather it is to make clear that these solutions themselves do not provide intelligence but serve as inputs to an organization’s intelligence program.

With a firm grasp on understanding of what threat intelligence is not, and the distinction of what commercially available intelligence reporting and threat feeds provide, the definition of what threat intelligence is becomes clearer. The Merriam-Webster Dictionary defines intelligence as the “information concerning an enemy or possible enemy or an area,” as well as “the act of understanding.” Both of these definitions are valuable, as they simultaneously capture the inputs of intelligence (the former definition), as well as the goal of intelligence (the latter definition). However, what often remains unclear to external observers is the transitional ground between the two definitions.

The process by which an organization moves from the former definition to the latter one is often referred to as “the intelligence cycle.” The intelligence cycle is a process which has wide adoption within many government agencies responsible for collection and analysis of intelligence, and while the descriptions for the cycle can vary, the core of the process remains unchanged for several decades and includes six key steps, namely: definition and planning, collection, processing, analysis and production, dissemination, and feedback. An important point to mention is that while the intelligence cycle is often laid out as a cyclical process, multiple phases of the intelligence cycle can be concurrent, and multiple iterations of the intelligence cycle can be running depending upon the requirements of an organization and the size of the intelligence team.

Definition & Planning

Interestingly, one of the often-overlooked stages of the intelligence cycle is the first step, specifically defining intelligence requirements, and designing a collection plan. The process for establishing intelligence requirements (often referred to as “priority intelligence requirements” (PIR)) cannot be done in a vacuum, however, and requires detailed discussions with the intelligence consumers and stakeholders to identify their priorities, often in the form of questions needing answers. The more specific the question, the more specific the answers will be. Examples could include:

What actors target the financial sector?
Who is targeting us?
Are actors actively targeting a known vulnerability in our web application?
What malware is known to exploit EternalBlue?

These questions are key, as they guide intelligence professionals in what they will collect, how they will collect it, and who or what they will target. Without these questions, intelligence teams may resort to collecting for the sake of collection, a waste of valuable resources and time.

Intelligence Collection Plan

Once intelligence requirements have been clearly defined, a plan is developed with which to meet those intelligence requirements. This plan, often referred to as an intelligence collection plan or ICP, has no set format, but must include the following information:

Requirements set out by intelligence consumers and stakeholders.
Sources, resources and challenges are identified; sources and resources are ranked according to value, and challenges are identified and plans to work around them (where possible) are included.
Priorities are established, with specific requirements and sources holding different priorities according to the established requirements.
Taskings are formal directives tasking specific resources with collection and analysis. This ensures that workloads are appropriately managed and that all requirements are addressed by specific subject matter experts.
Ongoing Evaluation will occur and have overall progress of individual tasks and overall PIRs tracked.

Sources for Collection

It should be noted that many organizations, in the development of an ICP, often look to external sources for collection, such as:

Deep Web
Dark Web
Social Media
Paste Sites
News Sites
Vendor Reporting

These are all valid sources within an ICP, however an area of collection that often is overshadowed or overlooked altogether is internal data sources, especially existing incident reporting. These incidents, on their own, may reveal little about the actor at play, however in aggregate, incident reporting is often far more indicative of existing threats and previous targeting than any external source. After all, unless an actor is trying to sell access to an organization’s environment within the underground, there may be very few, or no, external sources that inform you of a breach.

Collection

Intelligence collectors, once tasked, will begin the process of gathering relevant raw (or unanalyzed) data from the sources identified in the ICP. It is important to note that intelligence collectors do not have to be analysts — indeed, they often aren’t within the broader intelligence community. Instead, their job is to gather as much raw data as possible to support the PIRs, making sure that they record not only the events of interest but to capture the additional context surrounding those events. This contextualization is critical when that data begins to be processed (normalized).

Collection can often be a messy affair. Data, as it is collected, is captured in various raw formats: untranslated forum posts, open source reporting, threat data obtained from sharing groups, malware samples, and anything else that could potentially support the defined PIRs. While some organizations have advocated for the storage of such raw data in so-called threat intelligence platforms (TIP), this can often result in significant extraneous data, which can, in turn, lead to false correlations. Instead, storing the data in its original format, often using a search and indexing engine, can be more valuable, as it allows analysts access to the raw data as it was collected.

Processing

After collection has occurred — or indeed while it is happening — the next phase is that of processing. During this phase, data and information is “processed,” or normalized, from its raw formats into more usable formats. This might include (but is by no means limited to) translating and transliterating foreign languages, de-obfuscating encoded data, or reverse engineering malware. Again, there is no set format for this data, however the processed data must be traceable to the raw intelligence. This last step is crucial, as the analyst must be able to verify the normalized data, and should any challenges or failures be experienced, validation must be able to occur.

Analysis & Production

The analysis and production phase of the intelligence cycle is where the processed data and information is provided to analysts in order to answer the fundamental questions: who, what, when, where, how, why, as well as establishing the connective tissue between the collected events and how they answer the questions laid out in the PIR. Each assessment (that is a non-obvious result of the analysis of the processed data and information) made during the analysis is based upon the processed data and information and should use either estimative language (i.e. won’t, unlikely, possible, likely, very likely) or a numerical percentage (0–100%) to convey the certainty of the assessment being true. These assessments also form the basis for one of the most important Key Performance Indicators (KPI) for threat intelligence: the number of assessments made, against the number of which are true, false, or undetermined.

Once the assessments have been made and all the PIRs have been met, the analysts will prepare reporting based on the audience. Reporting destined for line analysts in a security operations center will vary drastically from reporting bound for various C-level executives, as each audience will have extremely different requirements.

Dissemination

The dissemination phase is relatively straightforward: once the audience-specific reporting is completed, the reports are then delivered to the intelligence consumers and stakeholders. While dissemination typically precedes the feedback, it is important to note that another aspect to dissemination, which is often overlooked, needs to occur and that is actioning of the data. This means that the intelligence reporting, if it is to be called successful, will result in direct action — or indeed perhaps direct inaction — by the consumers and stakeholders. When action (or deliberate inaction) does not occur, this should trigger additional review by the intelligence team during the feedback phase.

Feedback

The final phase in the intelligence cycle is perhaps one of the most critical components: feedback. In this phase, not only is the intelligence team engaged to provide feedback on how the process was conducted, but the intelligence consumers’ and stakeholders’ feedback is also sought to determine if the reporting met the PIR and the results were satisfactory.

While threat intelligence remains a somewhat controversial topic in cyber security, and its role in the security operations cycle is not always well understood, threat intelligence is capable of providing immense value to all levels within an organization, from the C-level executives, to the individual analysts. However, it is critical that organizations consider that when implementing commercially available “threat intelligence” that they consider how it will serve to augment their intelligence program as well as add value to their security operations cycle.

For more on threat intelligence, read Soupy’s blog, The Trouble With Threat Intelligence Today.