Hello again. Hiatus is over. Good to be posting again about Generative AI applications for the cyber threat intelligence (CTI) and cybersecurity domains. I’m particularly excited about the addition of ChatGPT Plus (4.0) to my kit.

Much of the press and research on the impact on Generative AI in CTI and cybersecurity have focused on low-level technical implementations, particularly in adversarial (3-6) and defender (7-9) contexts. In this post, however, we consider a more strategic application: how CISOs and security executives can harness Generative AI as an executive assistant for tasks ranging from best-practice guidance research, high-level solution design, budgeting and RFI preparation.

Key Takeaways:

For tasks like cybersecurity market research, a skilled CTI analyst using ChatGPT 4.0 could reduce the time it takes to generate a credible first draft from 20 hours to 1 hour (estimate).
While difficult to quantify, this productivity gain could enable analysts to improve the quality of their work, as analysts would gain time to consider alternate approaches and models.
ChatGPT 4.0’s capability to reference up-to-date sources marks a considerable advancement from version 3.5. However, crafting prompts to fetch the newest and most authoritative resources remains a challenge.
Effective use requires a skilled analyst equipped with domain expertise, professional skepticism, patience, persistence, and a knack for iterative dialogue refinement.
Despite its capabilities, ChatGPT requires human oversight. Given its current state, it should not be relied on to produce finished intelligence or reports without review.

The inspiration for this post was sparked by the publication dated18-Oct-2023, Best Practice Phishing Guidance for network defenders by CISA, NSA, FBI and MS-ISAC (Multi-State Information Sharing and Analysis Center). [1-2] The initial idea was to compare a ChatGPT-crafted best practice reference against the MS-ISAC authoritative guidance. However, a deeper reflection from a CISO’s vantage point led us to expand the scope of inquiry. Beyond guidance, CISOs may also be interested in understanding system designs that operationalize such guidance and getting ballpark figures on the acquisition and operating costs of these systems.

Findings:

For the formulation of best practice guidance on phishing defense, ChatGPT’s recommendations were comprehensive and credible when compared to the human expert guidance represented in the MSI-ISAC report, as illustrated in Figures 1 and 2.
ChatGPT’s suggestions on solution architectures and vendor choices seemed logical, though they remain unverified (Figure 3).
Cost estimations for a comprehensive phishing defense system from ChatGPT seemed rational, but again, they weren’t validated (Figures 4 and 5).
The superior speed and efficacy of ChatGPT-driven searches make it easier for analysts to do deeper analysis and assess multiple scenarios. For instance, while probing solution architectures, we also explored MSSP-based phishing solutions but haven’t detailed the outcomes here.
Analysts need to test for consistency; we found that ChatGPT can generate different results for nearly identical prompts, and its outputs can be swayed by user feedback.

Conclusion:

Any analytic task that requires traditional search, would also benefit from Generative AI in combination with conventional search.