Evaluating AI in Infosec

As part of the evaluation process, looking at dozens of technologies sprawling all across the industry, we are often asked about the best and highest use cases for AI. Should AI be part of a product at all?  Should it be the main feature, or in it, period? What exactly is the best use case anyway? Here is how we think about AI’s use within vendor applications in case it is helpful to you.

‍

To be useful, it needs to be three distinct things all at the same time:

1. There must be a fault-tolerant use case
2. It can be allowed to be expensive every time it is used
3. The use case must be allowed to be slow

‍

Let me explain each one, one at a time.  First, why must it be fault-tolerant? Well, many reasons!  Hallucinations, refusals, API outages, updates that change the underlying model without your knowledge, models getting EOL’d, API cost overruns, etc. Frankly, LLMs are unreliable, non-deterministic, and flakey.  So whatever the use case you have, it must be fault-tolerant in such a way that occasional or even frequent failures are acceptable. A good example of fault tolerance is anti-spam, where if a spam email gets through occasionally, it’s not a big deal. A dangerous use case would be automatically deleting user-level access based on perceived threats, because if it hallucinates, you’ve just DoS’ed your own users.

‍

Second, it must be allowed to be expensive - at least within whatever cost constraints you have pre-allocated. LLM tokens cost a lot, and they may start costing a lot more once frontier model LLMs stop subsidizing the token costs to get market share. So unless you are sitting on a lot of hardware that you can run local models on, you’re going to be paying for that LLM usage. And even if you do have the hardware, you still have power costs. Tokens cost money, and each time you use an LLM, you are spending money. So whatever that cost is, it’s got to be able to be absorbed by whatever your model is. If you have runaway code, it may spend a lot more than you are expecting, so a large buffer or tight cost controls are always a good idea. A good use case would be summarization of emails, because you don’t get that many emails per day so expense is mediated by the small volume. A bad use case would be analyzing every packet on a network, because there is just too much happening on a network and it would get very costly very fast.

‍

Lastly, it’s got to be allowed to be slow. Very slow. Token generation even for very fast models is nowhere near wire-speed fast. An LLM’s capacity to produce a paragraph every handful of seconds is nowhere near fast enough to make real-time packet-level decisions.  Whatever the use case you have, it’s got to be allowed to slowly respond to whatever the stimuli is. A good use case where this is okay is going over trend data, where you may want to look at macroscopic trends and waiting around for a few seconds or hours isn’t a big deal. A bad use case is in identifying malicious financial transactions, where any delay may result in money being lost, and could cause buyers to move elsewhere.

‍

As you look at business models, it would behoove vendors to decide if their product use cases meet all three criteria. Otherwise, they might be building something that will never survive contact with the real world, and the adversary.