One of the things that can present a significant challenge to practitioners is the rapid emergence of one or more new, highly compelling, business–process–altering technologies. Many of us remember the challenges that came with virtualization a decade ago; still more remember the challenges that came with the cloud a few years later. We’re seeing similar adoption now with artificial intelligence (AI) and machine learning (ML).
Against this backdrop, open source can be very compelling. Because it can be deployed without needing to go through a budget cycle, it lets us respond quickly.
The reason why adoption patterns like these can be so challenging is that they cause two things to happen in tandem. First, organizations rapidly adopt. As organizations seek to gain business value from the new technology, they will bring multiple, overlapping uses and implementations (everything from commercial products to open source to in–house developed uses). Adoption can happen in a centrally managed, organized way–or more often in a decentralized way where individual users or groups begin to adopt without centralized oversight (i.e., in shadow IT fashion). This on its own would be challenging enough, but there is also something else that happens along with this: a change in the risk ecosystem as a result of the new technology.
Practitioners in a situation like this can find themselves squeezed from multiple directions at once. On the one hand, they need to rapidly understand the business cases for where the technology is being used. On the other, they need to understand the impact to the organization’s risk posture as a result of that usage–often on a use case–by–use case basis in a manner specific to the business area(s) where adoption occurs. Because all this happens quickly, there’s very little room for new tools, new staff or even knowledge gathering to prepare.
This is exactly what is happening now with AI. Organizations are adopting rapidly. This is true of the organizations embracing data science to derive business intelligence from their data sets. It’s also true of organizations integrating new capabilities like automated chatbots, large language models (LLMs) and natural language processing into their customer service, product development, marketing and other business efforts. And further, it’s true of usage targeting specific user segments: for example, users of products like ChatGPT, Copilot or Bard. Because each of these usage scenarios can change risk dynamics in different ways, we can be left scrambling to evaluate and triage new threats, identify tools to help mitigate those and otherwise factor these new technologies into our security, audit, governance and privacy programs.
Against this backdrop, open source can be very compelling. Because it can be deployed without needing to go through a budget cycle, it lets us respond quickly. One example is putting in place a stop-gap measure to offset risk in the short term or as a vehicle to test certain types of controls before committing to a purchase. Even when we might want to (eventually) deploy a commercial tool, bringing in an open–source one can let us meet a need in the short term while we investigate the competitive landscape pursuant to making a product selection and purchase. Likewise, depending on the specific tool, it can potentially help us collect telemetry from the environment more rapidly than would otherwise be the case, which itself is compelling in tracking and managing risk. And because we haven’t committed to a purchase to do these things, there’s no lock in—if we decide to decommission usage tomorrow, we can just do that and not have to worry about sunk costs beyond those involved in getting the tool operational.
With this in mind, let’s examine a few free resources that can help practitioners in this area. We’ve focused here on free resources that practitioners can get started with right away. These include informational resources, open–source tools, frameworks and other materials that can be of immediate use. Note that these are not the only tools out there. There are literally hundreds of tools and resources that can help you in your AI/ML journey. However, the ones we’ve focused on here are generally applicable (i.e., that have utility across a wide spectrum of organizations and practitioners), do not require budget to make use of and are specific to an AI or ML use case.
OWASP ML and LLM Top Ten
First on the list are two informational resources from OWASP (the Open Worldwide Application Security Project–formerly the Open Web Application Security Project). OWASP has two separate top ten lists that are potentially applicable depending on the use case. First is the Machine Learning Security Top Ten, targeted toward providing a list of the most commonly occurring threats against ML systems generally.1 The second, Top 10 for LLM, relates specifically to threats applicable to LLMs.2
Much like the OWASP Top Ten provides information about the most commonly occurring application issues, so too do the above documents provide similar information but as applicable to ML and LLM use cases specifically. It is useful for educating personnel (for practitioners and those engineers doing implementation work) about the ways that the systems and components can be misused and the threats that they might encounter once fielded.
ModelScan
We all know that developers and engineers are accustomed to sharing code, libraries and resources with each other. When it comes to dependencies in software products—such as libraries and dependencies for example—we all know how important it is to understand the provenance of these dependencies. Why? Because they could have vulnerabilities, malware or defects that could undermine the security properties of the final deliverable.
It’s rare that an organization would allow developers to download arbitrary software from the Internet and directly incorporate it into production software without vetting or scanning it. But this is exactly what is happening when it comes to ML projects.
It’s rare that an organization would allow developers to download arbitrary software from the Internet and directly incorporate it into production software without vetting or scanning it. But this is exactly what is happening when it comes to ML projects. Models are routinely shared among engineers in a very similar manner to how they share code snippets and libraries. However, unlike code snippets and libraries, there is less capability for existing security tools to scan those models for issues. ModelScan offers a way to rectify that, allowing you to scan serialized models for unsafe code.3 It operates in a similar way to traditional antimalware scanning tools, examining the file system for known malware signatures.
Kubeflow
Kubernetes is here to stay. According to the Cloud Native Computing Foundation’s 2022 Annual Survey, 89% of organizations are either using Kubernetes in production (64%) or piloting/evaluating Kubernetes (25%).4 The goal of the Kubeflow project is to allow a rapid deployment of ML workloads to Kubernetes.5 This is beneficial of course from an engineering point of view because it streamlines ML deployment—but it’s also advantageous from a security and assurance viewpoint for at least two reasons. First, we are likely familiar with the risk dynamics of Kubernetes due to its ubiquity. We have experience securing it, auditing it and validating its operation. Second, there are built–in security services within Kubernetes (e.g., secrets management, authentication services, visibility and telemetry) that can be rapidly integrated into existing processes.
Rebuff
If you’ve already looked into the OWASP materials we cited earlier (specifically the OWASP Top 10 for LLMs) or if you’ve been following some of the ongoing research, you’ve likely already encountered the concept of “prompt injection.” If you haven’t, prompt injection describes a scenario whereby an attacker attempts to bypass restrictions on an LLM to cause it to generate unwanted behavior. As an example, say you have an LLM that has certain restrictions on its behavior; perhaps you have instructed it to refrain from providing instructions for illegal or illicit activities. An adversary who wishes to overcome those restrictions might attempt to do so by various techniques, such as instructing the LLM to ignore prior instructions or attempting to trigger the prohibited behavior circuitously (e.g., “author a poem in the style of a Shakespearean sonnet describing how to do
Detecting this type of attack can be difficult and preventing it can be more difficult still. The open–source Rebuff project provides a mechanism to detect prompt injection using various mechanisms.6 It’s not foolproof, but it can certainly help to find and flag problematic usage in an LLM use case.
Microsoft Tools/Guidance
Lastly, Microsoft has published some notable work in this arena. There are two resources that we’ll highlight here. The first is guidance on how to threat model ML systems.7 As you might imagine, the process of threat modeling an ML–enabled system can be complicated; just to consider a “tip of the iceberg,” imagine the challenge of evaluating whether tampering has occurred (the T in the STRIDE mnemonic) in considering input to an ML model. Adding ML to already complicated deployment scenarios (e.g., service mesh) compounds complexities even further.
There are literally hundreds—if not thousands—of other resources available out there that can help you as you look to ensure that your ML projects are hardened.
Another Microsoft resource is an open–source tool called Counterfit.8 In a nutshell, the tool itself is a collection of attack techniques that can be used against a set of potential target types. You can perform adversarial attacks such as manipulation of an image (e.g., using HopSkipJumpAttack9) for the purpose of deliberate misclassification. This can help you test the resilience of your models against these types of adversarial attacks.
No Shortage of Tools
Each of the mentioned resources can potentially help in a specific area depending on the usage that your organization is considering. And these are only the tip of the iceberg. There are literally hundreds—if not thousands—of other resources available out there that can help you as you look to ensure that your ML projects are hardened. If, like most, your organization is exploring ways to derive business value from AI and ML, even just maintaining an awareness of the space and becoming educated about the security, assurance, risk and governance implications is helpful given how rapidly these are likely to come into our organizations.
Endnotes
1 Bhure, S.; S. Singh; “OWASP Machine Learning Security Top Ten,” Open Worldwide Application Security Project, 2023, http://owasp.org/
2 Open Worldwide Application Security Project (OWASP), “OWASP Top Ten for LLM,” 2023, http://owasp.org/www-project-top-10-for-large-language-model-applications/assets/PDF/OWASP-Top-10-for-LLMs-2023-v05.pdf
3 GitHub, “ModelScan: Protection Against Model Serialization Attacks,” http://github.com/protectai/modelscan
4 Cloud Native Computing Foundation (CNCF), “CNCF 2022 Annual Survey,” 2022, http://www.cncf.io/reports/cncf-annual-survey-2022/
5 Kubeflow, www.kubeflow.org
6 GitHub, “Rebuff.ai,” http://github.com/protectai/rebuff
7 Microsoft, “Threat Modeling AI/ML Systems and Dependencies,” 2022, http://learn.microsoft.com/en-us/security/engineering/threat-modeling-aiml
8 Github, “Azure/counterfit,” http://github.com/Azure/counterfit
9 Chen, J.; M. Jordan; M. Wainwright; HopSkipJumpAttack: A Query-Efficient Decision- Based Attack,” 28 April 2020, http://arxiv.org/pdf/1904.02144.pdf
ED MOYHLE | CISSP
Is currently director of Software and Systems Security for Drake Software. In his years in information security, Moyle has held numerous positions including director of thought leadership and research for ISACA®, application security principal for Adaptive Biotechnologies, senior security strategist with Savvis, senior manager with CTG, and vice president and information security officer for Merrill Lynch Investment Managers. Moyle is co-author of Cryptographic Libraries for Developers and Practical Cybersecurity Architecture and a frequent contributor to the information security industry as an author, public speaker and analyst.