Rajesh Kanungo (CEO, TalaSecure, Inc.)
Tejpal Chadha (CEO, MyAiDr.ai)
Executive Summary
Investment in Artificial Intelligence (AI) and machine learning (ML) is skyrocketing, with the global AI market valued at $196.63 billion in 2023 and projected to grow at a CAGR of 36.6% from 2024 to 2030. The U.S. leads with $62.5 billion in private investment, followed by China and the EU. AI's impact spans industries, with the manufacturing sector expected to gain $3.78 trillion by 2035. Significant growth is also anticipated in AI applications like self-driving vehicles and AI-driven customer service.
However, AI/ML systems face significant security threats, especially with Generative AI (GenAI) and Large Language Models (LLMs). Examples include adversarial attacks on autonomous vehicles, data poisoning like Microsoft's Tay chatbot incident, and model extraction attacks on API-based services.
Healthcare AI/ML systems are particularly vulnerable, with Healthcare ransomware attacks costing, on average, $11 million per incident and posing. Healthcare ransomware attacks cost, on average, $11 million per incident and pose risks to patient safety.
This article outlines these problems and proposes mitigations, including robust security measures, continuous monitoring, strong data validation practices, and GenAI-driven auto-remediation solutions to ensure the reliability and integrity of AI/ML systems. Addressing these risks is crucial for leveraging AI's full potential while protecting against threats.
The AI/ML Economy is Big
Investment in AI and machine learning (ML) has been growing rapidly and is substantial. In 2023, the global market for AI was valued at approximately $196.63 billion and is projected to grow at a compound annual growth rate (CAGR) of 36.6% from 2024 to 2030 (Statista). The United States leads in private investment, with $62.5 billion invested in 2023, followed by significant contributions from China and the European Union (IEEE Spectrum).
Furthermore, AI technology's impact is broad and profound across various industries. For example, the manufacturing sector alone is expected to gain $3.78 trillion from AI by 2035, highlighting the technology's transformative potential in improving efficiency and productivity (Exploding Topics). Investments are also being channeled into developing AI-powered applications like self-driving vehicles and AI-driven customer service solutions, which are projected to see significant growth and revenue generation in the coming years (Exploding Topics) (IEEE Spectrum)
Documented Examples of Attacks on AI/ML Systems
AI/ML Systems have been attacked. The newer generation of AI systems, known as Generative AI (GenAI) and Large Language Model (LLM) systems, are of special concern. LLMs and GenAI offer incredible potential, but their power comes with security challenges.
1. Adversarial Attacks
Autonomous Vehicles: Researchers have demonstrated that by adding stickers or paint to road signs, they can cause autonomous vehicles to misinterpret the signs. For example, a stop sign might be interpreted as a yield sign, leading to dangerous driving decisions.
Image Recognition: Adversarial attacks on image recognition systems, like those used by Google, have shown that small perturbations can make a model misclassify objects.
2. Data Poisoning
Tay Chatbot: Microsoft’s AI chatbot Tay was manipulated by users who fed it offensive and inappropriate inputs, causing it to produce racist and sexist tweets. This is a form of data poisoning where the input data is designed to corrupt the model’s behavior.
Machine Learning in Cybersecurity: Attackers have poisoned datasets used by cybersecurity firms to train malware detection models, leading to lower detection rates of certain types of malware.
3. Model Inversion Attacks
Reconstructing Faces: Researchers have shown that it’s possible to reconstruct images of people’s faces from a facial recognition model’s outputs. This was demonstrated with models used for image recognition tasks.
4. Membership Inference Attacks
Online Services: There have been instances where attackers were able to determine if specific data points were part of the training data of ML models used by online services, such as those offering machine learning as a service (MLaaS).
5. Evasion Attacks
Spam Filters: Spammers constantly modify their tactics to evade machine learning-based spam filters. By tweaking their messages slightly, they can avoid detection by these systems.
Malware Detection: Similarly, malware creators use evasion techniques to bypass ML-based antivirus software. By making small changes to the malware code, they can prevent it from being detected.
6. Model Extraction Attacks
API-Based Services: Attackers have successfully extracted models from MLaaS providers by sending numerous queries to the service and using the responses to recreate the model locally. This has been demonstrated with services like those offered by Amazon, Google, and Microsoft.
7. Backdoor and Trojan Attacks
ML Models in Academia: Researchers have embedded backdoors in ML models used for academic purposes to demonstrate how these attacks can be carried out. These backdoors are triggered by specific inputs that were not part of the normal training data.
Commercial AI Systems: There have been concerns and reports about the possibility of backdoors being embedded in commercial AI systems, especially in critical infrastructure and defense applications.
Examples and Case Studies
CAPTCHA Systems: Adversarial attacks on CAPTCHA systems used by websites to distinguish between human users and bots have been demonstrated, where slight modifications to the CAPTCHA images can make them easier for automated systems to solve.
Healthcare ML Models: Researchers have shown that medical imaging models, such as those used for diagnosing diseases from X-rays, can be fooled by adversarial examples, potentially leading to misdiagnoses.
These instances highlight the importance of robust security measures in the development and deployment of AI/ML systems. Organizations need to be aware of these risks and implement strategies to mitigate potential attacks.
Problems Amplified in GenAI/LLMs
GenAI and LLMs lend themselves to attacks because they consume massive amounts of data and produce various outputs, sometimes hallucinatory.
One of the AI platforms we studied, Hugging Face, has opened up the possibility of targeted attacks. Some attacks that are amplified in GenAI and LLMs are:
1. Poisoned Data: LLMs are trained on massive datasets. If attackers can inject malicious or biased data into the training process, the model can be manipulated to generate harmful outputs. This is a major concern for Hugging Face, as anyone can upload models.
2. Prompt Injection: Attackers can craft specific prompts or questions to trick LLMs into generating misleading or unsafe content. This can be used for phishing attacks,
3. Model Theft: The complex algorithms and knowledge encoded within LLMs are valuable assets. Attackers might try to steal these models to replicate their capabilities for malicious purposes.
4. Supply Chain Risks: Hugging Face allows sharing and collaboration on models. Malicious actors could upload compromised models or exploit vulnerabilities in the platform to infect other models.
5. Disclosure of Sensitive Information: LLMs trained on real-world data might inadvertently memorize and leak sensitive information present in the training data. This could be a privacy nightmare if it involves personal details or confidential information.
Hugging Face Specific Concerns:
Platform Vulnerabilities: Recent incidents have shown vulnerabilities in Hugging Face's infrastructure that could allow attackers to upload malicious models or tamper with user data.
Big Investment Means Big Risk
AI and machine learning (ML) systems introduce several unique security risks that differ from those associated with traditional software. Here are some key security risks:
# | Risk Category | Description | Impact |
1 | Data Poisoning | Malicious actors introduce manipulated data into the training dataset, causing the AI/ML system to learn incorrect patterns. | Reduced accuracy, compromised decision-making, and potential exploitation of the system's weaknesses. |
2 | Model Inversion | Attackers infer sensitive information about the training data by querying the AI model and analyzing its outputs. | Breach of privacy and confidentiality, exposing sensitive or proprietary data. |
3 | Adversarial Attacks | Deliberately crafted inputs designed to deceive AI/ML models, causing them to make incorrect predictions or classifications. | Misclassification of data, evasion of detection systems, and undermining the reliability of AI models. |
4 | Model Stealing | Attackers replicate an AI/ML model by making repeated queries and using the responses to train their own models. | Intellectual property theft, loss of competitive advantage, and unauthorized use of proprietary models. |
5 | Bias and Fairness | AI/ML models inadvertently learn and propagate biases present in the training data. | Unfair treatment of individuals or groups, legal and ethical issues, and reputational damage. |
6 | Lack of Transparency and Explainability | AI/ML models can be complex and difficult to interpret, making it hard to identify and mitigate security risks. | Difficulty in identifying and mitigating security risks, lack of accountability, and challenges in regulatory compliance. |
7 | Overfitting and Generalization | Models that perform well on training data but fail to generalize to new, unseen data. | Poor performance in real-world scenarios, leading to incorrect decisions and potential vulnerabilities. |
8 | Insider Threats | Insiders with access to AI/ML systems can manipulate models or data for malicious purposes. | Data breaches, sabotage of models, and unauthorized access to sensitive information. |
9 | Deployment Vulnerabilities | Security flaws in the infrastructure used to deploy AI/ML models, such as APIs, cloud services, and edge devices. | Unauthorized access, data leakage, and exploitation of system vulnerabilities. |
10 | Regulatory and Compliance Risks | Failure to comply with laws and regulations governing the use of AI/ML, particularly concerning data privacy and security. | Legal penalties, fines, and reputational damage. |
11 | Prompt Injection | Attackers can craft specific prompts or questions to trick LLMs into generating misleading or unsafe content. This can be used for phishing attacks | Danger, safety, financial, system, and other attacks |
A High-Level View of an AI/ML System
Differences Between Normal and AI/ML Systems
The differences between AI systems and traditional software are primarily in how they are developed and operated/used and the variety of tasks they are designed to handle. Here are the key distinctions:
Category | Traditional Software | AI Systems |
Development | Predefined rules and logic | Machine learning algorithms trained on data |
Operation | Predictable and consistent behavior | Adaptive and data-driven, handling complex tasks |
Interaction | Specific user commands | Natural language processing for intuitive interactions |
Error Handling | Debugging code | Refining models and data |
Performance Metrics | Efficiency and specifications | Accuracy and statistical measures |
Problems in Securing AI/ML Systems
Reason | Description |
Complexity and Opacity | AI/ML models are complex with millions of parameters, and often operate as black boxes. |
Dynamic Learning Environment | Continuous learning and heavy data dependency can introduce new vulnerabilities. |
Adversarial Nature of Attacks | Adversarial examples are easy to craft, and attack techniques are becoming increasingly sophisticated. |
Lack of Robust Defense Mechanisms | AI/ML models often lack inherent robustness and defenses are usually reactive. |
Resource Constraints | Defending against sophisticated attacks requires significant computational resources and expertise. |
Diverse Attack Vectors | Multiple attack surfaces and supply chain vulnerabilities increase the complexity of securing systems. |
Regulatory and Ethical Challenges | Ensuring data privacy while preventing attacks, and balancing robustness with fairness, adds complexity. |
Evolving Threat Landscape | The rapidly advancing field and unknown vulnerabilities make it difficult to stay ahead of threats. |
AI/ML Security Mitigations
Mitigation Strategy | Description |
Adversarial Training | Incorporating adversarial examples into the training process to improve model robustness. |
Regular Audits and Testing | Conducting regular security audits and stress tests to identify and mitigate vulnerabilities. |
Explainable AI | Developing more interpretable models to understand decision-making processes and identify weaknesses. |
Access Control | Implementing strict access controls to limit exposure to potential attackers. |
Monitoring and Detection | Deploying continuous monitoring systems to detect and respond to suspicious activities in real-time. |
Collaboration | Encouraging collaboration between AI researchers and cybersecurity experts to develop comprehensive defense mechanisms. |
Security Best Practices for GenAI/ML
For the newer breed of AI systems, like GenAI/ML systems, special attention should be paid to the following:
Mitigation Strategy | Description |
Data Validation | Ensure the data used to train LLMs comes from trusted sources and is thoroughly vetted for bias or malicious content. |
Prompt Engineering | Develop robust prompt design techniques to minimize the risk of manipulation. |
Model Monitorin | Continuously monitor LLM outputs for signs of bias, misinformation, or other issues. |
Secure Development Practices | Implement strong security measures throughout the LLM development lifecycle. |
Transparency and Governance | Establish clear guidelines and best practices for sharing and using models on platforms like Hugging Face. |
Healthcare Safety and Privacy
Healthcare-related AI systems hold immense potential to significantly improve patient outcomes in the coming years. However, these systems must comply with cybersecurity regulations such as FDA guidelines, HIPAA, EU-MDR, among others, depending on their classifications. The frequency of attacks on healthcare systems is increasing, with numerous incidents already documented (see appendix).
In addition to general AI security concerns, healthcare AI systems face several unique challenges:
Protection Against Healthcare-related Ransomware Attacks: Each incident can cost up to $11 million.
Safety and Patient Harm: Human safety can be compromised, potentially leading to serious injury or death.
Regulatory Requirements: Compliance with FDA Cybersecurity PreMarket Guidelines is essential.
Third-party Model Verification and Validation: Ensuring security and privacy during third-party verification and validation processes.
Double-blind Studies: Conducting rigorous and secure clinical studies.
Contract Research Organizations (CROs): The impact of CROs on system implementation and security.
FDA Recall Letters and Response Times: The current average for fixing security issues is 295 days. The FDA requires that vendors inform customers of any breaches in at least 30 days and provide remediations in at most 60 days. The FDA may recall the vendor’s products if the vulnerabilities are not addressed in a responsible and responsive manner.
A critical point is that model verification, model validation, and clinical studies often require third-party, independent access to systems as mandated by regulatory bodies. This access introduces additional safety risks.
Gen-AI systems, in particular, must be continuously safeguarded even after FDA clearance to prevent attacks that could lead to patient harm.
TalaSecure GenAI/ML Tools and Solutions
TalaSecure has experience providing security products and solutions to AI/ML based healthcare and wellnes companies. It provides:
Continuous monitoring: Tala Defender provides continuous monitoring using Adabtable AI System
Autoremediation in Minutes instead of 295 Days: Taa Samrt GenAI system provides solutions in minutes.
Customized solutions: a team of Medical Device Experts, FDA regulatory compliance experts, AI Experts, and cybersecurity experts, each with decades of experience is available to get through your regulatory and compliance requirements.
References
1. Adversarial Attacks
Autonomous Vehicles:
Eykholt, K., et al. (2018). "Robust Physical-World Attacks on Deep Learning Models." IEEE Conference on Computer Vision and Pattern Recognition (CVPR).
Cao, X., et al. (2019). "Adversarial Objects Against LiDAR-Based Autonomous Driving Systems." European Conference on Computer Vision (ECCV).
Image Recognition:
Goodfellow, I. J., et al. (2015). "Explaining and Harnessing Adversarial Examples." International Conference on Learning Representations (ICLR).
2. Data Poisoning
Tay Chatbot:
Vincent, J. (2016). "Twitter taught Microsoft’s AI chatbot to be a racist asshole in less than a day." The Verge.
Machine Learning in Cybersecurity:
Biggio, B., & Roli, F. (2018). "Wild Patterns: Ten Years After the Rise of Adversarial Machine Learning." Pattern Recognition.
3. Model Inversion Attacks
Reconstructing Faces:
Fredrikson, M., et al. (2015). "Model Inversion Attacks that Exploit Confidence Information and Basic Countermeasures." ACM SIGSAC Conference on Computer and Communications Security (CCS).
4. Membership Inference Attacks
Online Services:
Shokri, R., et al. (2017). "Membership Inference Attacks Against Machine Learning Models." IEEE Symposium on Security and Privacy (SP).
5. Evasion Attacks
Spam Filters:
Biggio, B., et al. (2013). "Evasion Attacks against Machine Learning at Test Time." Joint European Conference on Machine Learning and Knowledge Discovery in Databases (ECML PKDD).
Malware Detection:
Anderson, H. S., & Roth, P. (2018). "EMBER: An Open Dataset for Training Static PE Malware Machine Learning Models." arXiv preprint.
6. Model Extraction Attacks
API-Based Services:
Tramèr, F., et al. (2016). "Stealing Machine Learning Models via Prediction APIs." USENIX Security Symposium.
7. Backdoor and Trojan Attacks
ML Models in Academia:
Gu, T., et al. (2017). "BadNets: Identifying Vulnerabilities in the Machine Learning Model Supply Chain." arXiv preprint.
Examples and Case Studies
CAPTCHA Systems:
Goodfellow, I. J., et al. (2015). "Explaining and Harnessing Adversarial Examples." International Conference on Learning Representations (ICLR).
Healthcare ML Models:
Finlayson, S. G., et al. (2019). "Adversarial attacks on medical machine learning." Science.
Healthcare Related Attacks
1 Cost of a Data Breach Report 2023, IBM Security Link
2 WannaCry Ransomware Attack (2017):
Reference: BBC News. "WannaCry ransomware cyber-attacks slow but fears remain." May 2017. Link
Medtronic Insulin Pump Vulnerabilities (2019):
Reference: US Food and Drug Administration (FDA). "Medtronic Recalls MiniMed Insulin Pumps for Potential Cybersecurity Risks." June 2019. Link
3 St. Jude Medical Cardiac Devices (2016):
Reference: US Food and Drug Administration (FDA). "Medical Device Safety Communications: Cybersecurity Vulnerabilities Identified in St. Jude Medical's Implantable Cardiac Devices and Merlin@home Transmitter: FDA Safety Communication." January 2017. Link
4 Pacemaker Vulnerabilities (2017):
Reference: Ars Technica. "FDA issues cybersecurity alert for Medtronic's insulin pumps." March 2019. Link
5 IoT Botnet Attacks (Various Instances):
Reference: Symantec. "Internet of Things (IoT) Attack Landscape Report." April 2019. Link
6 X-ray and MRI Machine Vulnerabilities:
Reference: Threatpost. "Security Flaws in Philips X-Ray Machine Can Let Hackers Control Devices." October 2020. Link
7 Phishing Attacks on Healthcare Providers:
Reference: Healthcare IT News. "Phishing attack prompts data breach at Minnesota health system." October 2020. Link
Comentarios