AIX

Moath! You raise such important and good points, data poisoning, and corrupt or biased training datasets do lead to inaccurate model outputs certainly. And ML systems are dependent on their training data to perform, which makes this a huge issue and makes the ML systems susceptible to these data attacks which can happen in the form of malicious prompts that alter training data too and data poisoning can add harmful back doors into systems. eg. Image recognition systems that are poisoned will misidentify objects if an attacker alters pixel patterns.

While fraud detection systems can be used to analyze data patterns they won't work if hackers introduce manipulations into training data to misclassify things like identifying fraud as legal etc.

So what to do?

1. Regular & consistent risk assessments of training data and incoming data to check for anomalies

2. Use comprehensive and diverse datasets in the training phase

3. Apply encryptions to verify data authenticity before a knowledge base receives new data files

4. Conduct adversarial / red teaming on the model before deployment to ensure resilience

5. Put in strict guardrails to restrict any modifications to training datasets
   

Real world examples, that are quite known have happened including: The Twitter attack of 2023 where a malicious prompt caused information leakage and misinformation posts; Google DeepMind’s ImageNet Incident also in 2023 and MIT LabSix’s Adversarial Attack where training images were manipulated leading to misidentification of objects. While those were examples affecting large companies with big products, the same can happen to any model from any group and it makes safety design so important for everyone developing and deploying AI models.