Online filtering

Curated PJ.

Curated dataset of predatory chats from Perverted Justice.

Haidra Horde Safety

Uses CLIP+BLIP for safety features for the AI Horde.

PDQ:

Open source hashing algorithm to detect known CSAM

PhotoDNA:

Hashing algorithm to detect known CSAM

Cybertip API:

Tool to report child exploitation to NCMEC

IWF URL List:

Removes URLs related to CSAM

Llama-Guard

LLM-based input-output safeguard models geared towards Human-AI conversation use cases.

ShieldGemma

A set of instruction tuned models for evaluating the safety of text prompt input and text output responses against a set of safety policies.

Perspective API

Classifiers which score content based on issues including toxicity, sexually explicit content, and threats. Can be used as a mitigation or evaluation tool.

Papillon filter

Uses LLM to create privacy preserving queries for the user.

Presidio

Provides fast identification and anonymization modules for private entities in text and images such as credit card numbers, names, locations etc

PII processing

Multilingual Named Entity Recognition and PII processor.

Scrub

Provides multiple levels of scrubbing with ML to ensure optimal anonymization of sensitive information and safeguarding of user privacy

Octopii

uses OCR, regex lists NLP to search public-facing locations or Government ID, addresses, emails etc in images, PDFs and documents.

Gitleaks

tool for detecting secrets like passwords, API keys, and tokens in git repos.

Prompt Guard

A classifier model trained on a large corpus of aacks capable of detecting both explicitly malicious prompts and prompts that contain injected inputs.

Prompt-Guard-86M

Detects both explicitly malicious prompts as well as data that contains injected inputs.

protectai/deberta-v3-base-prompt-injection-v2

Detects and classifies prompt injection aacks that can manipulate language models into producing unintended outputs.

deepset/deberta-v3-base-injection

Prompt injection detection and classification Programmable guardrails: though they are not specific to prompt injection and jailbreaking, they can allow to filter the inputs and outputs against these aacks.