Model-agnostic PII detection with LLMs

Amazon's new PII detection system leverages large language models (LLMs) to identify personally identifiable information in unstructured text data. Unlike traditional token-classification models that require retraining for new entity types, this system uses instruction-driven prompts that define what entities to detect, making it highly adaptable.

The detector operates by wrapping input text in a prompt that specifies the PII entities of interest, and then querying an LLM—either via Amazon Bedrock or a self-hosted model—to produce a structured list of detected entities. This approach allows for easy customization, as changing the detection scope involves editing the prompt rather than retraining the model.

The implementation comprises four main components: a prompt schema, a backend inference interface, a response parsing layer, and a call sequence that ties these together. The schema defines entity categories and their descriptions, while the backend can connect to various models through a uniform interface, such as Amazon Bedrock or custom-hosted models.

Benchmarking against five public datasets across multiple languages shows that the system achieves competitive accuracy, with span-level F1 scores ranging from approximately 75% to over 83%, depending on the model and configuration. The system's flexibility extends to detecting domain-specific entities, which can be added by updating the prompt instructions without retraining.

Overall, this approach offers a scalable, adaptable, and privacy-conscious method for PII detection, suitable for integration into data processing pipelines, compliance workflows, and privacy-preserving AI applications.