Comparison and Evaluation of Pseudonymization Algorithms for Client Reports in Intellectual Disability Care
Background
Within the academic collaborative center “Sterker op Eigen Benen”, we aim to improve healthcare for people with intellectual disabilities. Research using routinely collected healthcare data plays an important role in this. Client reports in electronic care records contain valuable, yet privacy-sensitive information about behaviours, symptoms, and events that are crucial for diagnostics and research, such as predicting health issues (e.g., cancer) or behavioural incidents. Due to the highly sensitive nature of this data, privacy protection is essential.
CARMENDA is the data infrastructure program within the “Sterker op Eigen Benen” collaborative and provides a privacy tool (available at https://www.carmenda.nl/oplossingen-diensten/privacy-tool/) for pseudonymizing textual data. This tool runs locally as a desktop application within care organizations. The current implementation uses the Deduce algorithm (see: https://github.com/vmenger/deduce), and Deidentify (see: https://github.com/nedap/deidentify) which detect personal data such as names, addresses, and phone numbers and replaces this sensitive data with labels. Although these algorithms perform reasonably well, other algorithms may perform better in terms of speed, accuracy, and usability.
Objective
The objective of this project is to improve the existing algorithms for pseudonymizing Dutch client reports based on accuracy and performance. When possible, we will integrate this updated algorithm into the existing CARMENDA privacy tool.
Activities
- Inventory
- What pseudonymization algorithms exist for Dutch free text?
- What are their main characteristics (language support, open source/license, API or local deployment)?
- Annotation & Test Set Development
- Annotate existing test datasets with entities such as first names, surnames, addresses, locations, institutions, and phone numbers.
- Optionally generate synthetic test data.
- Define Evaluation Criteria
- Accuracy: precision, recall, F1-score
- False positives and false negatives
- Performance: processing speed on large datasets (current tool: ~6 minutes for 100,000 rows)
- Evaluate
- Run the algorithms on the test datasets
- Analyze performance per algorithm
- Further development
- Adapt or retrain models to enhance performance
- Re-evaluate the finetuned models
- Reporting & Recommendation
- Write a summary report comparing the algorithms
- Provide a recommendation for the best-performing algorithm for integration into the CARMENDA tool
Expected Results
- Annotated dataset(s) for benchmarking
- Automated scoring system with confusion matrices
- Benchmark report of different algorithms
- Recommendation for integration into the privacy tool
Practical Information
- Level: BSc/MSc in Computer Science, AI, Data Science, Health Technology, or related field
- Location: Nijmegen, partly hybrid
- Start date: to be agreed upon
- Language: being fluent in Dutch is a prerequisite
Supervision & Contact Details
- Pim van Oirschot (Radboudumc / “Sterker op Eigen Benen”)
- Lola Vissers (Radboudumc / “Sterker op Eigen Benen”)
