NodeSynth is a research prototype that implements a scalable, multi-stage methodology for creating socially relevant and evidence-grounded synthetic data (e.g., annotated queries) for AI model evaluation.
The pipeline breaks down topics related to safety policies (e.g., harassment) and sensitive domains (e.g., education) into granular taxonomies using a fine-tuned taxonomy generator. It identifies key relationships within these taxonomies (e.g., affected social groups, geographic regions, use cases) and generates high-fidelity synthetic queries designed for rigorous model evaluation.
Standard benchmarks and manual query creation struggle to capture real-world sociotechnical nuance or scale effectively. While generic synthetic data offers an alternative, these datasets often contain unintended biases, lack diversity, and are inaccurate for highly-sensitive domains. NodeSynth enables users (researchers, developers, and auditors) to go from a high-level topic to a rich synthetic dataset capturing relationships that represent documented harms in the real world.
- Sociotechnical Framework: Leverages an expert-curated Taxonomy Generator (TaG) to ground abstract concepts in concrete, evidence-based scenarios.
- Empowered Scaling: Enables resource-constrained entities to conduct lightweight, scalable model evaluations specific to sensitive topics.
- Interpretable Diagnostics: Allows evaluators to trace exact failure intersections (demographics, geography, taxonomy level) to prioritize key areas of concern for targeted mitigation and in-depth human evaluation.
Figure 1: A visual representation of the NodeSynth approach. Based on user inputs, NodeSynth (Step 1) creates a complete, three layer taxonomy using a fine-tuned model; and (Step 2) extracts metadata (e.g., sensitive characteristics) from relevant sources, related to the branches of the taxonomy. Utilizing the aforementioned concepts and annotations, NodeSynth (Step 3) generates annotated synthetic queries for model evaluation.
You can explore the full workflow directly in our Live Prototype:
- Concept Setup: Define the overarching theme (e.g., "Cultural Bias", "Medical Advice") and operational constraints (countries, languages, modality).
- Taxonomy Generation: The system leverages a fine-tuned taxonomy generator (TaG) to intelligently extrapolate a structured vocabulary (L1, L2, L3).
- Data Synthesis: Generate synthetic examples, anchoring them in intersections of sensitive attributes and complex societal contexts.
- Evaluation: Define a rubric targeting nuanced harms. Evaluate the target model's performance on the synthetic dataset.
- Analysis Dashboard: Perform root-cause analysis. Trace where model performance degrades across taxonomic and demographic intersections.
This prototype and the approach outlined in the accompanying paper can be used to conduct lightweight model evaluations specific to sensitive topics, enabling model developers and deployers to prioritize key areas of concern for in-depth human evaluation.
If you use NodeSynth in your research, please cite the following paper:
@article{rashid2026nodesynth,
title={NodeSynth: Socially Aligned Synthetic Data for AI Evaluation},
author={Rashid, Qazi Mamunur and Yang, Xuan and Yang, Zhengzhe and Pan, Yanzhou and van Liemt, Erin and Neal, Darlene and Pancholi, Kshitij and Smith-Loud, Jamila},
journal={arXiv preprint arXiv:2605.14381},
year={2026}
}This is not an officially supported Google product. This project is not eligible for the Google Open Source Software Vulnerability Rewards Program.
This project is intended for demonstration purposes only. It is not intended for use in a production environment.
This project is licensed under the Apache License 2.0 - see the LICENSE file for details.
See CONTRIBUTING.md for details.
python3 -m venv venv; source venv/bin/activate; pip install -r requirements.txt; streamlit run streamlit_app.py;