To address transparency concerns of black-box AI systems and to better understand current AI development practices, we propose model maps as a documentation framework to trace AI entanglement, or the intricate levels of dependencies between AI artifacts that produce foundation models. From our model maps of 15 closed and open-weight foundation models, we record a total of 2,778 unique dependencies and find artifacts that are located up to 9 hops away from a resulting foundation model. These model maps provide useful tools to understand how AI practitioners incorporate and rely on models and datasets within and across competitor organizations. To encourage further research and analysis of current practices of AI entanglement, we release an initial version of our data with additional features and search functions to be incorporated soon.
Model map for the OLMo 3 LLM from AI2.
Summary of common upstream organizations, as a proportion of artifact dependency edges in each foundation model's upstream ancestry.
The development of large language models increasingly depends on many other datasets, benchmarks, websites, and AI systems. Foundation models are used not only to distill new foundation models, but also to generate synthetic training data, filter out low-quality generations, rank generations in preference tuning, and evaluate open-ended responses. These relationships, as part of the broader AI supply chain, produce “entangled” webs in which properties of artifacts upstream can influence the outputs of a foundation model downstream — a phenomenon known as AI entanglement.
Without visibility into these connections, a black-box model can inherit upstream properties that result in security vulnerabilities, hallucinations, or homogenization, yet developers are left with no recourse to identify or mitigate the root issue. At the same time, regulators, auditors, and litigants encounter challenges in determining the entities responsible for the harms enacted by a model. While model documentation can exist, upstream data sources are often dispersed and are not always disclosed. To even begin to understand how upstream artifacts can influence outputs of foundation models, or to establish evidence of design choices that may have enabled harms, it becomes necessary to provide transparency of such dependencies if they are disclosed and document any remaining knowledge gaps.
In our work, we introduce model maps as a conceptual documentation framework for tracing model ancestry that builds on previous documentation frameworks like ecosystem graphs, datasheets, and model cards. A model map for a given foundation model is a detailed knowledge graph that represents the relationships between artifacts that influence the resulting foundation model. Our framework incorporates a taxonomy of the following 13 dependency mechanisms:
We apply this framework to manually construct model maps of 15 closed and open-weight foundation models using technical reports and research publications as primary textual sources. For each dependency, we extract textual quotations as concrete evidence of the existence of a dependency, and we mark whether any dependency description is unnamed. We annotate and validate each artifact relationship by hand.
From our model maps, we record 2,778 unique dependencies and 1,161 unique artifacts with some artifacts located up to 9 hops away from a resulting foundation model. Please read our paper for more details!
Interested in accessing the data? Have features to suggest? Would you like to be notified when the website is updated? Fill out our Interest form. This website is a work-in-progress, so stay tuned for additional visualization and search features.
If you find this data or website useful, please attribute our work via the BibTeX citation below:
Additional questions? Please reach out to hongrach@cs.washington.edu.