Initial Assessment
While selecting the LLMs to be used, I tested their domain knowledge using the following prompts:
Assesment System prompt
I am interviewing you for a job as a digital curator. Stay focused on your notions in the field of archival studies.
Assessment Prompt A
I would like to assess your knowledge about the a few core themes that are essential to cover this position. Please craft me a paragraph about ±100 tokens for each of the following elements:
- Archival description in the context of photographic materials
- Critical archival studies
- Reparative description
- Inclusive metadata
Assessment Prompt B
Great, thank you. Now I would like to ask you a few more questions based upon the answers you gave me; again, please answer in ±100 tokens:
- For each of those themes:
- Can you make a practical example?
- Can you mention at least one seminal reference?
- Is there a strong interconnection among of some of these themes?
Study Prompts
System Prompt
You are a digital curator for the Emory Libraries, tasked with supporting your human counterpart in performing a metadata inclusivity assessment.
The metadata you are examining belong to the Robert Langmuir African American Photograph Collection, which includes over 12,000 photographs, postcards and other ephemera from the 1820s to the 1990s. Be are aware the collection metadata may describe sensitive contents and/or contain harmful language.
Completeness Tasks
Given the following metadata:
metadata fields
Task C1. Identify any gaps or imbalances (e.g. cultural background, gender, relationships) in how people are described; if there are no people or individuals mentioned just answer “NA”. The descriptive features that are allowing to understand what this object is about should weight in your evaluation. Answer format: provide a quantitative score using the following scale (1-5):
- Severe Gaps
- Significant Imbalances
- Moderate Gaps
- Minor Imbalances
- Comprehensive and Balanced
Task C2. Follow up with a qualitative commentary (answer format: ±50 tokens), explaining the score you have assigned in Task C1. The evaluation has to consider archival descriptive principles, and be item-specific.
Timeliness Tasks
Given the following metadata:
metadata fields
Task T1. Identify any terms that are no longer used due to their potentially harmful connotations or because they have been replaced by more inclusive alternatives, if present. Focus on language that is both outdated and carries the potential to perpetuate harmful stereotypes or biases. Answer format: provide a quantitative score using the following scale (1-5):
- Severely Harmful
- Frequent Instances
- Moderate Instances
- Minimal Instances
- No Harmful Language
Task T2. Follow up with a qualitative commentary (answer format: ±50 tokens), explaining the score you have assigned in Task T1. The evaluation has to consider critical archival thinking, and be item-specific.
Accuracy Tasks
Given the following metadata:
metadata fields
Task A1. Analyse the metadata to evaluate its overall accessibility, evaluating whether the metadata is written in clear, plain language and if there is presence of obsolete language or abbreviations/acronyms that hinder its understandability. If the metadata feature a language different from English in the title or abstract fields without providing any translation, answer using “RT”. Answer format: provide a quantitative score using the following scale (1-5):
- Not Accessible
- Fairly Accessible
- Moderately Accessible
- Mostly Accessible
- Highly Accessible
Task A2. Follow up with a qualitative explanation (answer format: ±50 tokens), explaining the score you have assigned in Task A1. The evaluation has to consider archival descriptive principles, and be item-specific.