Resources
All resources are released under the Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International licence. Please let us know when and how you use these resources and feel free to adapt them to your own use.
Workshop worksheets
Datasets for Misbehaving Machines Worksheets
These three worksheets can be used to collaboratively create a dataset for a misbehaving machine. They can also be used to think about our relationships with LLMs. Through three worksheets participants consider what kinds of questions we ask LLMs, what kinds of answers from an LLM could be considered machine 'misbehaviour' and explore ways in which to modify the behaviour of LLMs.
You can use this worksheet to help develop your own data sets and explore the possibilities in your own way.
Worksheet 1 Questions
Using this worksheet you can explore what types of questions we might ask Large Language Models (LLMs) like ChatGPT. Thinking about and ranking these questions across the directions of this quadrant chart is a useful way to think about both the limits and potential of chat-based LLMs.Write down questions on post-it notes and place them where you think they belong on this chart.
Worksheet 2 Question Answer Pairs
Large Language Models (LLMs) like ChatGPT can get things wrong and give an incorrect answer. Through this worksheet you can explore what types of answers might be examples of machine ‘misbehaviour’.Write down answers to questions from Worksheet 1 and rank the answer wherever you think it belongs between ‘most correct & useful answer’ and ‘most uncertain or misbehaving answer’. You can be playful with how you answer the questions. Include the question with your answer so that you have a question and answer pair.
Worksheet 3 System Prompts
Finally, you can explore changing how a Large Language Model (LLM) like ChatGPT answers your questions by prompting it to answer them in different ways. For example: “Answer with references” might be a prompt in the “behaving” category. “Answer as if you are Donald Trump” might be in the “misbehaving” category.Write down your own prompt to change the behaviour of an LLM. Experiment with giving an LLM the prompt and then asking it some of the questions from Worksheet 1. Place the prompt where you think it belongs on the chart between causing the LLM to “behave” or “misbehave”.
Large Language Model (LLM) Orientation Worksheet
This worksheet guides participants to write a list of characteristics of a responsible use of AI like Large Language Models in libraries. It can easily be repurposed for different disciplines or contexts.
Datasets & code
We will be making our datasets and code available shortly.