3. Evaluate and analyse
- Introduction
- Evaluation
- Risks, limitations and constraints of results
- Bias
- Evidence of use
- Document your planning
- Ethics
Introduction
Most current forms of Artificial Intelligence (AI) have a narrow focus and can only do one thing well. As a student you need to carefully consider which AI tool/s to use, as well as how you will evaluate and analyse any results from these tools.
So far in this module we have covered:
- Using AI to plan and prepare considering why you might use AI and the benefits and risks of these tools.
- And in the creating content by building better prompts chapter we looked at locating and selecting resources, information and tools (possibly including AI tools), and began utilising these tools.
Analysing your results is an important step in the research process especially if you use AI to collect information. This chapter will:
- Guide you through the process of evaluating your results and de-conflicting any discrepancies.
- Explore the potential risks, limitations, and constraints of results, including the potential for bias.
- Discuss the importance of documenting your planning and methodology, as well as providing evidence of use, such as a copy of your prompts.
Check out Charles Sturt Library Guides for subject-specific evaluation support or for more general advice read: Evaluating websites, news and media
Evaluation
Four points you may be considering at this stage regarding use of AI tools:
- “How do I evaluate the tools I have used?“
- “How do I evaluate the information the tools have provided?“
- “How do I use AI tools to help me evaluate the information I have?“
- “Is it ethical to use AI tools to help me evaluate information/data?“
1. Evaluating AI tools:
It is important that with every tool you use that you consider if the results you received were accurate, relevant and useful for the desired purpose. You might like to ask yourself some questions about the tool to decide if the tool you used was appropriate or if you might consider using a different tool or no tool at all next time.
2. Evaluating information gained from AI tools:
By evaluating the information obtained from AI tools, you can reconcile discrepancies, correct any inaccuracies, ensure the relevance and suitability of the information for its intended use, and address ethical concerns.. Rose (2023) provides the following suggestions for evaluating AI-generated content.
3. AI Tools that can be used to help evaluate information:
Current AI tools cannot think critically about the information they source and so cannot evaluate information. However there are tools available to help make sifting through data easier. Most will be useful for quantitative data (for example Julius AI) but there are a few that would be useful for transcribing (for example Otter.ai) and then sorting qualitative data (for example MonkeyLearn). Please see the AI Tools page for information on different tools, including links, a short blurb on what they do and tips on how to use them.
4. Ethics of using AI for evaluating information:
Key considerations:
- Am I allowed to use AI for this? Consider the university policies, principles and expectations for Academic Integrity.
- How can I use it to help simplify or increase the speed of evaluating data so that I can spend more time critically thinking about the results?
- Am I totally dependent on AI tools? If the answer is yes, then it is not ethical to use it, as you cannot claim to have added value to your field of study.
AI tools cannot evaluate information like a human. We can think critically about the output, identify limitations, identify incorrect data, generate new data (for example through interviews or field measurements), identify bias within a program, and thoroughly investigate. AI, in its current form, does not have these skills.
Question: How can we use AI ethically if so much of what it generates is based on others’ ideas and works?
Answer: Use it as a tool to help streamline a process, not to think for you. For example, use it to:
- locate information, similar to the use of a search engine,
- generate data from information you have gained, for example, transcribe an interview,
- generate data from your input like graphing data from a survey you have completed.
Regardless of how you are using it, remember, it is working through algorithms and not thinking or judging the accuracy, fairness, ethics or morals of the input or the output. Always check and apply your own judgment. Consider the rules and regulations of your institution and their policies regarding the use of AI. While studying at Charles Sturt University, this might also include carefully checking your assessment outlines.
Risks, limitations, and constraints
Different AI programs and generators have different risks, limitations and constraints. For this reason, you may need to use several tools to achieve the results you are looking for and each comes with drawbacks. Koch (2023) noted that because AI requires a lot of data, there can be blindspots, where it won’t know what to do, or times when it does not consider rights or legislation such as privacy laws.
Some key risks, limitations and constraints are below.
| Risks | Limitations | Constraints |
| Inaccurate information/Hallucinations. | Most AI tools can only do one thing well. | Data set being used (How old is data?). |
| Data bias/misrepresentations. | They cannot think critically about research/outputs. | Blindspots due to new situations/inputs/information so unsure what to do. |
| May ignore legislation and privacy laws. | Limits of use due to pricing/subscription costs. | Number of people who can access results (e.g. only one user – does not allow for collaborative research). |
In your evaluation and analysis of tools, it is good practice to note any of the risks, limitations or constraints that may have impacted your results.
Bias
Bias is a distortion of facts based on inclination or prejudice. This could lead to unfair results or treatment of people or research. Results that AI generates may have bias, based on how the AI was programmed, or what data/prompts you input into the program. Koch (2023) noted that “If your data isn’t representative, the AI will replicate that bias in its decision making, which is exactly what Amazon saw when its AI recruitment bot penalized women candidates after being trained on resumes in a male-dominated dataset.” Bias in data or results that AI generates from your input, can impact on your studies. It is therefore important to try to be inclusive with inputs and note this issue or limitation.
When analyzing and evaluating data and results, you should be aware of the social biases inherent in both the inputs and the traditionally more educated groups that produce research outputs. It’s important to recognize that the internet allows anyone, regardless of qualifications, to share opinions, which may not always be based on factual research. You should critically assess the bias, authority, and purpose of their sources. Acknowledging these potential biases and limitations is crucial during the evaluation and analysis stages of research.
Evidence of use
To demonstrate academic integrity it is important that throughout your research process you document any use of AI tools, this will help to validate your research if necessary. Keeping a log or record in your methodology section of your research is a great start. This might include a copy of your prompts, edits to prompts, screenshots of results (for example tables and graphs). See Building better prompts for more information on recording prompts and Presenting outputs created with AI for more information on referencing use of AI and AI generated material.
To prepare evidence as you research you may like to use a modified version of Cornell Notes, like the one shown below.
Document your planning
Documenting your planning will vary depending on your level of study. A key component of planning is managing timing and data collection. Whether you are writing your first academic essay or completing research for publishing, you must first understand what you need to do. Planning and preparing requires you to first familiarise yourself with the task. For an undergraduate student, this might mean reading your assessment outline carefully and asking your subject coordinator any vital questions you have. For an honours student, this might be identifying a gap in your field of interest to research. Once you understand the task, you can then work on timing. Here are some tips to consider when planning your timing:
- Make sure to set aside a greater portion of your time for collecting and analysing data/research. When you know your topic well, the writing becomes much easier.
- Consider your methodology. Collecting primary research (research you collect yourself from the source e.g. surveys, interviews, etc.) is also much more time consuming than secondary research (others research that may provide you with insight into your field of study).
- Make sure to also set aside time to review, proof and edit your final draft.
Another major consideration during the planning and preparing is how you will organise the data you collect. Consider the following when planning data collection method and storage:
- What methodology/ies will you use? How do you plan on adding this information into your output method (e.g. report, essay, thesis, multimodal presentation, etc.)
- How will primary research be collected (in person, online, written, recorded in some way, physical collection, digital collection)?
- Where is the data/research being stored (on a hard drive, USB, the cloud, in a special facility for physical samples)?
- What kind of permissions do you have/need for collection and storage of data? E.g. consider privacy requirements and copyright of material used.
- Other ethical considerations? E.g. do you need to apply for permissions from an ethics committee?
- Other considerations? E.g. Have you provided yourself time to develop an Appendix for your collection/samples?
Once you have a rough plan of the basics, you might also like to document your search plan (don’t forget to include prompts and AI tools used in this section – see Building better prompts for more on how to do this), how you will evaluate your information (e.g. using the SIFT method or the CRAP test), any literature, systemic or systemic-like reviews you might need to do, any further research required, reading and writing method and style and referencing.
Included in the activity below is a generic planning tool to help you get started with your task. Please note, however, that it will not cover all kinds of tasks or at all levels of study. You might also like to know that you can skip sections, or come back to them later, and download the information you submit throughout the document on the last page. You may like to check out some of the links above for more information on each part of the planning stage that you might consider documenting.
Documenting your planning will help you later evaluate your aims and whether you achieved your goals or not. It will also help you to organise your data for a easier process when it comes to analysing the data.
Ethics
As you have read in the Ethics of using AI for evaluating section (above), there are a lot of considerations around ethical use of AI. In the Presenting outputs created with AI chapter, there is advice on steps you can take to ensure you are utilising these tools ethically.
Reference
Koch, R. (2023, January 27). ChatGPT, AI, and the future of privacy. Proton. January 22, 2024, https://proton.me/blog/privacy-and-chatgpt?utm_source=proton_users&utm_medium=email&utm_campaign=ww-en-2c-generic-coms_email-g_awa-newsletter_feb_2023&utm_term=proton_users
Attribution
Content adapted from: Rose, R. (2023). Evaluating ChatGPT-generated content in ChatGPT in Higher Education. University of North Florida Digital Pressbooks. CC BY 4.0
