Evaluate & analyse content created
Most current forms of Artificial Intelligence (AI) have a narrow focus and can only do one thing well. This means that you have to carefully consider what AI tool/s you will use and then how you will evaluate and analyse the results from the tools you choose to use. When Using AI to plan and prepare you looked at why you are using AI and considered the benefits and risks of using these tools. In the Creating content by building better prompts chapter you will have located resources, information and chosen tools (possibly including AI tools) and begun utilising these tools. Now you need to consider how you will use the results and evaluate if the tools did their job properly or think about trying a tool that can help you evaluate your results.
Analysing your results is an important step in the research process especially if you use AI to collect the information. This chapter will guide you through the process of evaluating your results and de-conflicting or resolving any discrepancies. We will then explore the risks, limitations, and constraints of your results, including the potential for bias. Finally, we will discuss the importance of documenting your planning and methodology, as well as providing evidence of use, such as a copy of your prompts.
Evaluation
The evaluation stage is where you evaluate the information you have gained as well as where you got your information. Check out the Charles Sturt Libguides for some subject specific evaluation support or more general guides like “Evaluating websites, news and media” for how to evaluate sources of information.
There are four points you may be considering at this stage regarding use of AI tools:
- “How do I evaluate the tools I have used?“
- “How do I evaluate the information the tools have provided?“
- “How do I use AI tools to help me evaluate the information I have?“
- “Is it ethical to use AI tools to help me evaluate information/data?“
1. Evaluating AI tools:
It is important that with every tool you use that you consider if the results you received were accurate, relevant and useful for the desired purpose. You might like to ask yourself some questions about the tool to decide if the tool you used was appropriate or if you might consider using a different tool or no tool at all next time.
2. Evaluating information gained from AI tools:
Through the process of evaluation of the information gained from AI tools, you can resolve conflicting results, ensuring that any inaccuracies are addressed, that the information is relevant and suits the purpose for which it was gained and that ethical considerations have been addressed. Rose (2023) provides the following suggestions for evaluating AI-generated content.
3. AI Tools that can be used to help evaluate information:
Current AI tools cannot think critically about the information they source and so cannot evaluate information for you. However there are tools available to help make sifting through the data easier so that you have more time to do the deep analysis that critical thinking about the data requires. Most will be useful for quantitative data (for example Julius AI) but there are a few that would be useful for transcribing (for example Otter.ai) and then sorting qualitative data (for example MonkeyLearn). Please see the AI Tools page for information on different tools, including links, a short blurb on what they do and tips on how to use them.
4. Ethics of using AI for evaluating information:
Ethics is a tricky subject when it comes to use of AI. In regards to using it for evaluating information. The key considerations should be:
- Am I allowed to use AI for this? Consider the university policies, principles and expectations for Academic Integrity.
- How can I use it to help simplify or increase the speed of evaluating data so that I can spend more time critically thinking about the results?
- Am I totally dependent on AI tools? If the answer is yes, then it is not ethical to use it, as you cannot claim to have added value to your field of study.
AI tools cannot evaluate information like a human. We can think critically about the output, identify limitations, identify incorrect data, generate new data (for example through interviews or field measurements) identify bias within a program, and thoroughly investigate. AI, in its current form, does not have these skills.
So how can we use it ethically if so much of what it generates is based on others ideas and works?
Basically the answer is use it as a tool to help streamline a process, not do the thinking for you. You can use it to locate information, similar to the use of a search engine. You could use it to generate data from information you have gained, for example transcribe an interview. You could use it to generate data from your own input like graphing data from a survey you have completed. Regardless of how you are using it, remember, it is working through algorithms and not thinking or judging the accuracy, fairness, ethics or morals of the input or the output. Always check and apply your own judgment. You also have to consider the rules and regulations of your institution and their policies regarding the use of AI. While studying at Charles Sturt University, this might also include carefully checking your assessment outlines. Be sure to consider all relevant factors when making decisions about using AI tools, especially in the evaluation and analysis sections of your research.
Risks, limitations, and constraints
Different AI programs and generators have different risks, limitations and constraints. For this reason you may have to use several of them to achieve the results you are looking for and each come with their own drawbacks. Koch (2023) noted that due to the fact that AI requires a lot of data, there can be blindspots, where it won’t know what to do, or times when it does not consider rights or legislation such as privacy laws.
Some key risks, limitations and constraints are below.
| Risks | Limitations | Constraints |
| Inaccurate information/Hallucinations. | Most AI tools can only do one thing well. | Data set being used (How old is data?). |
| Data bias/misrepresentations. | They cannot think critically about research/outputs. | Blindspots due to new situations/inputs/information so unsure what to do. |
| May ignore legislation and privacy laws. | Limits of use due to pricing/subscription costs. | Number of people who can access results (e.g. only one user – does not allow for collaborative research). |
In your evaluation and analysis it is good practice to note any of the risks, limitations or constraints that may have impacted your results. This is especially important if you plan on publishing your work.
Bias
Bias is a distortion of facts based on inclination or prejudice. This could lead to unfair results or treatment of people or research. Results that AI generates may have bias, based on how the AI was programmed, or what data/prompts you input into the program. Koch (2023) noted that “If your data isn’t representative, the AI will replicate that bias in its decision making, which is exactly what Amazon saw when its AI recruitment bot penalized women candidates after being trained on resumes in a male-dominated dataset.” Bias in data or results that AI generates from your input, can impact on your research. It is therefore important to try to be inclusive with inputs and note this issue or limitation in your research results. Analysing and evaluating your data and results requires you to consider the social biases of not just your inputs, but also the social groups that have traditionally had higher education and therefore, research outputs. Also consider that anyone, regardless of qualifications, can post their opinions on the internet. Some popular opinions may not be founded in factual research. Always examine the bias, authority and purpose of the source. Incorporating these limitations and potential for bias should also be a consideration in your evaluation and analysis stage.
Evidence of use
It is important that throughout your research process you document any use of AI tools, to demonstrate academic integrity and to help other researchers replicate and thereby validate your research. Keeping a log or record in your methodology section of your research is a great start. This might include a copy of your prompts, edits to prompts, screenshots of results (for example tables and graphs). See Creating content by building better prompts for more information on recording prompts and Presenting outputs created with AI for more information on referencing use of AI and AI generated material.
To prepare evidence as you research you may like to use a modified version of Cornell Notes, like the one shown below.
Document your planning
Documenting your planning will vary based on what kind of research and what level of research you are doing. A key component of planning is managing timing and data collection. For this reason it is important that these are all mapped out before you start and to do this, you need to know your task. Whether you are writing your first academic essay or completing research for publishing, you need to first understand what you need to do. Planning and preparing requires you to first familiarise yourself with the task. For an undergraduate, this might mean reading your assessment outline carefully and asking your subject coordinator any vital questions you have. For an honours student, this might be identifying a gap in your field of interest to research. For a researcher, it might be familiarising yourself with a publisher and their expectations for a major publication/work or research grants available and their requirements. Once you understand the task, you can then work on timing. Here are some tips to consider when planning your timing:
- Make sure to set aside a greater portion of your time for collecting and analysing data/research. When you know your topic well, the writing becomes much easier.
- Consider your methodology. Collecting primary research (research you collect yourself from the source e.g. surveys, interviews, etc.) is also much more time consuming than secondary research (others research that may provide you with insight into your field of study).
- Make sure to also set aside time to review, proof and edit your final draft.
Another major consideration during the planning and preparing is how you will organise the data you collect. Consider the following when planning data collection method and storage:
- What methodology/ies will you use? How do you plan on adding this information into your output method (e.g. report, essay, thesis, multimodal presentation, etc.)
- How will primary research be collected (in person, online, written, recorded in some way, physical collection, digital collection)?
- Where is the data/research being stored (on a hard drive, USB, the cloud, in a special facility for physical samples)?
- What kind of permissions do you have/need for collection and storage of data? For example consider privacy requirements and copyright of material used.
- Other ethical considerations? For example do you need to apply for permissions from an ethics committee?
- Other considerations? For example have you provided yourself time to develop an Appendix for your collection/samples?
Once you have rough plan of the basics, you might also like to document your search plan (don’t forget to include prompts and AI tools used in this section – see Creating content by building better prompts for more on how to do this), how you will evaluate your information (e.g. using the SIFT method or the CRAP test), any literature, systemic or systemic-like reviews you might need to do, any further research required, reading and writing method and style and referencing.
Included in the activity below is a generic planning tool to help you get started with your task. Please note, however, that it will not cover all kinds of tasks or at all levels of study. You might also like to know that you can skip sections, or come back to them later, and download the information you submit throughout the document on the last page. You may like to check out some of the links above for more information on each part of the planning stage that you might consider documenting.
Documenting your planning will help you later evaluate your aims and whether you achieved your goals or not. It will also help you to organise your data for a easier process when it comes to analysing the data.
Ethics
As you have read in the Ethics of using AI for evaluating section (above), there are a lot of considerations around ethical use of AI. In the Presenting outputs created with AI chapter, there is advice on steps you can take to ensure you are utilising these tools ethically.
Reference
Koch, R. (2023, January 27). ChatGPT, AI, and the future of privacy. Proton. January 22, 2024, https://proton.me/blog/privacy-and-chatgpt?utm_source=proton_users&utm_medium=email&utm_campaign=ww-en-2c-generic-coms_email-g_awa-newsletter_feb_2023&utm_term=proton_users
Attribution
Content adapted from: Rose, R. (2023). Evaluating ChatGPT-generated content in ChatGPT in Higher Education. University of North Florida Digital Pressbooks. CC BY 4.0
