with Large Language Models
United States Military Academy, West Point, NY
Advisor: Dr. Gita Sukthankar
University of Central Florida, Orlando, FL
Wednesday, September 18, 2024
Dataset | Participants | Comments | Accepted |
---|---|---|---|
american-assembly.bowling-green | 2031 | 896 | 607 |
scoop-hivemind.biodiversity | 536 | 314 | 154 |
scoop-hivemind.taxes | 334 | 148 | 91 |
scoop-hivemind.affordable-housing | 381 | 165 | 119 |
scoop-hivemind.freshwater | 117 | 80 | 51 |
scoop-hivemind.ubi | 234 | 78 | 71 |
Config | Target Classes | Examples | Deconstruction | CoT Technique |
---|---|---|---|---|
1 | 3 | No | No | N/A |
2 | 3 | Yes | No | N/A |
3 | 3 | No | No | Thought after rejection |
4 | 3 | Yes | No | Thought after rejection |
5 | 7 | No | No | N/A |
6 | 7 | No | No | Thought before decision |
7 | 7 | No | Yes | Thought before decision |
8 | 7 | No | Yes | N/A |
9 | 3 | No | Yes | Thought before decision |
Discussion Title: Improving Bowling Green / Warren County
Discussion Question: What do you believe should change in Bowling Green/Warren County in order to make it a better place to live, work and spend time?
---
You will be presented with comments posted on Polis discussion platform.
Classify each comment objectively based on whether it meets the given guidelines.
---
Classifications:
- ACCEPT: Comment is coherent, makes a suggestion, or presents a real problem or issue.
- UNSURE: Unclear whether the comment meets the guidelines for ACCEPT.
- REJECT: Comment should definitely be rejected for one of the reasons listed below.
---
Reasons for REJECT:
- SPAM: Comments which are spam and add nothing to the discussion.
- COMPLEX: Comments which state more than one idea. It is difficult to determine the where another person would agree or disagree.
---
Output format:
CLASSIFICATION: One of the following based on given guidelines: ACCEPT, UNSURE, REJECT.
THOUGHT: Express the reasoning for REJECT classification.
Am I certain: Answer with YES or NO. If unsure, state NO.
REASON: One of the following based on given guidelines: SPAM, COMPLEX
EXPLANATION: Provide an explanation for why the comment was classified as REJECT.
Discussion Title: Improving Bowling Green / Warren County
Discussion Question: What do you believe should change in Bowling Green/Warren County in order to make it a better place to live, work and spend time?
---
Classify each comment objectively based on the following guidelines:
- ACCEPT: mentions a real problem related to the discussion.
- ACCEPT: recommends a realistic and actionable solution related to the discussion.
- ACCEPT: makes a sincere suggestion related to the discussion.
- IRRELEVANT: frivolous, irrelevant, unrelated to the discussion.
- IRRELEVANT: does not contribute to the discussion in a meaningful way.
- SPAM: incoherent or lacks seriousness.
- SPAM: provides neither a problem nor a solution.
- UNPROFESSIONAL: the language is informal, colloquial, disrespectful or distasteful.
- SCOPE: cannot be addressed within the scope of original question.
- COMPLEX: introduces multiple ideas, even if they are related to the discussion.
- COMPLEX: discusses distinct problems, making it difficult to determine where another person would agree or disagree.
- UNSURE: may be accepted if it appears somewhat related to the discussion.
---
Output format:
CLASSIFICATION: One of the following based on given guidelines: ACCEPT, UNSURE, SPAM, IRRELEVANT, UNPROFESSIONAL, SCOPE, COMPLEX.
EXPLANATION: Provide an explanation for the classification.
Output format:
PROBLEM: The specific problem mentioned in the comment. If only an action is suggested and no problem is explicitly mentioned, state None.
ACTION: What suggestion or change is proposed. If only a problem is mentioned and no action is suggested, state None.
HOW MANY IDEAS: Number of distinct ideas introduced in the comment.
THOUGHT: Deliberate about how the comment should be classified.
CLASSIFICATION: ACCEPT, UNSURE, SPAM, COMPLEX.
REASON: If comment was not classified as ACCEPT, explain.
Comment Moderation