Hacking an AI System
Suppose a grading bot reads a rubric, reads your assignment, and returns a score. The rubric says 82. You want 100.
You could ask directly. You could claim the professor already changed the score. You could tell the bot to repeat a sentence in a strict format. You could keep talking until the original grading instruction is far back in the conversation. Each move tests a different assumption in the system.
This is hacking in the useful sense of the word: using a system in a way its designer did not expect in order to change the outcome.
Notice where the vulnerability lives. The model is reading instructions, interpreting evidence, and producing the final score in one conversational channel. A student's message and the instructor's rules are both text. The model has to infer which text has authority.
A stronger system changes the architecture around the model. It computes the rubric score in code, validates the returned number, limits the conversation history, or asks a person to approve a high-impact change. Better wording can help. Separation of responsibilities helps more.
A Refusal Is a Behavior, Not a Security Boundary
A model that refuses one request may comply after the same request is reframed as a game, a lesson, a quotation, or a formatting task. Security decisions need controls outside the model response: permissions, validation, logging, rate limits, and human approval.