Jay Margalus
◆ Agents · For faculty

Grade the system, not the output.

The student's agent, the rules it runs under, and the log of what it did on real inputs are the deliverable. Here's how I assign and grade the five projects in a term, and what you can borrow.

Five rules

  1. The agent and its rules are the artifact. Plus the log of what it did on real inputs. The polished output is evidence about the system, not the deliverable.
  2. Your own agent does the first read. Mine checks the output and the skill that produced it, and flags a skill that's generic or borrowed. Then I read.
  3. Grade on specifications, not points. Satisfactory or needs revision against a checklist, with a couple of tokens for late or redo. Bundles of satisfactory work map to letter grades.
  4. Make them do it on something real. One tedious job at one real organization, with a named person who owns the process. Friends and roommates don't count.
  5. Two numbers out loud every week. Conversations with real people so far, and touch points running on real inputs. Everything else is commentary.

One term

  1. Weeks 1–4Project 1 and Project 2. Every student has a folder, a rules file, and one hook.
  2. By midtermProject 3 running five days on a real person's inputs. The two numbers start.
  3. Second halfProject 4 as the real deployment, with a gate the process owner controls.
  4. OptionalProject 5 as a mastery track for the ones who are hooked.
  5. Second time throughA transfer test instead of a sixth project. Hand them a system they didn't build with a few things wrong on purpose: a stale input, a rule that's only a request, a gate the agent can edit. They make it safe and explain it before they may extend it. It tests whether they learned principles or just my five projects.

Per project

  1. № 01

    A folder that knows you 90 minutes

    Assign
    This project on a folder from your field with three planted errors you keep a key for. A clinic's intake forms, a case file, a lab notebook. Thirty files is plenty.
    Grade
    What they caught, what the agent caught, what both missed, with the transcript. Not the summary. The summary is the agent's. The catch list is theirs.
    Watch for
    Students who accept the first answer. The one who argues with it has learned the course.
  2. № 02

    Teach it your rules 2 hours

    Assign
    A rules file, one skill, one hook, and the two-transcript comparison. The comparison is what teaches.
    Grade
    The rules file and the skill for specificity. “Be accurate” is a zero. “Dollar amounts come from invoices, never emails” is the course. Have your own agent flag anything generic or borrowed.
    Say out loud
    A rule is a request and a hook is a fence. Ask which three of their rules should be fences. That's the exam.
  3. № 03

    It brings it up first 2 to 3 hours

    Assign
    One scheduled job on real inputs from a real person outside the class, running five working days, with the manifest and ATTENTION.md turned in.
    Grade
    Five days of output next to what the process owner did with it. One honest “it got Tuesday wrong and here's why” beats five clean briefs.
    Ask weekly
    Two numbers out loud: conversations with real people, and touch points running on real inputs. Zero is fine to say and not fine to hide.
  4. № 04

    Something happens, it reacts 3 hours

    Assign
    One event-driven touch point for the real organization from Project 3, with a gate the process owner controls, not the student.
    Grade
    The approval log, the injection test, and the self-approval test. Refusing a planted instruction, and showing why it couldn't approve itself, beats a nicer reply.
    Warn them
    The first time it does something “obviously fine” without asking is the day to add a fence, not the day to be impressed.
  5. № 05

    Your own small model 3 hours

    Assign
    As a mastery option, not a requirement. It's where the student stops using someone else's system and starts owning the trade-offs.
    Grade
    The routing log, the twenty-prompt sheet, and one paragraph on “what did you keep local and why.” The reasoning about the two kinds of wrong matters more than the number.
    Equipment
    A 16 GB laptop runs a 12B model. One old Mac with 32 GB becomes the department's always-on box for Projects 3 to 5.

The alumni workshop is the same scaffold compressed into two days, with handouts you can lift.