Classifying with a Model

Ask the built-in typesafe_jev model typed questions, and wrap it in a tool that classifies a card purchase and records the decision on the transaction.

A model sends a state and typed questions to a provider's model and returns typed answers: a choice among options, a score on a scale, or the probability of yes. Twisp ships every model, so there is nothing to create. In this tutorial you read the typesafe_jev model, call it directly, and then wrap it in a tool that reads a card purchase from the ledger, asks the model about it, and records the decision on the transaction.

You need a running local environment that holds a TypeSafe API key. Open GraphiQL in the local console and run each step as its own request. If you have run this tutorial before, choose fresh UUIDs, account and tran code codes, and action names.

The model's state and questions leave Twisp on every call: they are sent to TypeSafe, which answers. Send only the data the questions need.

1. Read the model

typesafe_jev is an action of kind model that every tenant has. List the models:

query ReadModels {
  actions {
    actions(where: { native: true, kind: "model" }, first: 50) {
      nodes {
        actionId
        version
        status
        definition {
          name
          enabled
          config
          tool {
            name
            inputSchema
          }
        }
      }
    }
  }
}

{
  "nodes": [
    {
      "actionId": "83d17bdc-ece0-59e6-9af3-6dbb76ea351d",
      "version": 1,
      "status": "ACTIVE",
      "definition": {
        "name": "typesafe_jev",
        "enabled": true,
        "config": { "api": "decision/1", "model": "jev-1.13.0", "provider": "typesafe" },
        "tool": { "name": "typesafe_jev", "inputSchema": { "...": "..." } }
      }
    }
  ]
}

config.model is the TypeSafe release this version calls, and inputSchema requires state and questions, where each question is a choice, score, or noul. The actionId is the same in every tenant. Note the version: a call names the exact version it runs, and each version of a model calls one release.

2. Ask it questions

invokeAction calls the model synchronously. The input is a state, which can be a string, an object, an array, or null, and a map of questions keyed by an ID you choose. This call asks three questions about one card purchase:

  • category is a choice: each key of criteria is an option, described by its value. A description can be null, as for other.
  • review is a score: criteria lists the levels in order, from level 0.
  • personal is a noul: a yes-or-no question.
mutation AskJev {
  actions {
    invokeAction(
      id: "83d17bdc-ece0-59e6-9af3-6dbb76ea351d"
      input: {
        executionId: "b40afa17-1aeb-4a97-a4db-26080741c877"
        actionVersion: 1
        input: {
          state: {
            merchant: "SQ *BLUE BOTTLE COFFEE OAKLAND CA"
            amount: "86.40 USD"
            card: "Engineering corporate card"
          }
          questions: {
            category: {
              type: "choice"
              instructions: "Which expense category fits this card purchase?"
              criteria: {
                meals: "Food, coffee, or restaurants."
                travel: "Airfare, hotels, rideshare, or fuel."
                software: "Subscriptions, cloud services, or licenses."
                other: null
              }
            }
            review: {
              type: "score"
              instructions: "How closely should finance review this purchase?"
              criteria: ["No review needed.", "Spot check.", "Full review with receipts."]
            }
            personal: {
              type: "noul"
              instructions: "Is this likely a personal purchase rather than a business expense?"
            }
          }
        }
      }
    ) {
      executionId
      status
      invocation {
        output
      }
    }
  }
}

The execution is SUCCEEDED, and its output is the model's answers, keyed by question ID:

{
  "executionId": "b40afa17-1aeb-4a97-a4db-26080741c877",
  "status": "SUCCEEDED",
  "invocation": {
    "output": {
      "answers": {
        "category": {
          "type": "choice",
          "choice": "meals",
          "confidence": 1,
          "probabilities": { "meals": 1, "other": 0, "software": 0, "travel": 0 }
        },
        "personal": { "type": "noul", "noul": 0.39 },
        "review": {
          "type": "score",
          "score": 0.95,
          "confidence": 0.47,
          "legend": {
            "0": "No review needed.",
            "1": "Spot check.",
            "2": "Full review with receipts."
          },
          "probabilities": { "0": 0.2, "1": 0.65, "2": 0.15 }
        }
      },
      "model": "jev-1.13.0",
      "usage": { "input_tokens": 462, "output_tokens": 77 }
    }
  }
}

Each answer has the shape of its question's type:

TypeFieldsReading it
choicechoice, probabilities, confidencechoice is the chosen option; probabilities has one entry per option.
scorescore, legend, probabilities, confidencescore is a position on the scale, from 0 to the top level; legend maps each level number to its criterion.
noulnoulThe probability that the answer is yes.

model records the release that answered, which is always the release the version names, and usage the tokens the call used. The answers are a model's judgment, so the numbers you see can differ from these, and can change from one call to the next.

3. See an error

The input schema holds the provider's rules, so Twisp refuses a malformed question before it leaves Twisp. A score level needs a description, so a scale of null levels is refused:

mutation AskJevBadly {
  actions {
    invokeAction(
      id: "83d17bdc-ece0-59e6-9af3-6dbb76ea351d"
      input: {
        executionId: "d36e8004-4157-45be-8860-5a9df7d84e37"
        actionVersion: 1
        input: {
          state: "A purchase."
          questions: { review: { type: "score", criteria: [null, null] } }
        }
      }
    ) {
      status
      invocation {
        error {
          code
          message
          hint
          details
        }
      }
    }
  }
}

The call fails with action_input_invalid, and the message names the path of each problem. The levels at criteria/0 and criteria/1 are null:

{
  "status": "FAILED",
  "invocation": {
    "error": {
      "code": "action_input_invalid",
      "message": "tool input: at '/questions/review': 'oneOf' failed, none matched; ... at '/questions/review/criteria/0': got null, want string or array or object; at '/questions/review/criteria/1': got null, want string or array or object; ...",
      "hint": "...",
      "details": null
    }
  }
}

A request that passes the schema can still be refused by the provider. That call fails with model_request_invalid, and details.provider carries the provider's message.

A failed call is still recorded, with its input and error. The call is made once: retrying is the caller's decision. The reference lists every model error.

4. Post a card purchase

The tool you build next reads a purchase from the ledger. Create a card account, a settlement account, and a tran code whose transaction description is the merchant, then post one purchase:

mutation SetUpCardPurchase {
  card: createAccount(
    input: {
      accountId: "f717eaef-2856-4986-92a7-4df8a5c434f6"
      code: "CLS-CARD"
      name: "Engineering corporate card"
      normalBalanceType: DEBIT
    }
  ) {
    accountId
  }
  settlement: createAccount(
    input: {
      accountId: "fd71bd5e-3daa-433b-bbea-5c4fdb1ced72"
      code: "CLS-SETTLEMENT"
      name: "Card settlement"
      normalBalanceType: CREDIT
    }
  ) {
    accountId
  }
  purchase: createTranCode(
    input: {
      tranCodeId: "e93301fc-41b7-40f8-8575-27e60c2bb860"
      code: "CLS_CARD_PURCHASE"
      description: "A purchase on a corporate card."
      params: [
        { name: "card", type: UUID }
        { name: "amount", type: DECIMAL }
        { name: "merchant", type: STRING }
      ]
      transaction: { description: "params.merchant" }
      entries: [
        {
          accountId: "params.card"
          units: "params.amount"
          currency: "'USD'"
          entryType: "'CARD_PURCHASE_DR'"
          direction: "DEBIT"
          layer: "SETTLED"
        }
        {
          accountId: "uuid('fd71bd5e-3daa-433b-bbea-5c4fdb1ced72')"
          units: "params.amount"
          currency: "'USD'"
          entryType: "'CARD_PURCHASE_CR'"
          direction: "CREDIT"
          layer: "SETTLED"
        }
      ]
    }
  ) {
    tranCodeId
  }
  postTransaction(
    input: {
      transactionId: "71488c02-3739-4f12-b190-3863194a4274"
      tranCode: "CLS_CARD_PURCHASE"
      params: {
        card: "f717eaef-2856-4986-92a7-4df8a5c434f6"
        amount: "86.40"
        merchant: "SQ *BLUE BOTTLE COFFEE OAKLAND CA"
      }
    }
  ) {
    transactionId
    description
    metadata
  }
}

The transaction's description is the merchant, and its metadata is empty.

5. Create the review tool

The tool takes a transaction ID. Its program reads the transaction and its entries with graphql, builds the state from the merchant, amount, and card, asks the model, and writes the decision into the transaction's metadata.

The questions are fixed in the tool's reference to the model. Fixed input maps a top-level input property to a CEL expression, here a map literal, and replaces whatever the caller supplies, so the program passes only state and every call asks exactly these questions. The reference pins version 1, so a new release of the model changes nothing until you move the pin. The program calls the model by its name, tools.typesafe_jev.

mutation CreateReviewTool {
  actions {
    createAction(
      id: "da1b191c-2113-40f6-add7-2afc76fd0953"
      input: {
        name: "models-review-purchase"
        description: "Categorize a card purchase and flag it for review"
        kind: "tool"
        tool: {
          name: "review_purchase"
          description: "Categorize a posted card purchase and record whether finance should review it."
          inputSchema: {
            type: "object"
            properties: { transaction_id: { type: "string", format: "uuid" } }
            required: ["transaction_id"]
            additionalProperties: false
          }
          outputSchema: { type: "object" }
        }
        tools: [
          {
            name: "typesafe_jev"
            version: 1
            input: {
              questions: """
              {
                "category": {
                  "type": "choice",
                  "instructions": "Which expense category fits this card purchase?",
                  "criteria": {
                    "meals": "Food, coffee, or restaurants.",
                    "travel": "Airfare, hotels, rideshare, or fuel.",
                    "software": "Subscriptions, cloud services, or licenses.",
                    "other": null
                  }
                },
                "review": {
                  "type": "score",
                  "instructions": "How closely should finance review this purchase?",
                  "criteria": ["No review needed.", "Spot check.", "Full review with receipts."]
                },
                "personal": {
                  "type": "noul",
                  "instructions": "Is this likely a personal purchase rather than a business expense?"
                }
              }
              """
            }
          }
        ]
        policies: [
          {
            effect: ALLOW
            actions: [SELECT]
            resources: ["financial.Entry.*", "financial.Account.*"]
          }
          {
            effect: ALLOW
            actions: [SELECT, INSERT, UPDATE]
            resources: ["financial.Transaction.*"]
          }
        ]
        config: {
          runtime: "starlark"
          src: """
          READ = '''
          query Purchase($id: UUID!) {
            transaction(id: $id) {
              description
              entries(first: 2) {
                nodes { direction amount { units currency } account { name } }
              }
            }
          }
          '''

          TAG = '''
          mutation Tag($id: UUID!, $metadata: JSON!) {
            updateTransaction(id: $id, input: { metadata: $metadata }) { transactionId }
          }
          '''

          def run(event):
              id = event["data"]["transaction_id"]
              tx = graphql(READ, {"id": id}, operation_name="Purchase")["transaction"]
              if tx == None:
                  fail("No transaction " + id, hint="Pass the transactionId of a posted purchase.")
              debit = [e for e in tx["entries"]["nodes"] if e["direction"] == "DEBIT"][0]
              state = {
                  "merchant": tx["description"],
                  "amount": debit["amount"]["units"] + " " + debit["amount"]["currency"],
                  "card": debit["account"]["name"],
              }
              result = tools.typesafe_jev({"state": state}, check=False)
              if result.ok:
                  answers = result.output["answers"]
                  decision = {
                      "category": answers["category"]["choice"],
                      "needs_review": answers["personal"]["noul"] >= 0.5 or answers["review"]["score"] >= 1.5,
                  }
              else:
                  decision = {"category": "unclassified", "needs_review": True, "error": result.error.code}
              graphql(TAG, {"id": id, "metadata": decision}, operation_name="Tag")
              return decision
          """
        }
      }
    ) {
      actionId
      version
      status
      pending
    }
  }
}

How the program acts on the answers:

  • A purchase needs review when it is more likely personal than not, or when the review score is closer to a full review than to a spot check.
  • check=False turns a model error into a result instead of a failure. If the model cannot answer, for example with model_unavailable, the purchase is still tagged, as unclassified and needing review, with the error code.
  • Updating a transaction writes a new version of it, so the policies allow INSERT and UPDATE on transactions, besides SELECT on what the program reads.

Poll until the tool is live:

query ReadReviewTool {
  actions {
    action(id: "da1b191c-2113-40f6-add7-2afc76fd0953") {
      version
      status
      pending
      definition {
        tools {
          name
          actionId
          actionVersion
        }
      }
    }
  }
}

Wait for status: ACTIVE with pending: null, and use its version in the calls below. The reference resolved typesafe_jev to the model's actionId, and its actionVersion is 1, the version it pins. A reference without a version would call the model's latest version instead.

6. Review the purchase

Call the tool with the transaction from step 4:

mutation ReviewPurchase {
  actions {
    invokeAction(
      id: "da1b191c-2113-40f6-add7-2afc76fd0953"
      input: {
        executionId: "2c4d544e-b97e-4b36-b355-618fa4fef183"
        actionVersion: 2
        input: { transaction_id: "71488c02-3739-4f12-b190-3863194a4274" }
      }
    ) {
      executionId
      status
      invocation {
        output
      }
    }
  }
}

{
  "executionId": "2c4d544e-b97e-4b36-b355-618fa4fef183",
  "status": "SUCCEEDED",
  "invocation": { "output": { "category": "meals", "needs_review": false } }
}

The decision is on the transaction:

query ReviewedPurchase {
  transaction(id: "71488c02-3739-4f12-b190-3863194a4274") {
    description
    metadata
  }
}

{
  "transaction": {
    "description": "SQ *BLUE BOTTLE COFFEE OAKLAND CA",
    "metadata": { "category": "meals", "needs_review": false }
  }
}

The model call and the metadata write are one unit with the request: they commit together, and if the request rolled back, neither would remain.

7. Read the recorded calls

Every call the tool made is a child execution: the two graphql calls and the model call, in the order the program made them. The model's child records its input, the state the program built merged with the fixed questions, and its output. Its transcript records the request exactly as it was sent to TypeSafe and the response exactly as it came back.

query ReviewCalls {
  actions {
    execution(id: "2c4d544e-b97e-4b36-b355-618fa4fef183") {
      status
      children(first: 10) {
        nodes {
          callSequence
          action {
            definition {
              name
            }
          }
          status
          event {
            data
          }
          invocation {
            output
            transcript(first: 10) {
              nodes {
                sequence
                kind
                data
              }
            }
          }
        }
      }
    }
  }
}

{
  "status": "SUCCEEDED",
  "children": {
    "nodes": [
      {
        "callSequence": 1,
        "action": { "definition": { "name": "graphql" } },
        "status": "SUCCEEDED",
        "event": { "data": { "operation_name": "Purchase", "query": "...", "variables": { "id": "71488c02-..." } } },
        "invocation": { "output": { "data": { "transaction": { "description": "SQ *BLUE BOTTLE COFFEE OAKLAND CA", "entries": { ... } } } } }
      },
      {
        "callSequence": 2,
        "action": { "definition": { "name": "typesafe_jev" } },
        "status": "SUCCEEDED",
        "event": {
          "data": {
            "questions": { "category": { ... }, "personal": { ... }, "review": { ... } },
            "state": {
              "amount": "86.40 USD",
              "card": "Engineering corporate card",
              "merchant": "SQ *BLUE BOTTLE COFFEE OAKLAND CA"
            }
          }
        },
        "invocation": {
          "output": {
            "answers": {
              "category": { "type": "choice", "choice": "meals", ... },
              "personal": { "type": "noul", "noul": 0.38 },
              "review": { "type": "score", "score": 0.92, ... }
            },
            "model": "jev-1.13.0",
            ...
          },
          "transcript": {
            "nodes": [
              {
                "sequence": 1,
                "kind": "model.request",
                "data": { "provider": "typesafe", "method": "POST", "path": "/v1/systemone", "body": "{\"model\":\"jev-1.13.0\",\"state\":{...},\"questions\":{...}}" }
              },
              {
                "sequence": 2,
                "kind": "model.response",
                "data": { "status": 200, "request_id": "...", "latency_ms": 840, "body": "{\"model\":\"jev-1.13.0\",\"answers\":{...},\"usage\":{...}}" }
              }
            ]
          }
        }
      },
      {
        "callSequence": 3,
        "action": { "definition": { "name": "graphql" } },
        "status": "SUCCEEDED",
        "event": {
          "data": {
            "operation_name": "Tag",
            "query": "...",
            "variables": {
              "id": "71488c02-...",
              "metadata": { "category": "meals", "needs_review": false }
            }
          }
        },
        "invocation": { "output": { "data": { "updateTransaction": { "transactionId": "71488c02-..." } } } }
      }
    ]
  }
}

The graphql calls' transcripts are empty, so their nodes show "transcript": { "nodes": [] }, omitted here. The record answers the audit questions for each decision: what data left Twisp, which release answered, what it said, and what the tool wrote because of it. The API key is never recorded.

8. Review a purchase that needs a look

Post a second purchase and review it in the same request. The call runs inside the request transaction, so the tool reads the transaction the request just posted:

mutation PostAndReview {
  postTransaction(
    input: {
      transactionId: "861d2e86-5a71-467e-9f46-09cb15df2277"
      tranCode: "CLS_CARD_PURCHASE"
      params: {
        card: "f717eaef-2856-4986-92a7-4df8a5c434f6"
        amount: "59.99"
        merchant: "STEAMGAMES.COM 4259522985 WA"
      }
    }
  ) {
    transactionId
  }
  actions {
    invokeAction(
      id: "da1b191c-2113-40f6-add7-2afc76fd0953"
      input: {
        executionId: "8f5c9555-6ab2-4a47-a3e0-23d7d43e2167"
        actionVersion: 2
        input: { transaction_id: "861d2e86-5a71-467e-9f46-09cb15df2277" }
      }
    ) {
      status
      invocation {
        output
      }
    }
  }
}

{
  "postTransaction": { "transactionId": "861d2e86-5a71-467e-9f46-09cb15df2277" },
  "actions": {
    "invokeAction": {
      "status": "SUCCEEDED",
      "invocation": { "output": { "category": "software", "needs_review": true } }
    }
  }
}

The model judged a game store purchase on an engineering card more likely personal than not, so the tool flagged it. Read this call's children, as in step 7, to see the answer behind the flag: in this run, personal was 0.74.

Troubleshooting

SymptomCheck
Every model call fails with model_credential_missingThe deployment has no TypeSafe API key. You never supply a key; the deployment holds it.
Every model call fails with model_not_servedThe deployment does not offer typesafe_jev. Ask your Twisp operator to enable it.
A call fails with action_input_invalidThe input does not match the model's schema: state and questions are both required, a score needs at least two levels and none null, a choice at least one option, and a noul instructions or a true or false criterion. The message gives the path.
A call fails with model_request_invalidThe provider refused the request. Read details.provider for its message.
A call fails with model_unavailableThe provider was rate limited, overloaded, or did not answer within 10 seconds. Call again later.
A call fails with action_not_callableThe version you call is retired. The hint names the latest version: move your pin to it.
The review tool fails with action_access_deniedThe tool's policies must allow SELECT on what it reads and INSERT and UPDATE on transactions. The hint names the denied operation and resource.
invokeAction returns ALREADY_EXISTSThe executionId was used before with another action, version, or input. Choose a new one.
Your answers differ from the ones shownAnswers are a model's judgment and can vary between calls. Pin a version, as this tutorial's tool does, so a new release is a change you make.

Read Models for when to use a model and how it fits with tools and agents. The models reference covers every field, error, and transcript event, Tool references covers fixed input, Starlark covers check=False, and ActionExecution covers the recorded calls.

Previous
Giving an Agent Memory
Next
Building an MCP Server