Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 4 additions & 2 deletions src/usersim/engine/assets/financial_services/SCHEMA.md
Original file line number Diff line number Diff line change
Expand Up @@ -257,8 +257,10 @@ A task is a **parameterized generator**, not a fixed string. `tasks.yaml` holds
(see `dynamic.yaml` below), not scripted here. `templates`: each `id`, `tier`
(verifiable), `task_type`, `domain`, `placeholder`, `persona_tags` (see below),
`opening_user_messages` (a LIST of `{{slot}}` templates; author >=3),
`account_state` (seed state the tools mutate), `param_space` (per-instance
slot values), `gold_tool_sequence`, `ordered` (OPTIONAL, default `false`),
`account_state` (seed state the tools mutate; hold the account's money under
`balance`, the one amount the account view and the simulated customer see, as a
plain number like `12000.00`), `param_space` (per-instance slot values),
`gold_tool_sequence`, `ordered` (OPTIONAL, default `false`),
`gold_document_ids` (OPTIONAL -- see below), `expected_state_deltas`,
`allowed_tools` (OPTIONAL). Verifiable templates carry a
derivable gold spec whose docs/tools MUST live within the SAME institution (scope
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@ templates:
account_state:
account_id: acct_kr_0001
account_type: Depot
cash_balance: '6000.00'
balance: '6000.00'
param_space:
stock: [Siemens, SAP, Allianz]
qty: ['5', '10', '20']
Expand Down Expand Up @@ -157,7 +157,7 @@ templates:
Können Sie {{amount}} € aus meinem Depot-Guthaben an meine Bank überweisen?
account_state:
account_id: acct_kr_0001
cash_balance: '12000.00'
balance: '12000.00'
linked_bank: bank_kr_0001
param_space:
amount: ['1.500', '4.000']
Expand Down Expand Up @@ -287,22 +287,22 @@ templates:
opening_user_messages:
- >-
Ich löse mein Depot auf: Streichen Sie bitte meine offene Order, zahlen Sie
das Guthaben von {{amount}} € auf mein Referenzkonto aus und kündigen Sie
danach das Depot.
mein gesamtes Guthaben auf mein Referenzkonto aus und kündigen Sie danach
das Depot.
- >-
Bitte stornieren Sie meine offene Order, überweisen Sie die {{amount}} € auf
mein Girokonto und schließen Sie anschließend mein Depot.
Bitte stornieren Sie meine offene Order, überweisen Sie mein komplettes
Guthaben auf mein Girokonto und schließen Sie anschließend mein Depot.
- >-
Können Sie die offene Order löschen, mir die {{amount}} € auszahlen und dann
mein Depot kündigen?
Können Sie die offene Order löschen, mir das ganze Guthaben auszahlen und
dann mein Depot kündigen?
account_state:
account_id: acct_kr_0002
account_type: Depot
cash_balance: '{{amount}}'
balance: '{{amount}}'
orders:
- {order_id: ord_kr_0002, ticker: '{{stock}}', status: open}
param_space:
amount: ['3.000', '12.000']
amount: ['3000.00', '12000.00']
stock: [SAP, Allianz]
gold_tool_sequence: [cancel_order, withdraw_funds, close_account]
# strict chain: cancel the open order to free the cash, withdraw it, THEN close.
Expand Down Expand Up @@ -355,7 +355,7 @@ templates:
account_state:
account_id: acct_kr_0001
account_type: Depot
cash_balance: '9000.00'
balance: '9000.00'
param_space:
stock: [Siemens, SAP]
qty: ['10', '15']
Expand Down Expand Up @@ -384,7 +384,7 @@ templates:
account_state:
account_id: acct_kr_0001
kid_account_id: acct_kr_kid_0001
cash_balance: '5000.00'
balance: '5000.00'
param_space:
amount: ['500', '1.500']
monthly: ['25', '50']
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -78,7 +78,7 @@ templates:
account_state:
account_id: acct_nb_0001
savings_account_id: acct_nb_sav_0001
checking_balance: '10000.00'
balance: '10000.00'
param_space:
amount: ['250.00', '1000.00', '5000.00']
gold_tool_sequence: [transfer_funds]
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -81,7 +81,7 @@ templates:
- Please move ${{amount}} of cash out to my bank account.
account_state:
account_id: acct_si_0001
cash_balance: '30000.00'
balance: '30000.00'
linked_bank: bank_si_0001
param_space:
amount: ['2000.00', '10000.00']
Expand All @@ -99,7 +99,7 @@ templates:
- I'd like to sweep ${{amount}} of cash from my brokerage over to my bank.
account_state:
account_id: acct_si_0001
cash_balance: '30000.00'
balance: '30000.00'
param_space:
amount: ['500.00', '3000.00']
gold_tool_sequence: [transfer_funds]
Expand Down Expand Up @@ -275,7 +275,7 @@ templates:
the account down?
account_state:
account_id: acct_si_0002
cash_balance: '{{amount}}'
balance: '{{amount}}'
orders:
- {order_id: ord_si_0002, ticker: '{{ticker}}', status: open}
param_space:
Expand Down Expand Up @@ -351,7 +351,7 @@ templates:
update my address to {{address}}?
account_state:
account_id: acct_si_0001
cash_balance: '{{amount}}'
balance: '{{amount}}'
orders:
- {order_id: ord_si_0003, ticker: '{{ticker}}', status: open}
param_space:
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -105,7 +105,7 @@ templates:
- Can you help me request a ${{amount}} loan against my 401(k)?
account_state:
account_id: acct_er_401k_active_0001
vested_balance: '80000.00'
balance: '80000.00'
param_space:
amount: ['5000.00', '15000.00']
gold_tool_sequence: [plan_loan]
Expand Down Expand Up @@ -267,7 +267,7 @@ templates:
${{contrib}}, and enable loan-payment alerts.
account_state:
account_id: acct_er_401k_active_0001
vested_balance: '90000.00'
balance: '90000.00'
param_space:
amount: ['10000.00', '20000.00']
contrib: ['150.00', '250.00']
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -83,7 +83,7 @@ templates:
account_state:
account_id: acct_cm_0001
savings_account_id: acct_cm_liv_0001
checking_balance: '10000.00'
balance: '10000.00'
param_space:
amount: ['250', '1 000', '5 000']
gold_tool_sequence: [transfer_funds]
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -96,7 +96,7 @@ templates:
bancaire.
account_state:
account_id: acct_vb_0001
cash_balance: '30000.00'
balance: '30000.00'
linked_bank: bank_vb_0001
param_space:
amount: ['2 000', '10 000']
Expand All @@ -120,7 +120,7 @@ templates:
banque.
account_state:
account_id: acct_vb_0001
cash_balance: '30000.00'
balance: '30000.00'
param_space:
amount: ['500', '3 000']
gold_tool_sequence: [transfer_funds]
Expand Down Expand Up @@ -246,7 +246,7 @@ templates:
account_state:
account_id: acct_vb_0001
account_type: PEA
cash_balance: '8000.00'
balance: '8000.00'
param_space:
ticker: [Air Liquide, Schneider Electric, TotalEnergies]
qty: ['5', '10', '25']
Expand All @@ -270,7 +270,7 @@ templates:
account_state:
account_id: acct_vb_0001
account_type: PEA
cash_balance: '2000.00'
balance: '2000.00'
param_space:
etf: [MSCI World, S&P 500, Stoxx Europe 600]
amount: ['50', '100', '250']
Expand Down Expand Up @@ -357,22 +357,22 @@ templates:
persona_tags: [region:any, age:any]
opening_user_messages:
- >-
Merci d'annuler mon ordre en cours, de retirer les {{amount}} € de liquidités
vers ma banque, puis de clôturer le compte-titres.
Merci d'annuler mon ordre en cours, de retirer toutes mes liquidités vers ma
banque, puis de clôturer le compte-titres.
- >-
Je veux annuler mon ordre en attente, sortir les {{amount}} € et fermer mon
compte-titres.
Je veux annuler mon ordre en attente, sortir toutes mes liquidités et fermer
mon compte-titres.
- >-
Pouvez-vous annuler l'ordre en cours, virer {{amount}} € sur mon compte
bancaire lié et clôturer le compte-titres ?
Pouvez-vous annuler l'ordre en cours, virer la totalité de mes liquidités sur
mon compte bancaire lié et clôturer le compte-titres ?
account_state:
account_id: acct_vb_0002
account_type: compte-titres
cash_balance: '{{amount}}'
balance: '{{amount}}'
orders:
- {order_id: ord_vb_0002, ticker: '{{ticker}}', status: open}
param_space:
amount: ['3 000', '12 000']
amount: ['3000.00', '12000.00']
ticker: [un ETF MSCI World, Air Liquide]
gold_tool_sequence: [cancel_order, withdraw_funds, close_account]
# strict chain: cancel the open order to free the cash, withdraw it, THEN close.
Expand Down Expand Up @@ -437,21 +437,21 @@ templates:
persona_tags: [region:any, age:any]
opening_user_messages:
- >-
Merci d'annuler mon ordre en cours, de retirer les {{amount}} € de liquidités
vers ma banque et de mettre à jour mon adresse : {{address}}.
Merci d'annuler mon ordre en cours, de retirer toutes mes liquidités vers ma
banque et de mettre à jour mon adresse : {{address}}.
- >-
Annulez mon ordre en attente, sortez {{amount}} € et changez mon adresse pour
{{address}}.
Annulez mon ordre en attente, sortez toutes mes liquidités et changez mon
adresse pour {{address}}.
- >-
Pouvez-vous annuler l'ordre en cours, virer {{amount}} € sur mon compte
bancaire lié et mettre à jour mon adresse ({{address}}) ?
Pouvez-vous annuler l'ordre en cours, virer la totalité de mes liquidités sur
mon compte bancaire lié et mettre à jour mon adresse ({{address}}) ?
account_state:
account_id: acct_vb_0001
cash_balance: '{{amount}}'
balance: '{{amount}}'
orders:
- {order_id: ord_vb_0003, ticker: '{{ticker}}', status: open}
param_space:
amount: ['4 000', '15 000']
amount: ['4000.00', '15000.00']
ticker: [un ETF MSCI World, TotalEnergies]
address: ['12 rue des Lilas, 69003 Lyon', '4 allée des Tilleuls, 33000 Bordeaux']
gold_tool_sequence: [cancel_order, withdraw_funds, update_contact_info]
Expand Down
6 changes: 6 additions & 0 deletions src/usersim/engine/probes/financial_services/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -401,6 +401,12 @@ enough to cover the tool surface with depth left over.
- **Pin the parameters the opening names.** If the user says "card ending 0001",
the seeded `account_state` must contain that card, or the agent resolves
against something the user never mentioned.
- **Hold the account's money under `balance`.** It is the one amount the account
view reports and the simulated customer is told, so money under another key
(`cash_balance`, `checking_balance`) reads as 0.00 while the opening asks to
move it. Write it as a plain number like `12000.00`: a localized `3.000` reads
as 3. Amounts beside it, such as a second account's balance, a limit or a
contribution, can stay.
- **Only use persona tags a persona can actually carry.** See the table in
SCHEMA.md. An unreachable tag doesn't error: the task just never gets
selected.
Expand Down
42 changes: 42 additions & 0 deletions tests/engine/probes/test_financial_services.py
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,7 @@
import types
import unicodedata
from contextlib import contextmanager
from decimal import Decimal, InvalidOperation
from pathlib import Path
from unittest.mock import patch

Expand Down Expand Up @@ -180,6 +181,47 @@ def test_all_committed_banks_load():
load_finance_bank(locale_dir) # raises FinanceBankError on any gold/scope break


def _decimal_places(value: object) -> int:
"""Digits after the decimal point when ``value`` reads as a number, else 0."""
try:
exponent = Decimal(str(value)).as_tuple().exponent
except InvalidOperation:
return 0
return -exponent if isinstance(exponent, int) else 0


def test_every_task_holds_the_account_money_under_balance():
"""The account view and the customer's grounding read only ``account_state.balance``,
so money held under another key such as ``cash_balance`` shows as 0.00 and the
customer is told their account is empty while asking to move money out of it.
Amounts beside a ``balance`` (a second account, a limit, a contribution) stay
allowed. A ``balance`` that reads as a number carries at most two decimals, so a
localized ``3.000`` cannot pass for 3; one filled from a slot is checked for every
value the slot can take.
"""
runnable, _ = _partition_committed_banks()
assert runnable, "no runnable committed banks found"
templates = [
(f"{locale_dir.name} {tpl.id}", tpl)
for locale_dir in runnable
for inst in load_finance_bank(locale_dir).institutions
for tpl in inst.templates
]
hidden, misread = [], []
for where, tpl in templates:
state = tpl.account_state
if "balance" not in state:
if any(key.endswith("_balance") for key in state):
hidden.append(where)
continue
raw = state["balance"]
slot = re.fullmatch(r"\{\{\s*(\w+)\s*\}\}", str(raw))
values = tpl.param_space.get(slot.group(1), ()) if slot else (raw,)
misread += [f"{where}: {value!r}" for value in values if _decimal_places(value) > 2]
assert not hidden, f"tasks hold the account's money outside balance: {hidden}"
assert not misread, f"balances with more than two decimals: {misread}"


def test_corpus_only_banks_are_declared_and_still_unrunnable():
"""Pin the corpus-only locales so the exemption cannot rot.

Expand Down
Loading