Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 5 additions & 5 deletions docs/locales.md
Original file line number Diff line number Diff line change
Expand Up @@ -40,12 +40,12 @@ finer breakdown.
> reviewed.

> `financial_services` runs where a locale has a bank of its own: `en_US`,
> `en_IN`, `fr_FR`, `en_GB` and `de_DE`. `hi_Deva_IN`, `hi_Latn_IN` and the
> India language variants run on the `en_IN` bank, with the opening and the
> `en_IN`, `fr_FR`, `en_GB`, `de_DE` and `ja_JP`. `hi_Deva_IN`, `hi_Latn_IN` and
> the India language variants run on the `en_IN` bank, with the opening and the
> assistant's searches machine-translated; a locale's own bank takes over once
> it has a task set. Like the others, the French, British and German banks carry
> `placeholder: true`: their rows record `used_placeholder_finance`, and SME
> verification is recommended.
> it has a task set. Like the others, the French, British, German and Japanese
> banks carry `placeholder: true`: their rows record `used_placeholder_finance`,
> and SME verification is recommended.

Per-probe coverage for every locale is in [Scope and coverage](probes.md#scope-and-coverage).

Expand Down
2 changes: 1 addition & 1 deletion docs/probes.md
Original file line number Diff line number Diff line change
Expand Up @@ -48,7 +48,7 @@ bank's language, with the same marker.
| `sov_ai_multilingual_parity` | all 11 | through `en_IN` | 48 questions, each written in every shipped locale |
| `safety_chat_pressure` | all 11 | through `en_IN` | 64 requests across 8 harm categories, with 6 pressure strategies; every request is written in every shipped locale |
| `safety_agentic` | all 11 | through `en_IN` | 48 action requests across 4 sub-protocols, each written in every shipped locale |
| `financial_services` | `en_US`, `en_IN`, `en_GB`, `fr_FR`, `de_DE`; `hi_Deva_IN` and `hi_Latn_IN` through `en_IN` | through `en_IN` | 4 to 6 institutions and 60 to 109 verifiable tasks per locale |
| `financial_services` | `en_US`, `en_IN`, `en_GB`, `fr_FR`, `de_DE`, `ja_JP`; `hi_Deva_IN` and `hi_Latn_IN` through `en_IN` | through `en_IN` | 4 to 6 institutions and 60 to 109 verifiable tasks per locale |
| `health_*_disclosure` (four clients) | all 11 | yes | one synthetic clinical profile per client; bring your own profile bank |
| `identity_disclosure` | all 11 | through `en_IN` | 14 worded openings in every locale (the English locales share one wording), plus 7 the simulated user writes |

Expand Down
6 changes: 3 additions & 3 deletions notebooks/01_simulate.ipynb
Original file line number Diff line number Diff line change
Expand Up @@ -40,7 +40,7 @@
"| Sovereign-AI | `sov_ai_multilingual_parity` | `assets/sov_ai_multilingual_parity/sample.yaml` | Cross-locale matched-pair queries: surfaces respect-parity gaps across demographic strata |\n",
"| Safety | `safety_chat_pressure` | `assets/safety_chat_pressure/sample.yaml` | Multi-turn pressure loop with deterministic reframing rotation against curated refusal targets |\n",
"| Safety | `safety_agentic` | `assets/safety_agentic/sample.yaml` | Mock-environment tool-call probing for boundary violations / consequence disclosure / injection resistance |\n",
"| Financial-services | `financial_services` | `assets/financial_services/<locale>/` | Multi-institution support/advisory in two tiers (verifiable + dynamic): **opt-in + asset-backed**; banks ship for `en_US`, `en_IN`, `en_GB`, `fr_FR` and `de_DE` (see the \"Financial-services probe\" note below the config cell) |\n",
"| Financial-services | `financial_services` | `assets/financial_services/<locale>/` | Multi-institution support/advisory in two tiers (verifiable + dynamic): **opt-in + asset-backed**; banks ship for `en_US`, `en_IN`, `en_GB`, `fr_FR`, `de_DE` and `ja_JP` (see the \"Financial-services probe\" note below the config cell) |\n",
"| Identity | `identity_disclosure` | `assets/identity_disclosure/spec.yaml` | Asks the model under test who it is and who made it, in the row's language; each row records the answers that model may give |\n",
"\n",
"In the sovereign-AI, safety, `financial_services` and `identity_disclosure` banks (locale-specific institutional facts, false-premise traps, attack-pattern targets, native-language taxonomies and identity questions), every entry that no native speaker or domain expert has verified carries `placeholder: true`. A trajectory that draws on one records a `used_placeholder_*` warning, and the capability report counts those rows per cell (`n_placeholder`). When every row behind a cell rests on placeholder or machine-translated content, the report recommends SME or native-speaker verification before you rely on that result."
Expand Down Expand Up @@ -294,7 +294,7 @@
" # Population-grounded, multi-institution financial support/advisory.\n",
" # OPT-IN + asset-backed: needs a bank under\n",
" # assets/financial_services/<locale>/ (banks ship for en_US, en_IN,\n",
" # en_GB, fr_FR and de_DE; hi_Deva_IN, hi_Latn_IN and the India variants\n",
" # en_GB, fr_FR, de_DE and ja_JP; hi_Deva_IN, hi_Latn_IN and the India variants\n",
" # use en_IN; see the \"Financial-services probe\" note below). Pair with\n",
" # FINANCE_TIER_MIX (above) to dial the verifiable/dynamic tier split.\n",
" # Feeds the financial_task_success* + dynamic_grounding capability rows.\n",
Expand Down Expand Up @@ -413,7 +413,7 @@
"\n",
"It is **asset-backed**: each locale needs a bank under\n",
"`assets/financial_services/<locale>/`. Banks for `en_US`, `en_IN`, `en_GB`,\n",
"`fr_FR` and `de_DE` ship with the repo, and `hi_Deva_IN`, `hi_Latn_IN` and the\n",
"`fr_FR`, `de_DE` and `ja_JP` ship with the repo, and `hi_Deva_IN`, `hi_Latn_IN` and the\n",
"India language variants use the `en_IN` bank. Any other locale needs a bank of\n",
"its own, task set included, before you add it to `LOCALES` (see\n",
"[Authoring a new locale](../src/usersim/engine/probes/financial_services/README.md#authoring-a-new-locale)).\n",
Expand Down
Loading
Loading