[0x5t4l1n]

Description

Fixes #384. truncate_by_length() can return 534 characters for a 500-character limit because its JSON path omits the truncation marker and JSON escaping from its length calculation. The text fallback also appends its marker after taking the full character budget.

Account for each field's serialized JSON length, including its marker, and use a binary search to retain the longest fitting prefix. Preserve the existing minimum retained field length and process further fields when needed. Reserve marker space in the fallback, handle zero-sized budgets, and return content that already fits unchanged.

The existing training challenges are preserved. If JSON cannot fit while keeping minimum field lengths, the helper continues to fall back to text, now within the requested limit. Small limits that cannot hold a marker retain only the content prefix.

Testing

On Windows with Python 3.10:

• PYTHONPATH=services/chatbot/src python -m pytest services/chatbot/src/chatbot/tests/test_agent_utils.py -q: 34 passed, including all 15 pre-existing tests.
• Regression cases cover ASCII, escaped newlines, Unicode, nested fields, fallback text and JSON arrays, zero/tiny budgets, exact fallback marker accounting, and unchanged content below the limit. The newly added regression cases exposed failures in the original implementation.
• Black check for both changed Python files and git diff --check: passed.

Tested independently from the Helm fix. Tests use the existing dependency stubs and do not require LLM credentials or databases. A live chatbot session and the full container stack were not run. Added a focused CI job for this utility suite and formatting checks.

Documentation

Added utility test instructions and truncation behavior to services/chatbot/README.md, and added pytest to the development requirements. The existing test module was formatted with Black.

Checklist:

• My changes generate no new warnings
• I have added tests that prove my fix is effective or that my feature works
• Any dependent changes have been merged (none required)
• I have documented any changes if required in the docs.