.. SPDX-License-Identifier: CC-BY-4.0 SPDX-FileCopyrightText: Netresearch DTT GmbH .. include:: /Includes.rst.txt .. _administration-analytics: ============================ Usage analytics ============================ The Analytics submodule turns the per-request data in :sql:`tx_nrllm_service_usage` into an at-a-glance view of what your AI spend and usage look like over time — cost and request trends, breakdowns by provider, model, and service, and per-user consumption against this month's budget. .. figure:: /Images/backend-analytics.png :alt: LLM Usage Analytics dashboard — KPI tiles, a cost and request trend chart, breakdowns by provider, model, and service, and a per-user table with monthly budget bars :class: with-border with-shadow :zoom: lightbox The Analytics dashboard: KPI summary tiles, the cost/request trend, the provider / model / service breakdowns, and per-user consumption against each user's monthly budget. .. _administration-analytics-open: Opening the module ================== Open :guilabel:`AI > Operation > Analytics`. The submodule sits next to the other LLM sections in the left-hand navigation and is admin-only, like the rest of the module. .. _administration-analytics-range: Choosing a date range ===================== A range switcher at the top of the page selects the reporting window. The range is a plain ``?range=`` link, so changing it is an ordinary page reload — there is no AJAX. Four presets are available: .. list-table:: :header-rows: 1 :widths: 20 80 * - Preset - Window * - ``7d`` - The last 7 days (today and the six preceding days). * - ``30d`` - The last 30 days. This is the default — any unknown range value falls back to ``30d``. * - ``90d`` - The last 90 days. * - ``month`` - From the first of the current calendar month to today. .. _administration-analytics-kpis: KPI tiles ========= A row of tiles summarises the selected range: - **Total cost** — the summed estimated cost across the window. - **Total requests** — the number of AI requests recorded. - **Total tokens** — prompt plus completion tokens consumed. - **Providers** — how many distinct providers were used. - **Models** — how many distinct models were used. These are totals for the chosen range, not all-time figures. .. _administration-analytics-trend: Cost and requests trend ======================= A line chart plots daily estimated cost and daily request count across the range. Days with no usage are filled in as zero so the line is continuous rather than skipping gaps. .. _administration-analytics-breakdowns: Breakdown charts ================ Four bar charts split the window's usage along different axes: - **By provider** — cost and requests per ``service_provider`` (OpenAI, Anthropic, Ollama, …). - **By model** — cost and requests per model. This dimension is new: it relies on the ``model_uid`` / ``model_id`` columns added to the usage table, so it only reflects usage recorded after that change. - **By service** — cost and requests per service type (chat, vision, translation, speech, image). - **By extension** — cost and requests per calling extension, see :ref:`administration-analytics-per-extension`. .. _administration-analytics-per-extension: Per-extension table =================== A table lists cost, requests and tokens per **calling extension**, ordered by cost. It answers which piece of software spent what: a consumer names itself with ``AbstractOptions::withCallerSource()`` (:ref:`ADR-177 `), and that name is stored on the usage row alongside the money (:ref:`ADR-178 `). Where the entries come from: - The compatibility layer `nr-llm-compat `__ tags every call it reroutes, so an intercepted third-party extension (``ai_filemetadata``, ``texter``, ``ns_t3ai``, …) shows up under its own extension key. - Any other consumer that annotates its calls appears the same way. - Everything else — wizard tasks, scheduler runs, playground calls and any consumer that does not annotate — is listed as **Unattributed**. That is the normal state, not an error. .. note:: The name is what the caller claims. Attribution is an inventory of the installation, not an access control: a caller can name itself anything, and nothing verifies it. Rows written before the column existed are unattributed — no migration invents an origin for them. .. _administration-analytics-per-user: Per-user table ============== A table lists usage grouped by backend user, ordered by cost. Each row shows the user's request count, token total, and estimated cost for the selected range, plus a **monthly-budget bar** that visualises how much of their per-user budget (see :ref:`administration-user-budgets`) they have consumed. .. note:: The budget bar always reflects the **current calendar month**, regardless of the date range selected above. The other columns follow the selected range; the budget bar does not, because a budget is a monthly ceiling. Requests made without an authenticated backend user (CLI, scheduler, ``be_user = 0``) are grouped under a **system** row. Since TYPO3 14.3.8 a console command that authenticates, the scheduler included, records its usage under core's command-line user ``_cli_`` instead, and this table lists that user by name. .. _administration-analytics-rescues: Fallback rescues ================ A table lists the runs a **different** configuration answered after the requested one failed — each line is one request the configuration you configured did not serve. It shows what was requested and what answered, each with its provider and model, how many configurations were tried, and how long the whole run took. Unlike the rest of this module the list is read from the telemetry log (``tx_nrllm_telemetry``), not from the usage table, so it also covers runs that produced no billable usage. Two things it deliberately does not show: * **Runs nobody served.** A chain that was tried and exhausted names no serving configuration — it is a failure, not a rescue, and appears in the provider health scores instead. * **Runs recorded before this feature existed.** Rows written by an older version carry no serving configuration and are left out rather than guessed at. At most the 200 newest rescues of the period are listed. The limit counts rescues, not failed attempts, so a long outage — which writes one row per request — cannot push the rescues out of the list. A configuration appearing here repeatedly is the signal to look at: its calls are being answered by a sibling, which may use a different provider, model and price than the one you selected. .. _administration-analytics-health: Provider health and circuits ============================ A table lists every provider that is either configured and active or named by a run in the telemetry window, with its **health score**, the **number of samples** the score is based on, the **window** those samples were taken over, and the state of its **circuit breaker**. Health and circuit state are both keyed by **adapter type**, not by provider record — two provider records on the same adapter share one score and one circuit, because it is the provider that is unhealthy, not the record. .. note:: **Direct calls with a pinned provider are not in this table.** :php:`chat()`, :php:`complete()`, :php:`embed()` and their siblings called with a ``provider`` option skip the default configuration entirely and run against a transient one carrying no model. Such a run records no provider in the telemetry log, and its circuit is kept under the call's own identifier (``ad-hoc:chat:openai``, one per operation) rather than under ``openai``. An ``openai`` row here therefore reports the circuit of the configuration-backed calls only; a circuit opened for direct pinned calls is not shown anywhere on this page. Route traffic through a configuration if you need it covered. .. list-table:: :header-rows: 1 :widths: 25 75 * - Column - Meaning * - Score - A single 0.00–1.00 number combining success rate and mean latency, success rate weighted four times as heavily (see :ref:`adr-063`). Higher is healthier. * - Samples in window - How many runs the score was computed from. Read it before the score: 0.90 over two calls and 0.90 over two thousand are different statements. * - Success rate - Share of runs the provider served **itself**. A run a fallback rescued counts as a failure of the requested provider. * - Avg latency - Mean end-to-end time of the self-served runs. A provider whose runs in the window were **all** rescued by a fallback has no self-served run to measure: the cell says the latency was not measured instead of showing ``0 ms``. Its score and success rate are real — the first attempt did lose. * - Circuit - ``closed`` (normal), ``open`` (failing fast for the cooldown) or ``half-open`` (cooldown elapsed, one probe due), plus the current consecutive-failure streak. Unlike the rest of this module the table ignores the date range selected above. Scores come from a rolling telemetry window (15 minutes by default, named on the page) and circuit state is live cache state — neither can be re-cut to a 90-day report period. .. important:: **A score only changes something when you switched it on.** :guilabel:`Health-Aware Fallback Reorder` (``health.reorderFallback`` in the extension configuration) is **off by default**. While it is off, the scores are diagnostic only: the fallback chain keeps the order you configured. The page states which of the two positions the switch is in, above the table. The circuit breaker (``circuitBreaker.enabled``) is on by default, and the page says so when it is not. A provider with no telemetry in the window shows **no data** — not a score of zero. It was not called; that is not the same as failing. .. _administration-analytics-billing: Monthly billing report ====================== :guilabel:`Monthly billing report`, a button below the range switcher, opens a page for one calendar month (:ref:`ADR-231 `). It shows the previous complete month by default; the running month and the twelve months before it can be selected. A month runs from its first to its last second in the server timezone, and the page names both. The running month is marked as not complete. The page has four parts: Cost Requests, tokens, the estimated cost of the month, the markup in percent, the markup amount and the total including the markup. Active editors Every backend user with at least one request in the month, with cost, requests and tokens. A user counts as active whatever their account state is today; the column :guilabel:`Account state today` shows when the user is an administrator, disabled or deleted. Requests made without a backend user (command line, scheduler) are listed in a separate row and are not counted as an editor, so the rows add up to the month total. That row includes the usage of core's command-line user ``_cli_``: since TYPO3 14.3.8, console commands such as the scheduler record their usage under it. Entitled to AI Tasks Backend users who could use the AI Tasks module, **as of today**: users, groups and permissions keep no history, so this list shows today's state for every month, and the page says so with the date. A user is counted when all of these hold: * the account can log in now: not disabled, not deleted, within its start and end time; * the permission :guilabel:`Execute AI tasks` (see :ref:`administration-permissions`) is granted through one of the user's backend groups or a subgroup of one; hidden and deleted groups grant nothing; * the module :guilabel:`AI Tasks` is allowed on the user record or on one of those groups. Administrators are not counted: they can use every AI feature without the permission. The page states how many administrators can log in now, leaving out core's command-line user ``_cli_``, which TYPO3 creates as an administrator with a random password. Extensions that call the AI services with their own permission checks, such as an editor assistant or a chat, grant access their own way; users whose access comes only from such an extension are not counted here, so an active editor is not necessarily entitled to AI Tasks. Do not bill per entitled editor without deciding which of these accesses counts. Results by kind Requests, tokens and estimated cost by service type, calling extension (see :ref:`administration-analytics-per-extension`) and task. .. _administration-analytics-billing-markup: Markup ------ .. confval:: billing.markupPercent :name: nrllm-billing-markuppercent :type: int :default: 0 A whole percentage from 0 to 1000 the billing report adds to the estimated cost of the month. Set it under :guilabel:`Admin Tools > Settings > Extension Configuration > nr_llm`. The markup is added once, to the month total. Stored costs do not change. The report calculates in micro-dollars, the 6 decimals the cost is stored with: the markup is rounded half up to one micro-dollar, and cost plus markup equals the total exactly. The CSV keeps all 6 decimals. The page shows cents: cost and total are each rounded half up, and the markup line is the rounded total minus the rounded cost, so the three lines add up. A value outside 0 to 1000 is clamped and a value that is not a whole number reads 0; the page then says that the stored value was not used as it is. .. _administration-analytics-billing-export: CSV export ---------- :guilabel:`Download CSV` downloads the selected month as :file:`nr-llm-billing-YYYY-MM.csv`. Like the page, it is available to administrators only. The file is one table, so a spreadsheet imports it in one step. The first column, ``section``, says what a row is; columns that do not apply to a section stay empty: .. list-table:: :header-rows: 1 :widths: 30 70 * - ``section`` - Row * - ``total`` - Requests, tokens, ``estimated_cost_usd``, ``markup_percent``, ``markup_usd`` and ``total_usd`` of the month. * - ``active_editor`` - One per active editor: ``be_user_uid``, ``username``, ``real_name``, ``account_state`` (``admin``, ``disabled``, ``deleted``, space-separated; empty for an ordinary account) and the usage. * - ``without_backend_user`` - The usage recorded without a backend user or under the command-line user ``_cli_``, if any. * - ``entitled_editor`` - One per entitled editor, with ``state_as_of``: the date of today's state. * - ``by_kind`` - One per combination of ``service_type``, ``source_extension`` and ``task_uid`` / ``task``, with the usage. The format: comma-separated, fields in double quotes where needed, CRLF line ends, UTF-8 without a byte order mark. Numbers have a dot as decimal separator and no thousands separator; amounts are USD with 6 decimals, and the column names say the currency. A text cell that starts with ``=``, ``+``, ``-``, ``@``, a tab or a carriage return gets a leading ``'`` so a spreadsheet does not run it as a formula. .. _administration-analytics-cost-note: A note on cost ============== All cost figures are **estimated**. They are computed from the model pricing you configured (cents per 1M tokens, applied to the recorded prompt/completion token split), not billed back from the provider. Treat them as a planning and trend signal, not as an invoice. Costs are captured at call time, so they reflect the pricing in effect when each request ran. See :ref:`adr-029` for the design rationale. Specialized services (DALL·E, text-to-speech, Whisper, DeepL) still record their requests and units, but their cost is currently shown as ``0`` — token-based pricing does not apply to them yet. Streaming responses are not recorded at all, because chunked output has no single terminal token count to price. .. _administration-analytics-list-columns: Usage columns in the list views =============================== The Providers, Models, Configurations, and Tasks list views each carry three extra columns — :guilabel:`Cost (30d)`, :guilabel:`Requests (30d)` and :guilabel:`Tokens (30d)` — summarising the last 30 days of usage for that row, so you can spot the heavy hitters without leaving the list. .. figure:: /Images/backend-models-usage.png :alt: Models list with Cost / Requests / Tokens (30d) columns showing per-model usage and estimated cost :class: with-border with-shadow :zoom: lightbox The Models list with the 30-day usage columns. Models with no usage in the window show blank cells; free local models show ``~$0.00``. Numbers follow the backend user's language: a German backend shows ``~0,00 $`` and ``5.138.706``. Amounts are always USD. Two attribution notes: * The Providers column aggregates by **adapter type** (the value stored on each usage row), not by individual provider record — two providers that share an adapter therefore show the same figures. * The Tasks column relies on per-task tracking: each task execution records its ``task_uid`` so usage rolls up to the task that triggered it. Calls made outside a task (direct API/service use) are not attributed to any task row. .. _administration-analytics-demo-data: Demo data for local development =============================== To populate the module with something to look at during local development, run the dev-only DDEV command: .. code-block:: bash ddev seed-demo It generates roughly 90 days of realistic historic usage across providers, models, services, and users so the trend line, breakdown charts, and per-user table all have content. This command is for local DDEV environments only — do not run it against production data.