Operational Dashboard Catalog
Назначение
Каталог задает обязательные operational dashboards для production-эксплуатации Pin. Dashboard считается готовым только если у него есть владелец, SLO или рабочий порог, алерты, ссылка на runbook и bounded labels без PII, секретов, access tokens, raw provider payload и unbounded identifiers.
Общие Правила
- Каждый backend service имеет service dashboard с RED: request rate, error rate, duration.
- Все графики фильтруются минимум по
environment,service,tenantTier,brandIdтам, гдеbrandIdимеет bounded cardinality. - Для provider flows допускаются labels
providerCode,operation,status,errorCode; raw external reference в labels запрещен. - Для финансовых и security-critical панелей mandatory alerts являются blocking requirement для release.
- Runbook хранится рядом с service/system документацией и содержит impact, первые проверки, mitigation, replay/reconciliation и escalation owner.
- Dashboard не является source of truth для денег, KYC, incentive decision check или user limits; он показывает состояние source of truth и documented projections.
Обязательные Dashboards
| Dashboard | Owner | Цель | Основные сигналы |
|---|---|---|---|
platform-api-red | Platform | Здоровье gateway и backend API. | http_requests_total, http_server_duration, http_errors_total, p95/p99 latency. |
accounting-ledger-operations | Accounting | Контроль ledger postings, holds и reconciliation. | posting latency, failed postings, imbalance detector, active holds, reconciliation gap. |
payments-deposits-withdrawals | Payments | Депозиты, withdrawals, webhooks и provider degradation. | callback failures, pending withdrawals, provider latency, duplicate callbacks, payout failures. |
gamehost-provider-callbacks | GameHost | Игровые session/round callbacks и settlement. | bet/win/rollback latency, duplicate round events, settlement errors, provider timeout rate. |
identity-security | Identity | Auth, sessions, MFA, permission changes и tenant isolation. | login failures, MFA failures, suspicious sessions, permission mutations, scope violations. |
kyc-aml-review | KYC | KYC queue, AML decisions и manual review backlog. | queue size, review age, provider failure rate, decision latency, blocked withdrawals. |
incentive-economy-guardrails | Incentive | Incentive cost, wagering, abuse и guardrail breaches. | redemption errors, incentive liability, wagering stuck count, abuse flags, budget burn. |
partner-traffic-payouts | Partner | Partner traffic quality, commissions и payouts. | click/signup conversion, qualified FTD, commission pending, payout failures, fraud flags. |
events-and-outbox | Platform | Kafka, MassTransit outbox/inbox и DLQ. | outbox pending, publish delay, consumer lag, duplicate inbox count, DLQ growth. |
data-platform-analytics | Analytics | ClickHouse/Elasticsearch freshness и report availability. | ingestion delay, failed exports, freshness SLA, query latency, storage errors. |
release-and-flags | Platform/Release | Rollout safety, risky flags и release health. | active risky flags, kill switch activations, cohort error delta, release error budget burn. |
RED Метрики
| Область | Rate | Errors | Duration |
|---|---|---|---|
| API | http_requests_total by route, method, status | 5xx, controlled business failures by errorCode | p50/p95/p99 http_server_duration |
| MediatR | <service>_<command_or_query>_requests_total | <service>_<command_or_query>_errors_total | <service>_<command_or_query>_duration_ms |
| Provider adapter | <service>_<provider>_<operation>_requests_total | signature, timeout, validation, provider error | callback/client p95/p99 |
| Messaging | consumed/published messages | handler failures, DLQ, duplicates | publish delay, handler duration |
| Frontend | page/API call count | JS errors, API problem responses | page load, interaction latency |
Business Metrics
| Domain | Метрики | Alert Intent |
|---|---|---|
| Accounting | ledger transactions count, debit/credit sum, imbalance count, reconciliation gap, hold amount. | Найти нарушение double-entry, зависшие holds и расхождение с projections. |
| Payments | deposit success rate, withdrawal approval age, provider callback failure rate, payout pending amount. | Защитить денежные операции и SLA provider flows. |
| GameHost | active sessions, settled rounds, rollback ratio, provider duplicate count, jackpot settlement failures. | Выявить broken callbacks, replay storms и settlement drift. |
| Incentive | active incentive liability, incentive cost by brand, wagering completion rate, expired incentive amount. | Ограничить incentive abuse и runaway cost. |
| Identity/KYC | login failure rate, account lockouts, KYC review age, AML hold count. | Отследить fraud/security spikes и compliance backlog. |
| Partner | FTD quality, chargeback/refund impact, commission approval age, payout failure count. | Не выплатить ошибочную commission и увидеть плохой traffic source. |
| Analytics | event ingestion delay, report freshness, export failure rate. | Не принимать решения по устаревшим данным. |
SLO И Error Budget
| SLO | Target | Dashboard | Alert |
|---|---|---|---|
| Public API availability | 99.9% monthly | platform-api-red | burn rate 2% за 1 час или 5% за 6 часов. |
| Accounting posting availability | 99.95% monthly | accounting-ledger-operations | любая серия failed postings выше порога 5 минут. |
| Accounting posting p95 | < 300 ms | accounting-ledger-operations | p95 выше 300 ms 10 минут. |
| Provider callback p95 | < 500 ms | gamehost-provider-callbacks | p95 выше 500 ms 10 минут по provider. |
| Deposit processing p95 | < 2 s | payments-deposits-withdrawals | p95 выше 2 s 10 минут или spike failures. |
| Outbox publish delay p95 | < 30 s | events-and-outbox | p95 выше 30 s 10 минут. |
| Critical report freshness | < 15 min | data-platform-analytics | freshness выше SLA 15 минут. |
Alerts, Runbooks И Owners
| Alert | Severity | Owner | Runbook Requirements |
|---|---|---|---|
accounting_ledger_imbalance_detected | Critical | Accounting + Finance | Stop affected rollout, isolate tenant/brand, run ledger reconciliation, block direct balance repair. |
payment_provider_callback_failure_spike | High | Payments | Check signatures, provider status, replay inbox, switch provider route only through approved flag. |
gamehost_settlement_error_spike | High | GameHost | Pause provider cohort, inspect idempotency, reconcile game rounds with accounting postings. |
tenant_scope_violation_detected | Critical | Security | Disable actor/session, preserve audit evidence, notify security owner, review recent deploy/flag. |
kyc_review_backlog_sla_breach | Medium | KYC/Ops | Rebalance review queue, fail closed for withdrawals needing KYC, record compliance note. |
incentive_liability_guardrail_breach | High | Incentive + Finance | Disable risky incentive flag, freeze new activations, reconcile liability and brand budget. |
outbox_publish_delay_breach | High | Platform | Inspect DB/Kafka, avoid manual event inserts, resume publisher, verify idempotent consumers. |
release_cohort_error_delta | High | Release Owner | Stop rollout, compare flagged/unflagged cohorts, decide rollback, kill switch or forward-fix. |
Dashboard Definition Checklist
- Название dashboard и owner team указаны явно.
- Описаны RED metrics, business metrics и bounded labels.
- Для critical path есть SLO или operational threshold.
- Alert имеет severity, owner, runbook и escalation path.
- Есть links на logs/traces по
correlationId,tenantId,brandId,operation. - Нет PII, secrets, raw payment/KYC/provider data в labels, annotations и screenshots.
- Для release dashboards есть фильтр по
releaseVersion,gitSha,featureFlagKey,rolloutCohort.
Обратные ссылки
Автоссылки: где используется этот документ.
- Дашборды (2): Панель Наблюдаемости: news-overview (эксплуатация), Панель Наблюдаемости: support-overview (эксплуатация)
- observability.md (1): Observability И Наблюдаемость
- backend-shared-standards-index.md (1): Backend Shared Standards Index