Observability И Наблюдаемость

Цели

• Сделать каждую финансовую операцию, provider callback, KYC decision, admin action и partner payout трассируемыми.
• Обнаруживать broken callbacks, stuck MassTransit outbox, Kafka lag, failed payments, provider degradation и tenant boundary violations.
• Не допускать попадания PII/secrets в logs, traces, metrics и error trackers.
• Сделать observability обязательной частью постановки задачи, а не технической доработкой после реализации.

Обязательство Для Операций

Каждая операция в документации описывает:

• logs: вход, ключевые шаги, controlled errors и unexpected errors;
• metrics: latency, error rate, throughput;
• tracing: correlationId, traceId, causationId для event/callback flows;
• audit: actor, action, entity id, before/after fields для admin/security/money решений.

Если операция критична для денег, KYC, identity, provider callbacks или withdrawals, отсутствие observability секции блокирует Definition of Ready.

Обязательные dashboards, SLO, alerts, owners и runbook requirements описаны в Operational Dashboard Catalog.

Сигналы

Логи

Используются structured Serilog logs:

• timestamp
• level
• messageTemplate
• service
• environment
• tenantId
• brandId
• actorId
• actorType
• correlationId
• traceId
• requestId
• route
• operation

Трейсы

Инструментируются:

• gateway request;
• controllers/actions;
• MediatR handlers;
• EF Core queries/transactions;
• HTTP clients;
• Kafka publish/consume;
• Redis;
• provider adapters;
• file storage calls.

Метрики

Metric	Purpose
http_server_duration	API latency.
http_requests_total	API traffic/errors.
kafka_consumer_lag	Consumer delay.
masstransit_outbox_pending_count	Stuck/unpublished events через MassTransit outbox.
masstransit_inbox_duplicate_count	Duplicate event volume через MassTransit inbox.
accounting_ledger_posting_duration	Financial posting latency.
accounting_hold_active_amount	Held withdrawal amount.
payment_callback_failure_count	Payment provider failures.
game_callback_latency	Provider callback latency.
game_callback_error_rate	Provider callback error rate.
identity_login_failure_rate	Brute force/fraud signals.
kyc_review_queue_size	Compliance backlog.
incentive_redemption_failure_count	Incentive issues.

Нейминг Операционных Метрик

Тип	Формат	Пример
Latency	<service>_<operation>_duration_ms	accounting_create_deposit_duration_ms
Throughput	<service>_<operation>_requests_total	kyc_approve_verification_requests_total
Errors	<service>_<operation>_errors_total	execution_gateway_external_integrator_request_errors_total
State transition	<service>_<entity>_<from>_to_<to>_total	accounting_withdrawal_requested_to_approved_total
Queue/backlog	<service>_<queue>_pending_count	accounting_masstransit_outbox_pending_count

Metric labels должны быть bounded: service, operation, tenantTier, providerCode, status, errorCode. Нельзя использовать raw user id, email, phone, document id или unbounded external reference как label.

Correlation Flow

• Gateway создает correlationId, если клиент не передал валидный id.
• Backend service принимает correlationId из headers, кладет его в log scope, trace baggage и event envelope.
• MassTransit event через MassTransit outbox сохраняет correlationId и causationId в envelope/headers.
• Consumer переносит correlationId в downstream logs, traces, metrics exemplars и external calls.
• Frontend сохраняет correlationId из response headers для support diagnostics, но не показывает raw technical details пользователю.

Единые правила propagation: Execution Context Propagation. Единый API middleware/filter pipeline: API Middleware And Filters Standard.

Черновик SLO

SLO	Target
Public API availability	99.9%
Accounting posting availability	99.95%
Accounting posting p95 latency	< 300 ms
Provider callback p95 latency	< 500 ms
Deposit callback processing p95	< 2 s
MassTransit outbox publish delay p95	< 30 s
Kafka consumer lag recovery	< 5 min
Admin panel availability	99.5%

Sentry

• Enable backend and frontend Sentry.
• Scrub PII, tokens, payment credentials and provider secrets.
• Tag errors with tenant/service/domain where safe.
• Use higher sampling for accounting, payments, game callbacks and KYC errors.
• Release tracking must include git sha/build version.

Группы Алертов

Group	Examples
Finance critical	Accounting posting failures, negative balance invariant, stuck withdrawals.
Provider callbacks	Signature failures spike, callback latency, duplicate external refs spike.
Security	Login failure spike, MFA failures, tenant boundary violation, permission changes.
Pipeline	Kafka lag, MassTransit outbox stuck, DLQ growth.
Infrastructure	PostgreSQL/Redis/Kafka/Elasticsearch/ClickHouse health.

Требования К Runbook

Every critical alert needs:

• impact description;
• dashboard links;
• first checks;
• rollback/mitigation steps;
• escalation owner;
• replay/reconciliation instructions where applicable.

Dashboard definition checklist и обязательные domain dashboards: Operational Dashboard Catalog.

Обратные ссылки

Автоссылки: где используется этот документ.

• adr (1): ADR-0047: Structured Logging И Observability Standards
• architecture-overview.md (1): Архитектура Pin