Payment operations and recovery
Use Payment operations for scoped alerts and unresolved initiation. A payment's operational trace connects its operation, attempts, provider reference and normalized observations; the delivery panel connects signed merchant events and HTTP attempts. Raw provider payloads, credentials and customer contact details are excluded from diagnostics.
Alert delivery and scope
The dashboard evaluates thresholds when refreshed. An authorized external monitor can poll GET /v1/internal/payments/health with the deployment's monitoring token. HTTP 503 means active alerts or unavailable diagnostics; inspect the response before attributing the cause. The endpoint itself does not send email, page an on-call team or measure availability. Configure the monitor and escalation recipient for each deployment.
Rates measure attempts, not unique merchant orders or causal uplift. Retries can contribute multiple attempts. Initiation and final outcomes use separate denominators. Unresolved initiations and webhook backlog include old work; completion alerts use a 24-hour creation cohort. Use payment history and delivery records alongside alerts to follow up on individual payments.
Unresolved initiation
Threshold: At least one keyed operation remains executing or unknown after 15 minutes.
Open Payment Operations, then the operation. Inspect the existing provider reference and recovery reason. Use Resolve with provider to query the original attempt. Keep the original reservation and idempotency key; never initiate another collection to clear uncertainty. Escalate unsupported or exhausted recovery to an administrator with a provider evidence reference.
Unresolved completion
Threshold: At least one accepted attempt is pending or unknown after 15 minutes in the last 24-hour creation cohort.
Open the payment operational trace, verify with its provider and review callback or query observations. Successful initiation does not prove payment completion. Older unresolved completion and conflicts remain in the existing investigation workflow; these cohort alerts do not replace that workflow.
Provider-interface failures
Threshold: At least five timeout or provider-down failures, representing at least 20% of at least 20 settled initiation attempts in a 24-hour creation cohort.
Inspect the affected app, provider credentials, eligibility and provider status. A timeout does not prove that the provider declined or did not accept the collection. Multiple connectors may share a card network, bank or mobile-money rail; switching connectors is not independent-rail redundancy.
Unclassified initiation failures
Threshold: At least five unclassified failures, representing at least 20% of at least 20 settled initiations in 24 hours.
Investigate normalized attempt codes and operation evidence before attributing responsibility. Validation errors and insufficient funds are excluded from this infrastructure alert. Unknown codes are not labelled as provider or Tinker Payments failures.
Verified final failures
Threshold: At least five failed, expired or cancelled results, representing at least 20% of at least 20 verified terminal outcomes in a 24-hour creation cohort. Conflicted results are excluded.
Review the payment timeline and provider evidence. These are final payment outcomes, which can include customer or issuer decisions; they do not directly measure provider availability or Tinker Payments uptime.
Merchant webhook backlog
Threshold: At least one queued event lacks a durable acknowledgement after 15 minutes, across all event sources and creation dates.
Open payment delivery evidence and inspect the event ID, last result and next eligible retry. Restore the recipient endpoint or delivery worker. The existing worker retries retained events; a restart recovers expired leases. Do not delete the queue or re-create events. A 2xx delivery log without an outbox acknowledgement can be delivered again.
Refund outcome unknown
Threshold: At least one refund is processing with an unknown provider outcome for 15 minutes, per app and provider.
The provider may have refunded. Keep the wallet hold and check the provider records, then settle the refund. A retried refund reuses its idempotency key; an unknown outcome is not a failure.
Refund payout stuck
Threshold: A refund has an M-Pesa B2C payout sent or claimed for 2 hours with no result. Visible to administrators only.
Never resend. Use Check with provider on the refund, then follow the refund operations runbook. If the customer may have been paid, confirm with Safaricom before marking the refund refunded or not refunded.
Withdrawal payout stuck
Threshold: A withdrawal has an M-Pesa B2C payout sent or claimed for 2 hours with no result. Visible to administrators only.
Never resend. Open the withdrawal and use Check payout. If there is still no answer, confirm with Safaricom whether the merchant was paid before you abandon the payout.
Refund payout may have paid twice
Threshold: A processing or refunded refund has a B2C payout and a reversal that may both have paid the customer, and no admin has acknowledged it. Visible to administrators only.
Check the Daraja portal for both transactions and recover any extra payment. Then acknowledge it on the refund; that clears the alert and changes nothing else. A processing refund: check before marking it refunded.
M-Pesa refund not confirmed
Threshold: An M-Pesa refund was accepted and recorded as refunded, but its signed result has not arrived for 2 hours. Visible to administrators only.
Never resend. Check the Daraja portal for the reversal. The signed result settles the refund when it arrives.
Withdrawal payout may have paid twice
Threshold: A withdrawal has a signed B2C success that could not be settled. Visible to administrators only.
Check the Daraja portal. Then abandon the payout with the confirmation and a reason, and reject the withdrawal or mark it paid by hand. If it was already rejected, the merchant may hold both the money and the released balance.
M-Pesa B2C Utility account low
Threshold: The Utility account balance in the latest signed B2C result is below Ksh 10,000. Counts and amounts only. Visible to administrators only.
B2C pays from the Utility account only, not Working or MMF. Top it up yourself: on the M-PESA portal, with the B2B API, or with B2C Account Top Up from a paybill. Tinker never moves the money. The alert clears with the next signed payout result.
Provider has no record of a payment
Threshold: A pending payment has been unknown to its provider for at least 1 hour (for example PayPal answered not found). Visible to administrators only.
Check that the gateway environment (live or sandbox) and credentials match the ones the payment was created with. It can also fire for orders the provider has expired that merchants keep polling. Never start a new collection for the same order from this alert.
Provider reports paid, no signed event
Threshold: A pending payment is more than 1 hour old and the provider reports it as paid, but no signed event has completed it. Visible to administrators only.
Do not mark it paid by hand. Check that the gateway webhook is configured, reachable and signing correctly for this environment, and look for it in the webhook inbox. Once the signed event arrives the payment completes and the alert clears. If no event comes, ask the provider to resend it.
Subscription cancelled: price changed at the provider
Threshold: In the last 7 days, a provider subscription was cancelled because its plan, price or currency no longer matched the billing terms. Visible to administrators only.
Tinker cancelled the subscription at the provider and locally. The buyer may already have paid: check the provider records and refund the buyer if so. Ask the merchant to subscribe again, and find who changed the plan at the provider.
Subscription renewal not recorded
Threshold: In the last 31 days, a provider subscription renewed and the provider charged the buyer, but Tinker has not recorded the charge. Visible to administrators only.
The renewal date moved forward, but no invoice or ledger entry was made. Confirm the charge at the provider, then record it by hand. Do not record it twice.
Stuck provider webhook events
Threshold: At least one incoming provider webhook event has been in retry for 24 hours. Visible to administrators only.
Open the webhook inbox for the gateway shown and read the last error. Fix the cause, such as missing credentials or an unavailable service. The event is retried until it succeeds. Do not delete it: the payment or subscription it carries would be lost.
Workers disabled
Threshold: A required worker is disabled by deployment configuration. Visible to administrators only.
Check TP_MANAGED_WORKERS_ENABLED and TP_PAYMENT_RECOVERY_WORKERS_ENABLED. Restore the intended worker configuration and restart the process. Read retained work and the next successful heartbeat before closing the incident.
Worker freshness
Threshold: No successful worker pass in five minutes, including startup before the first heartbeat. Visible to administrators only.
Inspect process health, database access and queue progress. A long-running or deferred pass requires investigation; this threshold does not identify the failing dependency. Restore the worker and confirm a fresh successful heartbeat.
Worker pass deferred
Threshold: The latest worker pass deferred work. Visible to administrators only.
Correlate the worker with per-payment and delivery evidence. Recipient outages, provider lookup failures and local failures have different remedies. The pass alert alone does not attribute fault to Tinker Payments or a provider.
Routing rollback
Use Routing Control to roll back the affected app's rollout with a specific reason. Confirm rolled_back, zero treatment traffic and the audit event. New traffic uses the existing app route; accepted sessions and in-flight payment evidence remain intact. Repeated or concurrent actions must reload the latest state. Never roll back by editing historical payment outcomes.
Delivery and restart semantics
Core payment outcome events and managed billing outbox events survive worker restarts. Claims lease for three minutes. Durable retries become eligible after 2, 4, 8 minutes and continue exponentially to a maximum delay of 1,024 minutes; the worker polls approximately once per minute. Processing or an outage can delay this further. There is no finite attempt cap or automatic expiry for this outbox.
Retries retain the event ID, signed payload and queued destination. Receivers must verify the signature in the JSON security field and atomically deduplicate event IDs with business effects. A recipient that accepts an event before the sender loses its acknowledgement can receive that event again. Events may arrive out of order. Existing historical goroutine deliveries are not backfilled into durable events; other legacy event publishers can retain their own bounded retry behavior.
Configured destination or signing-secret changes affect newly queued events. Existing events retain their original destination and signature; coordinate receiver key overlap and endpoint migrations. A malformed payload or invalid destination remains queued for investigation. Updating a merchant setting does not silently redirect retained financial events.
Business-hours support and incident handling
Support operates during business hours. Each support agreement defines the timezone, hours, contacts, acknowledgement and update targets. There is no blanket 24/7 response or uptime guarantee on this page.
- Critical: suspected duplicate collection, incorrect financial attribution, credential exposure or broad inability to collect. Contain unsafe execution, preserve evidence and escalate through your agreed urgent contact.
- High: persistent unknown outcomes, material provider-interface failures or aged webhook backlog. Investigate the affected scope, recover retained work and use the agreed update cadence.
- Routine: isolated noncritical integration or reporting questions. Use business-hours support and your agreed response target.
Share app, operation, payment, attempt and event IDs, UTC timestamps and normalized error codes. Keep credentials, full provider payloads and customer contact details out of incident messages.
Reliability measurement boundaries
Measure API availability, initiation acceptance, provider-confirmed completion, unresolved-age distribution, event acknowledgement latency and worker freshness separately. Define environment, probe, denominator, time window and exclusions in the support agreement before publishing a percentage. Use your agreed monitoring setup to track these measures consistently.