Production error triage

Compiled from 20 minutes of live production logs (crm-backend-app). Totals in that window: 239 ERROR + 1142 WARN lines. Almost all collapse into a handful of root causes below, ordered by impact (frequency × severity). All file:line references are in crm-backend. Business logic stays the same.

Summary table

Problem 1 — WAHA revoked-message retry storm (CRITICAL, 192/20m)

Chain: When a WhatsApp user deletes a message we never stored (old message, untracked chat), MessageRevokedHandler.markMessageAsRevoked() throws NotFoundException (message-revoked-handler.service.ts:49-51). That bubbles to UnifiedWebhookController.handleWahaWebhook (unified-webhook.controller.ts:63-70) which returns HTTP 500. WAHA treats 500 as "delivery failed" and retries the same event every few seconds indefinitely — that's why the exact same stanza appears 16× in 3 minutes.

Fix A (root cause) — a missing message is not an error; there's nothing to revoke. In markMessageAsRevoked, return null instead of throwing, and have handle() skip the broadcast when null:

private async markMessageAsRevoked(providerId: string): Promise<Message | null> {
  const messageRepository = this.getRepository(Message);
  const existingMessage = await messageRepository.findOne({ where: { stanza: providerId } });
  if (!existingMessage) {
    this.logger.debug(`Revoked message ${providerId} not in DB — nothing to revoke`);
    return null;
  }
  existingMessage.isRevoked = true;
  existingMessage.body = '';
  return messageRepository.save(existingMessage);
}
// in handle():
const message = await this.markMessageAsRevoked(revokedMessageId);
if (message) await this.broadcastMessageRevokedEvent(message);

Fix B (defense-in-depth, stops ALL such retry storms) — in the WAHA controller catch block, acknowledge non-retryable errors so WAHA stops retrying. A NotFoundException / 4xx means "retrying won't help":

} catch (error) {
  if (error instanceof NotFoundException || error instanceof BadRequestException) {
    this.logger.warn(`WAHA webhook non-retryable: ${(error as Error).message}`);
    return { status: 'ignored' }; // 200 → WAHA stops retrying
  }
  const err = error as Error;
  this.logger.error(`Error processing WAHA webhook: ${err.message}`, err.stack);
  throw new HttpException('Error processing webhook', HttpStatus.INTERNAL_SERVER_ERROR);
}

Do both A and B. Keep the revoked-handler unit test (message-revoked-handler.service.spec.ts) updated — it currently asserts a throw; change it to assert graceful no-op.

Problem 2 — Unknown WAHA event noise (440/20m)

webhook-router.service.ts:81 logs .warn for every event with no handler. engine.event (and message.reaction) are normal WAHA lifecycle events we intentionally don't handle — they should not be warnings.

Fix: keep a set of known-ignorable events at debug, warn only on truly unknown:

private static readonly IGNORED_WAHA_EVENTS = new Set(['engine.event', 'message.reaction', 'presence.update']);
...
} else if (WebhookRouterService.IGNORED_WAHA_EVENTS.has(event as string)) {
  this.logger.debug(`Ignoring unhandled WAHA event: ${event}`);
} else {
  this.logger.warn(`No handler found for WAHA event: ${event}`);
}

Problem 3 — Contact-profile fetch logged as ERROR (26/20m)

waha-contact-provider.service.ts:140-146 already catches and return null (non-fatal — the avatar just couldn't be fetched, usually a private/absent profile pic). But it logs at .error, inflating the error count and paging noise.

Fix: this.logger.error( → this.logger.warn( (or .debug). No behavior change.

Problem 4 — Dead FCM tokens never pruned (~38/20m, grows forever)

unified-push-notification.service.ts:

• handleSendError (line 660) treats messaging/invalid-registration-token and messaging/registration-token-not-registered as a warn, but SenderId mismatch falls into the generic .error branch (line 669) — and it is equally permanent (token belongs to a different Firebase project). These tokens are never deleted, so every future notification retries them and re-logs the error.

Fix:

1. Classify SenderId mismatch / messaging/mismatched-credential as permanent alongside the two invalid-token codes.
2. Return permanently-dead tokens up to the caller and delete them from the token store. send() already returns failedTokens (line 180) but the errors aren't classified. Add permanentlyFailedTokens: string[] to the result, populate it in the catch (line 155), and have the caller (the service that loads device tokens) delete those rows after send.

private isPermanentTokenError(error: any): boolean {
  const code = error?.code || error?.errorInfo?.code || '';
  const msg = (error?.message || '').toLowerCase();
  return code === 'messaging/invalid-registration-token'
    || code === 'messaging/registration-token-not-registered'
    || code === 'messaging/mismatched-credential'
    || msg.includes('senderid mismatch');
}

This stops the recurring FCM errors permanently and shrinks each notification's work.

Problem 5 — Chat not found for presence.update (16/20m)

Presence (online/typing) for a chat we don't track. Non-fatal. Fix: log at .debug. (Also covered by adding presence.update to the ignored-events set in Problem 2.)

Problem 6 — Stanza … inserted concurrently; retrying as update (22/20m)

This is a correctly handled race (two webhook copies of the same message; the retry-as-update works). Cosmetic only. Fix (optional): log at .debug.

Problem 7 — No default Waha session configured (7/20m)

admin-assignment-notification.service.ts:52 and comment-mention-notification.service.ts:60. An admin was assigned / mentioned but the tenant has no default WAHA session, so the WhatsApp DM can't be sent. This is a config/data condition, not a code bug. Fix: (a) keep as .warn (correct), and (b) decide product-side whether these tenants should have a default session set, or whether assignment-WhatsApp should be disabled when none exists (to avoid the warn on every assignment).

Problem 8 — Dropping chat-scoped "call" event with no sessionSlug (4/20m)

Call events are built without a sessionSlug, so the realtime layer can't apply visibility filtering and drops them → call events may not reach the frontend. Fix: populate sessionSlug when constructing the call event (trace the WAHA call handler → event payload).

Problem 9 — Object 'lead' not found — skipping (2/20m)

A webhook trigger and a distribution rule reference object slug lead, but the object is leads (plural, per the auto-create logs). Likely a misconfigured webhook/distribution targeting a non-existent slug. Fix: data — correct the slug in that workspace's config; optionally validate slugs when saving webhook/distribution configs.

Problem 10 — FK_waha_chats_session violation (1/20m, rare)

A messaging_chats insert referenced a session_id that didn't exist (session being created/deleted concurrently). Rare race. Fix: ensure the session row exists/committed before inserting chats, or catch the FK error and retry once after re-resolving the session.

Priority

1. Problem 1 — kills the 192-error retry storm (biggest single win; also cuts CPU bursts).
2. Problems 2 + 3 + 5 + 6 — one-line log-level/ignore-set changes; remove ~500 warns + 26 errors of pure noise.
3. Problem 4 — FCM token pruning; stops a class of errors permanently and speeds up every push.
4. Problems 7–10 — config/data + rare races; fix after the above.

Verification

• npm run build + npm run lint.
• Update/keep message-revoked-handler.service.spec.ts (assert graceful no-op, not throw).
• After deploy, re-watch 10 min of logs: Problem 1 stanza errors should be gone (WAHA stops retrying once it gets 200), and warn volume should drop by ~40%.

Deploy

Code-only; rebuild + restart the single crm-backend-app container (~30–60 s downtime) in a quiet window, same as the last performance commit. No migrations needed.