If your n8n workflow reports success but data is missing, you likely fell into the pagination trap. In a real-world case from AI Automation Lab, a nightly client sync ran green for three weeks while silently dropping twelve percent of its rows. The API returned 100 orders per page, but the workflow was configured to fetch only one page. As order volume grew past 100 a night, the extra orders sat on page two, untouched, while the run reported a false success.

The 'Got a Response' Fallacy

The core mistake was treating a successful HTTP response as proof of complete data retrieval. Paginated APIs do not promise everything; they promise one chunk. The burden of asking for the rest lies entirely with the developer. If you are using a single HTTP node in n8n to pull a list, you are not building a sync; you are building a sample. Samples presented as totals are how clients lose data without anyone noticing until a manual audit happens weeks later.

Three Checks to Catch Silent Failures

First, never trust a page without checking for more. Your loop should end only on an explicit signal, such as an empty next cursor or a page smaller than the limit. Second, distinguish between offset and cursor pagination. Offset pagination breaks when rows shift mid-sync, causing duplicates or gaps, while cursor pagination is stable but can expire. Store the cursor or offset in your workflow state after every page so a crash on page four resumes at page four, not page one. Third, dedupe on the business key, not row position. When retries or offset shifts cause overlaps, the record's unique ID is the only truth. Position-based deduplication fails when the underlying data moves. If the API provides a total count, assert that received rows equal the total at the end of the loop. If not, log the page count on every run so a sudden jump from one to three pages becomes visible rather than silent.

Key Takeaways

  • A single HTTP node is not a sync; it is a sample that can mislead you.
  • End loops on explicit signals (empty cursor, short page), not just a 200 OK.
  • Store pagination state after every page to enable safe resumption after crashes.
  • Dedupe using the source system's unique ID, never row position or order.
  • Report four metrics: pages fetched, rows received, rows written, and rows skipped.

The Bottom Line

If your sync reports 'done' without showing page counts, it is hiding the most critical number. One page is not a success; it is a warning sign.