Scrape Infinite Scroll & Pagination to CSV

To scrape infinite scroll to CSV, use a tool that auto-scrolls the page, waits for new rows to load, and keeps going until the list stops growing. For paginated lists, it follows the "Next" link or numbered pages and merges every page into one file. Both methods capture the full dataset, not just the first screen.

This guide explains the three ways lists load more rows, how automatic collection avoids losing or duplicating items, and a step-by-step workflow you can reuse on any site.

What is the difference between pagination, infinite scroll, and load-more buttons?#

These are three patterns sites use to split a long list into chunks. Pagination shows a fixed set of rows per page with numbered links. Infinite scroll adds rows automatically as you reach the bottom. A "Load more" button waits for a click before adding the next chunk. Each needs a slightly different collection strategy.

Here is how they compare:

Pattern	How more rows appear	How to collect all of it
Numbered pagination	You click page 2, 3, 4… or a "Next" link	Follow the next-page control, merge each page
URL-based pages	The address bar changes (?page=2)	Visit each numbered URL in turn
Infinite scroll	Rows append as you scroll down	Auto-scroll and wait until the list stops growing
Load-more button	Rows append after you click a button	Click the button repeatedly until it disappears

Some sites mix patterns — for example, infinite scroll that switches to a "Load more" button after a few screens. A good collector handles each transition without you noticing.

Why is one screen of data never the full dataset?#

What you see on screen is only the rows the site has rendered so far. Modern pages load a small batch first for speed, then fetch the rest in the background as you scroll or click. So copying the visible page captures maybe 20 rows out of hundreds. The rest never existed in the HTML you grabbed.

This is the core trap with simple methods. Copy/paste, Google Sheets IMPORTHTML, and Excel's Data → From Web (Power Query) all read the initial HTML. They cannot scroll, click, or wait for JavaScript to append more rows. On a static <table> they work fine. On a feed, a job board, or a product grid that grows as you move, they stop at the first batch.

An in-page browser extension has a structural advantage. It reads the live DOM — the page after the site's scripts run — and it can drive the same scroll and click actions a person would. That is why it sees every row instead of the first chunk.

How does auto-scroll collection work without losing rows?#

Auto-scroll collection scrolls the page down in steps, pauses after each step so new rows can load, reads every matching item, then scrolls again. It repeats this loop until the row count stops increasing for several tries in a row. Reading happens during the loop, so rows are captured even if the site later removes them from view.

The tricky part is virtualized or "windowed" lists. To stay fast, some sites keep only the visible rows in the DOM and discard rows you scroll past. If a tool waited until the end to read, those early rows would be gone. The fix is to read continuously, on every scroll step, and keep a running collection in memory. That way a row is recorded the moment it appears, before the site can recycle it.

A reliable auto-scroll routine does three things:

• Scrolls in small steps, not one jump, so lazy-loaded content has time to fetch.
• Waits for the network and DOM to settle after each step before reading.
• Stops on a stable count — if no new rows appear after a set number of tries, the list is exhausted.

This is the same idea covered in Export Any Website to CSV Without Code, applied specifically to lists that grow as you move down them.

How do you detect and follow the next-page link automatically?#

A collector finds the next-page control by looking for common signals: a link or button labeled "Next", a rel="next" attribute, numbered page links, or a ?page= parameter in the URL. It clicks or visits the next page, waits for the new rows to render, reads them with the same column pattern, and repeats until there is no next control left.

The three pagination styles each have a tell:

1. Numbered links — a row of page numbers, often with a "Next" or arrow at the end. The tool clicks the next number and re-reads the list using the pattern it already learned.
2. Load-more buttons — a single button that appends rows in place. The tool clicks it, waits, reads the new rows, and clicks again until the button vanishes or stops adding items.
3. URL-based pages — the address changes to ?page=2, &start=20, or a path like /page/3. The tool can step the number directly and load each page in sequence.

Because the same repeating pattern applies on every page, the column layout (title, price, link, and so on) stays consistent from page 1 to the last page. The tool does not need to re-learn the structure each time — it reuses the pattern and only swaps in the new rows.

How do you collect a full infinite-scroll list to CSV, step by step?#

Open the list, let the tool detect the repeating item, confirm the columns, then run collection so it scrolls and follows pages on its own. When it finishes, review the rows and export. The whole flow takes about a minute on most pages and works the same on a feed, a directory, or a search results grid.

Here is the reusable workflow with a pattern-detecting extension like Calquer:

1. Open the page with the list you want and scroll once so the first batch of rows is visible.
2. Launch the tool and let it auto-detect the repeating items. It highlights the largest repeating group, which is usually the list you care about. No clicking field by field.
3. Confirm the pattern and pick columns. Toggle off fields you don't need and rename columns in plain English. If it locked onto the wrong list, click one item to re-teach it. For native tables and ARIA grids, columns are named from the real headers.
4. Start collection. The tool auto-scrolls lazy-loaded lists and follows the "Next" control or numbered pages, gathering every matching row across the whole dataset.
5. Review and export. Open the results, scan for anything odd, and download as CSV (or JSON or TSV). The file opens directly in Excel or Google Sheets.

This same staged flow handles a 50-page job board or an endless social feed — see the worked example in Scrape Job Listings to a Spreadsheet.

How do you avoid duplicates and know when to stop?#

Deduplicate by treating a stable field — usually the item's link or a unique ID — as a key, and skip any row whose key you've already seen. To know when to stop, watch the row count: when scrolling or paging stops adding new unique rows for several tries, the list is finished. Both checks run automatically during collection.

Duplicates creep in for a few reasons:

• Overlapping pages — page 2 sometimes repeats the last row of page 1.
• Re-reading on scroll — a virtualized list may show the same row twice as it recycles.
• Sponsored or pinned items — an ad or "featured" row that appears on every page.

Keying on the link (or a clear ID) removes all three, because the same item resolves to the same key no matter how many times it shows up. The stop condition matters just as much: without it, an infinite feed never ends. A count that holds steady across several scroll-and-wait cycles is the signal that there is nothing left to load.

How do you stay polite and not hammer the site?#

Collect at a human pace: scroll and click with short pauses between actions, and read the data the page already loaded for you rather than firing rapid background requests. Because an in-page tool reuses your own logged-in session and the rows the site sent to your browser, it behaves like a fast reader — not a flood of automated traffic.

A few good habits keep collection respectful and reliable:

• Pace your actions. Brief waits between scrolls and page clicks give the server time and avoid tripping rate limits.
• Take only what you can see. Reading the rendered page is far gentler than scripting hundreds of direct API calls.
• Check robots.txt and the site's terms. They signal what the operator is comfortable with. They are not the whole legal picture, but they matter.
• Avoid logins and paywalls you don't have rights to. Public, visible data is the safe zone.

The legal landscape is nuanced. U.S. cases like hiQ Labs v. LinkedIn dealt with public data, and privacy laws such as the GDPR and CCPA govern personal data regardless of how you collect it. For a plain-English overview, read is web scraping legal.

How do you troubleshoot missing rows, stalls, and lazy images?#

Most issues trace back to timing or pattern detection. If rows are missing, the tool likely read too fast or stopped early; slow the pace or raise the stop threshold. If collection stalls, the next-page control changed shape or a button stopped appearing. If images are blank, they are lazy-loaded and need a scroll pass before the real source loads.

Common symptoms and fixes:

• "It stopped after 20 rows." The list uses scroll or pagination and collection wasn't following it. Confirm collection across pages is enabled, then re-run.
• "Some rows are missing." A virtualized list recycled rows before they were read. Scroll more slowly, or use a tool that reads on every step.
• "Collection stalled mid-run." The "Next" control disappeared or changed label, or an overlay blocked the click. Dismiss pop-ups, then resume.
• "Image columns are empty." Images load only when scrolled into view. A full scroll pass before reading fills in the real src values.
• "Duplicate rows in the file." Pagination overlapped. Dedupe on the link column in your spreadsheet, or let the tool key on the link.

For lists that are actually clean <table> elements rather than infinite feeds, a simpler path may be enough — see Scrape an HTML Table to CSV in Chrome.

FAQ#

Can I scrape an infinite scroll page to CSV without code?#

Yes. A pattern-detecting browser extension auto-scrolls the page, waits for new rows to load, reads each one, and stops when the list stops growing. It then exports everything to CSV, JSON, or TSV. There is no Python, XPath, or CSS selector to write — you point at the list and run.

How do I scrape all pages of a paginated list at once?#

Use a tool that detects the next-page control — a "Next" link, numbered pages, or a ?page= URL — and follows it automatically. It reads each page with the same column pattern and merges every page into one file. You start it once and it works through the pages on its own.

Why does my scraper only capture the first page?#

Because simple importers read the initial HTML and can't scroll or click. Sites load most rows later with JavaScript, so the first batch is all a static reader sees. A tool that drives the live page — scrolling and following pagination — captures the rest of the rows the static method misses.

How do I stop duplicate rows when scraping multiple pages in Chrome?#

Deduplicate on a stable field, usually the item's link or ID. The collector skips any row whose key it has already recorded, so overlapping pages and recycled rows resolve to a single entry. If duplicates slip through, sort by the link column in your spreadsheet and remove repeats.

Will this work on a logged-in feed or dashboard?#

Yes, if you can already see the data. An in-page extension reads the page after the site's scripts run, using your existing session, so it captures rows behind a login the same way it reads a public list. Only collect data you have the right to access and export.

---

Ready to try it? Add Calquer to Chrome and capture the next endless list — every page, every scroll — as a clean CSV, free and 100% in your browser.