Selenium WebDriver: How to navigate through data table on non-scrollable webpage?

Other Questions Airtable
1

I’m struggling with extracting data from a specific webpage that contains a data table. The main issue is that I cannot scroll through the table to access all the records. The page itself doesn’t scroll normally like most websites.

I’ve attempted multiple approaches including JavaScript execution and ActionChains, but none seem to work. Even manual JavaScript commands in the browser console don’t produce the desired scrolling effect.

Here’s my current implementation:

from selenium import webdriver
from selenium.webdriver import ActionChains
from selenium.webdriver.common.by import By
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.chrome.service import Service
import time

def extract_table_data(page_url):
    service = Service(executable_path='C:\\automation\\chromedriver.exe')
    browser_options = webdriver.ChromeOptions()
    browser = webdriver.Chrome(service=service, options=browser_options)
    browser.maximize_window()
    
    try:
        browser.get(url=page_url)
        time.sleep(8)
        
        data_container = browser.find_element(By.XPATH, '//*[@id="dataGrid"]').send_keys(Keys.ARROW_DOWN)
        ActionChains(browser).move_to_element(data_container).perform()
        
        # Alternative attempts
        # browser.execute_script("document.querySelector('#dataGrid').scrollTop = 400")
        
        # grid_element = browser.find_element(By.XPATH, '//div[@id="mainWrapper"]')
        # actions = ActionChains(browser)
        # actions.move_to_element(grid_element).perform()
        # browser.execute_script('window.scrollTo(0, 250)', grid_element)
        
    except Exception as error:
        print(error)
    finally:
        browser.close()
        browser.quit()

def run_script():
    extract_table_data("https://example-data-table.com/embed/view123")

if __name__ == "__main__":
    run_script()

I also tried these JavaScript commands manually:
document.querySelector("#tableView").scrollTop=250
document.querySelector("#containerDiv").scrollTop=250

I believe the solution involves targeting the correct HTML element that controls the table’s internal scrolling mechanism, but I’m having trouble identifying it.

2

Had this exact headache with a financial dashboard at my previous job. Table looked normal but wouldn’t scroll at all.

Turns out it was a focus issue. The table needs to be actively focused before scroll commands work. Click the table first, then scroll:

# Click the table to give it focus
table_element = browser.find_element(By.ID, "dataGrid")
table_element.click()
time.sleep(1)

# Now try scrolling
browser.execute_script("arguments[0].scrollTop = 400", table_element)

Also check if your table has an internal scrollable div. Inspect the structure and look for elements with overflow: auto or overflow: scroll in their CSS. The actual scrollable container might be nested inside your dataGrid.

Another trick that worked for me - keyboard navigation after focusing:

table_element.click()
for i in range(10):  # Scroll down 10 rows
    table_element.send_keys(Keys.ARROW_DOWN)
    time.sleep(0.1)

Some tables respond better to keyboard events than scroll events. Try both.

3

I hit the same issue with a data grid using virtual scrolling. You need to figure out if your table uses pagination or virtual scrolling instead of regular scrolling. Most modern data grids don’t actually scroll DOM elements - they just load data in chunks.

First, check the network tab in dev tools while manually navigating the table. Look for AJAX calls or API requests when new data loads. Sometimes there are hidden pagination buttons or you need to trigger specific events on the table rows.

For mine, I had to use browser.execute_script("arguments[0].dispatchEvent(new Event('scroll'))", table_element) to trigger the scroll event programmatically instead of actually scrolling. Also check if there’s a CSS class or attribute that changes when you interact with different table parts. The scrolling might be controlled by a parent container that’s not obvious from the table structure.