I’m building a web scraper that extracts content from a database page. When my script clicks on the first item, it should wait for all entries to appear on the page before collecting them. The problem is that it only captures 4 entries instead of the full 25 that should be there.
I can get all 25 entries if I add a fixed delay like time.sleep(10), but this makes the script really slow. Is there a better way to wait for all the content to load dynamically?
class WebExtractor:
def __init__(self):
pass
def setupDriver(self):
options = Options()
self.browser = webdriver.Chrome(service=Service(ChromeDriverManager().install()), options=options)
def loadPage(self):
self.browser.get('https://example-database-url.com')
def selectFirstItem(self):
items = WebDriverWait(self.browser, 10).until(
EC.presence_of_all_elements_located((By.XPATH, '//div[@class="item-container"]')))
items[0].click()
def extractContent(self):
contentElements = WebDriverWait(self.browser, 10).until(
EC.presence_of_all_elements_located((By.XPATH, '//div[@placeholder="No content"]')))
results = []
for element in contentElements:
results.append(element.text)
print(f"Found {len(contentElements)} items")
return results
extractor = WebExtractor()
extractor.setupDriver()
extractor.loadPage()
extractor.selectFirstItem()
extractor.extractContent()
wait for the last element you expect instead of just any random elements. I use EC.presence_of_element_located with an xpath targeting the 25th item directly, or EC.element_to_be_clickable on the bottom element. way better than counting - it actually waits for everything to load properly.
This happens all the time with progressively loaded content. Your presence_of_all_elements_located only waits for the first batch - not everything that’s coming.
I hit the same issue scraping a product catalog that loaded in chunks. Here’s what fixed it: create a custom wait that monitors when the element count stops changing. Don’t wait for elements to appear - wait for the count to stabilize.
Set up a loop that counts current elements, waits briefly, then counts again. Keep going until the number stays the same for a few rounds. Way more reliable than fixed delays and handles different load speeds.
One more thing - check if the page uses infinite scroll or pagination triggers. Sometimes you’ve got to scroll to the bottom first to trigger the remaining items before your wait condition will work.
Yeah, this is super common with AJAX content. You’re waiting for elements to show up, but there’s probably still stuff loading in the background. I’ve hit this same issue scraping e-commerce sites with lazy-loaded grids. Here’s what worked for me: instead of just checking if elements exist, wait for the loading indicators to disappear. Most sites show spinners, loading bars, or placeholder text while they’re fetching data. Hunt for elements with classes like “loading” or “spinner”, or check if your placeholder divs have loading states. Use invisibility_of_element_located to wait until those disappear. You can also watch for changes to the parent container’s attributes - lots of sites update data attributes or classes when loading finishes. Pro tip: open Chrome DevTools and monitor network activity to find the actual API calls. Then just hit those endpoints directly with requests. Skips all the rendering wait time and usually runs way faster than browser automation if you just need the data.