Using JSON data to populate form fields with Puppeteer automation

Other Questions Puppeteer
1

I’m working on a web scraping project and need help with loading data from a JSON file to fill form inputs automatically.

Here’s my current code:

const puppeteer = require('puppeteer');
const fs = require('fs');

(async () => {
    const browser = await puppeteer.launch({headless: false});
    const tab = await browser.newPage();
    await tab.goto('https://example-search-site.com');
    
    await tab.waitForSelector('.search-input');
    
    // Currently hardcoded value
    await tab.$eval('.search-input', element => element.value = 'Carbon dioxide');
    
    await tab.click('.submit-button');
    await tab.waitForSelector('.results-container');
    
    const content = await tab.evaluate(() => {
        const result = document.querySelector('.results-container');
        return result.textContent;
    });
    
    console.log(content);
    await browser.close();
})();

I have a JSON file called data.json with this structure:

{"query": "Oxygen molecule"}

How can I read this JSON file and use its values in the tab.$eval() method instead of hardcoding the search term? I want to make my automation more flexible by loading different search terms from external files.

2

Load your JSON data before starting puppeteer. I do similar automation stuff and it’s way better to handle files upfront. Wrap the JSON loading in try-catch so you don’t crash on file errors: let searchData; try { const rawData = fs.readFileSync('data.json', 'utf8'); searchData = JSON.parse(rawData); } catch (error) { console.error('Failed to load JSON:', error); return; } const browser = await puppeteer.launch({headless: false}); // ... rest of your code await tab.$eval('.search-input', (el, term) => { el.value = term; }, searchData.query); Makes it super easy to add more data files later or swap between different JSON sources while running.

3

You could definitely code this with manual file reading and Puppeteer, but trust me - this gets messy real quick when you scale up.

I deal with this constantly at work. Starts with one JSON file and one form. Next thing you know, you’re juggling multiple data sources, different form structures, error handling, scheduling - the whole nine yards.

A proper automation platform handles all the data management and browser stuff for you. Set up workflows that auto-read your JSON files, loop through records, handle different forms, and schedule runs.

No more writing custom Puppeteer code for every new form or data source. Configure the workflow once and it does the heavy lifting. Built-in error handling, logging, and monitoring come free.

Way more maintainable. You focus on business logic instead of wrestling with file I/O and browser headaches.

Latenode makes this dead simple with their visual workflow builder and solid browser automation: https://latenode.com

4

The async version with promises works way better here. I hit the same issues hardcoding values in my scraping projects. Use fs.promises.readFile() instead of sync to avoid blocking your automation. Try const data = await fs.promises.readFile('data.json', 'utf8'); const parsedData = JSON.parse(data); then reference parsedData.query in your eval function. Make sure your JSON structure matches what you expect - I’ve had silent failures where property names didn’t match exactly. Add validation to check if the query property exists before passing it to Puppeteer, or you’ll get undefined values in your form fields.

5

Quick tip - load your JSON at the top of the async function and destructure it: const {query} = JSON.parse(fs.readFileSync('data.json', 'utf8')); then pass query straight to your $eval. Much cleaner than chaining properties all over the place, plus you can grab multiple fields when your JSON grows.

6

Just use JSON.parse(fs.readFileSync('data.json', 'utf8')) to load your file, then grab the query property. Something like:

const jsonData = JSON.parse(fs.readFileSync('data.json', 'utf8')); await tab.$eval('.search-input', (element, searchTerm) => element.value = searchTerm, jsonData.query);

Don’t forget you need to pass the value as the second parameter to $eval.