Stopping duplicate entries when adding new students to Airtable via API

Other Questions Airtable
1

I’m having trouble with my script that adds new students to Airtable. It works but it’s making duplicate entries. I’ve checked the data I’m sending and it looks fine. Here’s what I’m doing:

  1. I look through a list of students and find the new ones.
  2. I make a list of these new students.
  3. I use a function to send this list to Airtable.

Here’s a simplified version of my code:

function findNewStudents(allStudents, existingStudents) {
  let newOnes = [];
  for (let kid of allStudents) {
    if (kid.isActive && !existingStudents.includes(kid.id)) {
      newOnes.push(kid);
    }
  }
  return newOnes;
}

function sendToAirtable(newStudents) {
  const url = 'https://api.airtable.com/v0/mybase/Students';
  const options = {
    method: 'POST',
    headers: { 'Authorization': 'Bearer mytoken', 'Content-Type': 'application/json' },
    body: JSON.stringify({ records: newStudents })
  };
  fetch(url, options);
}

const newKids = findNewStudents(allStudents, airtableStudents);
sendToAirtable(newKids);

Any ideas why this might be creating duplicates? Thanks for your help!

2

I’ve encountered a similar issue before. The problem might be in your comparison logic. Instead of using includes() with just the ID, try creating a Set of existing student IDs for faster lookup. Also, consider implementing a check in your Airtable base to prevent duplicates.

Here’s a modified version of your findNewStudents function that might help:

function findNewStudents(allStudents, existingStudents) {
  const existingIds = new Set(existingStudents.map(student => student.id));
  return allStudents.filter(kid => kid.isActive && !existingIds.has(kid.id));
}

This approach is more efficient and less prone to errors. Additionally, you might want to add error handling in your sendToAirtable function to catch and log any API errors. This could provide more insight into what’s happening during the data transfer.

3

hey there, i had a similar problem. maybe try adding a delay between api calls? sometimes airtable can’t keep up if you’re sending a bunch of requests at once. also, double-check ur existingStudents list - make sure it’s up to date before comparing. good luck!

4

I’ve dealt with this exact issue before. The problem might be in the asynchronous nature of your API calls. Your script could be sending multiple requests before the previous ones have been processed, leading to duplicates.

Try implementing a queue system for your API calls. This way, you can ensure each request is completed before sending the next one. Here’s a basic example:

async function sendToAirtableQueue(newStudents) {
  for (const student of newStudents) {
    await sendSingleStudent(student);
    await new Promise(resolve => setTimeout(resolve, 1000)); // 1 second delay
  }
}

async function sendSingleStudent(student) {
  const url = 'https://api.airtable.com/v0/mybase/Students';
  const options = {
    method: 'POST',
    headers: { 'Authorization': 'Bearer mytoken', 'Content-Type': 'application/json' },
    body: JSON.stringify({ records: [student] })
  };
  return fetch(url, options);
}

This approach sends students one by one with a small delay between each request. It might take longer, but it should prevent duplicates caused by race conditions. Also, consider implementing a unique constraint in your Airtable base if possible.