I’m working on a Discord bot project and running into some issues. I’m using a local database (nedb) to store message data. My main problem is completing a sync function that needs to grab all messages from specific users on a given date.
The tricky part is that I need to check each message against what’s already in my database. If a message doesn’t exist or if the existing one has an older edited_date, then I should add the new one. I’ve been stuck on this for a while and could really use some guidance from someone who has experience with this kind of thing.
const axios = require('axios');
const dayjs = require('dayjs');
const settings = require('./settings');
const params = require("url");
var Database = require('nedb');
var storage = new Database({ filename: 'messages.db', autoload: true });
const MULTIPLIER = 4194304 * 1000 * 60 * 60 * 24;
const BASE_DATE = dayjs('2018-04-30');
const BASE_ID = 440557948108800000;
// sample user insertion
// var user = {
// username: 'John',
// handle: '@JohnDoe'
// };
// storage.insert(user, function (error, result) {
// console.log('Added', result.username, 'with ID', result._id);
// });
// retrieve all records
// storage.find({}).exec(function (error, results) {
// console.log(results);
// results.forEach(function (item) {
// console.log('Located user:', item.username);
// });
// });
const GENERATE_ID = function (dateObj) {
return BASE_ID + dateObj.diff(BASE_DATE, "days") * MULTIPLIER;
}
class ChatAPI {
constructor() {}
fetchData(targetDate, userId, start = 0) {
const requestUrl =
settings.apiUrl +
params.stringify({
min_id: GENERATE_ID(targetDate),
user_id: userId
});
return axios.get(requestUrl, {
headers: {
Authorization: settings.token
}
}).then(response => {
console.log(response.config.url);
return response.data;
});
}
// HELP NEEDED: complete this method to download and store conversations
// need to verify if messages exist in database and compare edited_date
// insert new or updated messages only
syncMessages(targetDate) {
settings.userList.map(userId => {
this.fetchData(targetDate, userId).then(({ count, conversations }) => {
console.log(conversations);
conversations.map(chat => {
chat.map(({ files, sender, text, created_at }) => {
console.log(created_at, sender.name, text);
});
});
});
});
}
}
module.exports = new ChatAPI();
I hit the same issue building a message sync bot. You’re logging the messages but not actually doing the database comparison. Inside that innermost map where you console.log, add the storage operations: storage.findOne({senderId: sender.id, createdAt: created_at}, (err, existingMsg) => { if (!existingMsg || new Date(existingMsg.editedDate) < new Date(edited_date)) { storage.insert({senderId: sender.id, text, createdAt: created_at, editedDate: edited_date}); } }). Your async flow’s broken too - map functions don’t wait for database ops. I switched to async/await and processed messages one by one instead of handling multiple concurrent writes. Also, you’re missing edited_date in your destructuring but you’ll need it for the comparison.
Your main problem is the async handling in syncMessages. You’re using map() instead of forEach() or a proper async loop, so it won’t wait for your database operations to finish. I’d restructure it with async/await and process messages one by one. For the database comparison, create a unique identifier for each message - maybe combine sender ID and created_at timestamp. Query your storage with that ID, then check if the existing record has an older edited_date than your incoming message. Something like const existing = await storage.findOne({messageId: uniqueId}) then compare dates before inserting. Also add error handling around your database operations since network issues can make the sync fail silently.
your syncMessages method doesn’t actually compare the database. use storage.findOne() to check if each message exists, then compare edited_date before running storage.insert(). and make sure you’re handling async correctly with await or proper promise chains.
you’re missing the actual database insert logic. after you console.log the message data, add something like storage.findOne({sender_id: sender.id, created_at}, (err, doc) => { if (!doc || doc.edited_date < edited_date) storage.insert({...messageData}) }). also, those nested maps won’t wait for database operations to finish.
Yeah, database syncing is tricky, but you’re overcomplicating it.
Ditch the nested async mess and manual database comparisons. Just automate the whole thing - API calls, database checks, message updates, all of it.
I handle similar Discord syncs at work. The trick is proper async handling with parallel database comparisons. Queue up user requests, fetch data, then batch process everything.
Those nested maps you’re using? They’ll cause timing issues and make debugging hell. You’re also not handling Discord’s rate limits or database connection failures.
A good automation platform handles error cases, retries, and async coordination for you. Just define the workflow: fetch data, compare records, update when needed.
I sync data across multiple services this way and it eliminates hours of async callback debugging. The workflow manages all database queries and API calls automatically.
Latenode works great for this kind of Discord bot automation: https://latenode.com
Your code’s mapping over conversations but not actually checking the database. You need to add the database queries inside those nested map functions. For each message, use storage.findOne() with sender.id and created_at as your unique identifier. Then compare the edited_date field if a record exists. If there’s no record or the existing one has an older edited_date, run storage.insert(). I’d also wrap your database operations in try-catch blocks and switch to async/await - you’ve got multiple async operations happening and the current nested maps make error handling a pain.