NextAuth email authentication setup with Mailgun SMTP failing

I’m having trouble setting up email-based authentication in my Next.js app using NextAuth and Mailgun as the SMTP provider. The goal is to send magic link emails for passwordless login, but I keep getting authentication errors.

Here’s my current NextAuth configuration:

import dbConnection from "@/lib/database";
import NextAuth from "next-auth";
import EmailProvider from "next-auth/providers/email";
import { MongoDBAdapter } from "@next-auth/mongodb-adapter";

const authConfig = NextAuth({
    adapter: MongoDBAdapter(dbConnection),
    providers: [
        EmailProvider({
            server: {
                host: process.env.SMTP_HOST,
                port: process.env.SMTP_PORT,
                auth: {
                    user: process.env.SMTP_USERNAME,
                    pass: process.env.SMTP_API_KEY,
                },
            },
            from: process.env.FROM_EMAIL,
            maxAge: 600, // Links expire in 10 minutes
        }),
    ],
});

export { authConfig as GET, authConfig as POST };

My environment variables look like this:

SMTP_HOST="smtp.mailgun.org"
SMTP_PORT="587"
SMTP_USERNAME="[email protected]"
SMTP_API_KEY="key-xxxxxxxxxx"

I’m getting this authentication error: message: 'Invalid login: 535 Authentication failed'

What could be causing this authentication failure? Are there specific settings needed for Mailgun SMTP with NextAuth?

Had the same issue with Mailgun and NextAuth last year. Your SMTP_USERNAME format’s probably wrong - you’re missing the sandbox domain prefix Mailgun assigns. Check your Mailgun dashboard under Domain Information. Your SMTP username should be “[email protected]” not “[email protected]”. Mailgun auto-creates this sandbox domain for new accounts. Also make sure you’re copying the Private API key from API Keys, not the domain verification key. Authentication errors usually mean credential mismatch, not config issues. Another thing - your FROM_EMAIL domain needs to match what’s verified in Mailgun. If you’re using the sandbox domain for SMTP, your from address has to use that same domain or Mailgun rejects it during handshake. Fix the username format first, then test with a simple nodemailer script before putting it back into NextAuth. Way easier to debug.

Check if your Mailgun domain’s verified first - that’s what got me. If you’re in the EU region, use smtp.eu.mailgun.org instead of the regular SMTP settings. Auth will keep failing even with the right credentials if you don’t.

Sounds like an SMTP port issue. Port 587 works for most providers, but Mailgun can be picky with NextAuth TLS settings. Try adding secure: false and tls: { rejectUnauthorized: false } to your server config. I hit the same auth failures until I got the TLS handshake right. Here’s another gotcha - your FROM_EMAIL has to exactly match an authorized sender in Mailgun’s domain settings. Even with correct SMTP credentials, Mailgun will reject auth if the from address isn’t pre-approved. Make sure your FROM_EMAIL domain matches your verified Mailgun domain, not just any email you own. Also check that SMTP sending is enabled in your Mailgun account - it’s not always on by default.

That 535 error means your Mailgun credentials are wrong. Most people mess up the username format.

Don’t use “[email protected]” - you need the full address Mailgun creates for your domain. Check Domain Settings in your dashboard. It’ll be something like “[email protected]”.

Make sure you’re using your actual API key, not the public validation key. Mailgun gives you multiple keys and it’s easy to grab the wrong one.

If 587 keeps failing, try port 465 with secure: true.

Honestly, I gave up wrestling with email configs after getting burned by deliverability issues and API changes. Now I handle all auth flows through Latenode. You can build the entire magic link system visually, connect any email service without SMTP configs, and it handles edge cases automatically.

Way cleaner than debugging NextAuth settings, plus you get better monitoring of your emails.

The Problem:

You’re receiving the message: 'Invalid login: 535 Authentication failed' error when trying to set up email-based authentication in your Next.js application using NextAuth and Mailgun. This indicates a problem with your Mailgun SMTP credentials or configuration within your NextAuth setup.

:thinking: Understanding the “Why” (The Root Cause):

The 535 Authentication failed error from Mailgun typically means that the username or password you’re providing doesn’t match what Mailgun has on record for your account. Several common mistakes lead to this:

  • Incorrect SMTP Username: The most frequent cause is an incorrectly formatted SMTP username. You must use the full address that Mailgun generates for your domain, which is usually in the format postmaster@[your_mailgun_domain]. Using [email protected] (as in your example) is almost certainly wrong. Mailgun creates a unique domain for each user, and this is what needs to be used. Failure to do this is a common source of authentication errors.

  • Wrong API Key: Mailgun provides different API keys for various purposes. You need the SMTP API key, not a general API key or a webhook key. Confirm you are using the correct key type from your Mailgun account’s API Keys section.

  • Unverified Mailgun Domain: If your Mailgun domain isn’t fully verified, authentication may fail even with the correct credentials. Ensure your domain is properly verified in your Mailgun dashboard. Look for any outstanding steps or warnings related to DNS record verification.

  • Incorrect Port or TLS Settings: While port 587 is common, Mailgun might require adjustments to your TLS settings depending on your region and server configuration.

:gear: Step-by-Step Guide:

Step 1: Verify and Correct Mailgun Credentials:

  1. Locate the Correct Username: Go to your Mailgun dashboard, find your domain settings, and locate the correct SMTP username. It will look like postmaster@<your_mailgun_domain>.com or similar. This is crucial. Do not use your primary domain (mysite.com in your example).
  2. Obtain the SMTP API Key: In your Mailgun dashboard, navigate to “API Keys” and ensure you are copying the SMTP API key, not the HTTP API key or any other.

Step 2: Update your .env file:

Replace the values in your .env file with the corrected username and API key obtained in Step 1:

SMTP_HOST="smtp.mailgun.org" # Or smtp.eu.mailgun.org if you're in the EU region.
SMTP_PORT="587"
SMTP_USERNAME="postmaster@<your_mailgun_domain>.com" #Corrected Username
SMTP_API_KEY="key-<your_actual_mailgun_api_key>" #Corrected API Key
FROM_EMAIL="<your_approved_sender>@<your_mailgun_domain>.com"  #Ensure this matches your verified Mailgun domain

Step 3: Verify Mailgun Domain Verification:

  1. Log in to your Mailgun dashboard.
  2. Go to the Domains section and check the status of your domain. Ensure there are no warnings or pending verification steps.
  3. If there are any pending DNS records, add them to your DNS settings and wait for propagation (this can take some time).

Step 4 (If necessary): Adjust TLS Settings:

If you still face issues, try modifying your EmailProvider configuration in next-auth.js to explicitly control the TLS settings:

EmailProvider({
    server: {
        host: process.env.SMTP_HOST,
        port: process.env.SMTP_PORT,
        auth: {
            user: process.env.SMTP_USERNAME,
            pass: process.env.SMTP_API_KEY,
        },
        secure: true, // Or false depending on your port and Mailgun settings.
        tls: {
            rejectUnauthorized: false, //Consider setting to true for production environment
        },
    },
    from: process.env.FROM_EMAIL,
    maxAge: 600,
}),

:mag: Common Pitfalls & What to Check Next:

  • FROM_EMAIL Domain: Verify that the domain specified in FROM_EMAIL is the same as the verified domain in your Mailgun settings. Mailgun often rejects emails from unverified domains.
  • Double-Check for Typos: Carefully review all your environment variables. A single typo can cause authentication failure.
  • Mailgun Sandbox Limitations: If you’re using a sandbox account, check your sending limits and ensure you haven’t exceeded them.
  • Port Conflicts: If using port 587 is causing persistent issues, try alternate ports (e.g., 465, but remember to adjust the secure flag accordingly).
  • NextAuth Version: Ensure you’re using a reasonably recent version of NextAuth; older versions might have compatibility problems with Mailgun.

:speech_balloon: Still running into issues? Share your (sanitized) config files, the exact command you ran, and any other relevant details. The community is here to help!

Check if your SMTP_API_KEY is missing the ‘key-’ prefix in your env file. Mailgun needs the complete key string with that part included. Also try port 25 instead - some hosts block 587 to prevent spam, and NextAuth won’t warn you about it.

Your Mailgun domain probably isn’t fully activated. I had the same issue with NextAuth - authentication kept failing because my domain was stuck in sandbox mode even though I thought it was verified. Check your Mailgun dashboard for any yellow warnings about domain verification. If you see one, add those DNS records and wait for propagation (can take up to 48 hours). SMTP credentials work differently between sandbox and verified domains. Also make sure your SMTP_API_KEY is the private API key from the API Keys section - not the webhook signing key or public validation key. I wasted hours on that mistake. If you’re running an older NextAuth version, try upgrading. Older versions had TLS issues with Mailgun’s servers.

This topic was automatically closed 24 hours after the last reply. New replies are no longer allowed.