I’m having trouble with a quota limit error in my machine learning pipeline on Google Cloud AI Platform. When I try to run my training job, I keep getting this error message:
com.google.cloud.ai.platform.common.errors.AiPlatformException: code=RESOURCE_EXHAUSTED, message=The following quota metrics exceed quota limits: aiplatform.googleapis.com/custom_model_training_cpus, cause=null; Failed to create custom job for the task.
My setup uses n1-standard-4 instances for the training process, which should be fine for my workload. I’ve looked through the documentation but can’t find clear information about this specific quota error. Has anyone else run into this issue? What steps did you take to resolve it? Any help would be great since I’m stuck on this problem.
ugh, i’ve been there too! It’s super frustrating. Just check your quota limits for aiplatform training vCPUs in IAM & Admin. If it’s maxed out, you can either ask for more or just try a different region. Hope that helps!
This happens when you’ve hit the default CPU limit for custom training jobs. Those n1-standard-4 instances eat up 4 vCPUs each, and Google sets pretty low quotas for new projects. I ran into the exact same thing last month when I was scaling up. Head to the Google Cloud Console quotas page and find the AI Platform Training quota. Hit the edit button and request more. Takes about 24-48 hours to get approved in my experience. While you wait, try switching to n1-standard-2 instances if your data can handle it, or just break your training job into smaller chunks.