How to assign bigquery.datasets.create permission for automated Vertex AI notebook execution

Other Questions ai employee
1

I’m working with a Vertex AI notebook that processes API data and stores results in both BigQuery tables and Cloud Storage buckets. The notebook runs perfectly when executed manually, but fails during scheduled execution.

The error I encounter is:

GenericGBQException: Reason: 403 POST https://bigquery.googleapis.com/bigquery/v2/projects/my-project/datasets: Access Denied: Project my-project: User does not have bigquery.datasets.create permission in project my-project

Here’s the relevant code section that triggers the issue:

# Writing DataFrame to BigQuery
processed_data.to_gbq('analytics.results_table', project_context.get_project_id(), 
                      chunksize=5000, if_exists='replace')

# Saving to Cloud Storage
processed_data.to_csv('gs://my-storage-bucket/' + 'results_' + location_id + '_' + timestamp + '.csv', 
                      sep=',', encoding='utf-8')

# Creating visualization
visualization = processed_data.plot.bar(x="category", y="count", 
                                       title="Data Analysis for {}".format(timestamp))

def upload_chart(file_path):
    chart_figure = visualization.get_figure()
    buffer = io.BytesIO()
    chart_figure.savefig(buffer, format='png')
    buffer.seek(0)
    
    storage_client = Storage.Client()
    target_bucket = storage_client.get_bucket('chart-storage-bucket')
    chart_blob = target_bucket.blob('analysis_chart_' + location_id + '.png')
    chart_blob.upload_from_string(buffer.getvalue(), content_type='image/png')

upload_chart('chart_output.png')

I realize that scheduled notebooks run under different service account credentials than manual execution. How can I properly configure the necessary permissions for the scheduled notebook to create BigQuery datasets automatically?

2

Hit this same issue. You need to check which service account your scheduled notebook is actually running under - it’s usually not the obvious one. Go to Vertex AI Workbench settings and look at the service account field. Then give that account BigQuery Admin role in IAM console. BigQuery Data Editor won’t work for creating datasets. Also verify your project-level permissions since that’s often the real problem even when BigQuery roles seem correct.

3

The 403 error happens because your scheduled notebook runs with different permissions than manual runs. Hit this exact issue last year.

Check which service account your scheduled notebook uses. Go to Vertex AI Workbench, find your notebook instance, and check the service account field.

Two main options:

Option 1: Grant permissions to existing service account
Go to IAM in Google Cloud Console and find that service account. Add:

  • BigQuery Admin (dataset creation rights)
  • Storage Admin (bucket uploads)

Option 2: Create custom service account
I prefer this. Create a new service account just for your notebook with only needed permissions. Then update your notebook instance to use it.

Don’t just add BigQuery Data Editor like some guides say. You need dataset creation permissions, which requires BigQuery Admin or a custom role with bigquery.datasets.create.

Make sure your service account has permissions for both BigQuery datasets AND the specific tables you’re writing to. Sometimes the dataset exists but the service account can’t write tables.

Restart your notebook after IAM changes. Permissions don’t always refresh immediately.

4

Same frustrating experience here when I switched from interactive to scheduled execution. Most people miss the permission hierarchy - when you schedule a Vertex AI notebook, it defaults to the Compute Engine service account which barely has any permissions. You’ve got to modify it or create a new one. For your error, the service account needs BigQuery Admin role, not just Data Editor. The datasets.create permission only comes with BigQuery Admin or custom roles that specifically include it. Here’s a gotcha I hit - even after assigning roles, permissions can take time to kick in. Wait 10-15 minutes after making changes before testing your scheduled notebook. Also check your project-level permissions. Sometimes the service account has BigQuery access but lacks the broader project permissions needed for dataset creation. Your error mentions project-level access denial so that might be it. For production stuff, I’d create a custom service account instead of tweaking the default Compute Engine one. Gives you better control and follows least privilege.

5

This issue arises because your scheduled Vertex AI notebook operates under the Compute Engine default service account instead of your personal credentials. To resolve this, navigate to your notebook’s settings in Vertex AI Workbench to confirm the service account in use. Subsequently, access IAM & Admin in the Cloud Console, locate the relevant service account, and assign it the “BigQuery Data Editor” role. This will grant the required datasets.create permission. Alternatively, you can download a service account key and set the GOOGLE_APPLICATION_CREDENTIALS environment variable in your code, although using IAM is the preferred and more secure approach for scheduled executions. I’ve encountered this challenge myself when transitioning from manual to scheduled notebook runs, as differing permissions between the two modes can be quite confusing.