Bulk importing CSV or Google Sheets data into App Engine database

Other Questions Google docs
1

I’m working on a project using webapp2 on Google App Engine. My database model looks like this:

class Game(db.Model):
    date = db.DateProperty()
    time = db.TimeProperty()
    home_team = db.StringProperty()
    away_team = db.StringProperty()
    location = db.StringProperty()
    score = db.IntegerProperty()

I’ve got all the data (about 150 entries) in a CSV file or Google spreadsheet. Entering each record by hand would take forever. Is there a way to import this data into my App Engine database quickly? I’m looking for an efficient method to populate my database without manual input. Any tips or tricks would be really helpful!

2

Having dealt with similar data import tasks, I can suggest using the Google Cloud Storage and Datastore Admin console method. It’s straightforward and doesn’t require additional tools.

First, convert your CSV or Google Sheet to a CSV file. Then, upload it to Google Cloud Storage. In the Datastore Admin console, you’ll find an option to import entities. Select your CSV file from Cloud Storage and map the columns to your Game model properties.

This approach is efficient for datasets of your size and integrates well with App Engine. It also provides a user-friendly interface for mapping fields, which can save time and reduce errors compared to manual configuration methods.

Remember to back up your existing data before importing, just in case. Also, double-check your data types match your model to avoid any import hiccups.

3

I’ve faced a similar challenge before, and I can share what worked for me. For bulk importing data into App Engine, I found the bulkloader tool to be incredibly useful. It’s part of the App Engine SDK and designed specifically for this purpose.

Here’s a quick rundown of the process:

  1. Create a bulkloader.yaml file that maps your CSV columns to your Game model properties.
  2. Use the appcfg.py upload_data command to import your data.

The bulkloader is quite flexible and can handle various data formats, including CSV. It’s also efficient for large datasets.

One tip: make sure your CSV data is clean and matches your model’s property types. I spent more time cleaning my data than actually importing it!

If you prefer working with Google Sheets, you can export it as a CSV first, then use the bulkloader. Alternatively, you could write a custom script using the Google Sheets API to read the data and create entities programmatically.

Hope this helps point you in the right direction!

4

yo, i’ve done this before. use the google app engine remote api. it’s pretty sweet. write a python script that connects to ur app, reads the csv, and creates Game objects. then use db.put() to save em all at once. way faster than manual entry. just make sure ur csv data matches ur model properties or you’ll get errors. good luck!