How to import CSV data into Google App Engine datastore model automatically

Other Questions Google docs
1

I’m working with webapp2 framework on Google App Engine and need help with bulk data import. My current datastore model looks like this:

class GameEvent(db.Model):
    event_date = db.DateProperty()
    event_time = db.TimeProperty()
    home_team = db.StringProperty()
    away_team = db.StringProperty()
    location = db.StringProperty()
    score = db.IntegerProperty()

I have about 150 records in a CSV file that I need to load into my database. Entering each record by hand would take forever and be really tedious. What’s the best way to automatically populate my datastore from this CSV file? I’ve also considered using Google Sheets if that makes the process easier. Any suggestions for handling this bulk import would be really helpful.

2

Utilize the App Engine admin console’s bulk loader for your CSV data import. Start by creating a transform file that maps your CSV columns to the properties of your datastore model. After that, execute the appcfg.py upload_data command to upload all records simultaneously. I’ve successfully employed this method with comparable data sets, and it proves effective. Just ensure that your CSV format aligns perfectly with your model, as any discrepancies in date and time formatting could cause issues. For 150 records, this approach is far more efficient than writing individual handlers or using the web interface.

3

just write a python script to read your csv and create the model instances. use the csv module to parse each row, then loop through creating gameevent objects - something like for row in csv_reader: event = GameEvent(event_date=..., home_team=row[2]). way easier than bulkloader and you get better control over data validation.

4

Try using the Remote API to run your import remotely. Set it up in your app.yaml, then write a local script that connects to your production datastore. You can handle data transformations and validation locally while pushing directly to your live database. I found this super useful for inconsistent CSV formats - you can debug and test your parsing logic without deploying code changes. The remote API handles batching automatically, so you won’t hit timeout issues with larger datasets. For 150 records, you’ll probably finish the whole import in under a minute once your script works.