How to send HTTP requests to REST API endpoints with Python

Other Questions API
1

I need help converting a curl command into Python code. I have an Elasticsearch service running on AWS that indexes my documents. Currently I can search it using this curl command from my Mac terminal:

curl -XGET 'http://my-search-api.com/content/item/_search?pretty=true' -d '{
  "query": {
    "bool": {
      "must": [
        {
          "match": {
            "item.source": "TECH_MAGAZINE"
          }
        },
        {
          "match": {
            "item.title": "developers"
          }
        }
      ],
      "must_not": [],
      "should": []
    }
  },
  "from": 0,
  "size": 25,
  "sort": [],
  "facets": {}
}'

What’s the best way to make this same request using Python? Should I use the requests library or urllib2? I’ve been working with urllib2 but people say requests is easier. How do I handle the JSON payload - does it go in headers or request body?

2

Definitely go with requests over urllib2 - it’s way more straightforward and handles JSON automatically. I made that switch years ago and never looked back. Just remember the JSON payload goes in the request body, not headers. Here’s your curl converted to Python:

import requests
import json

url = 'http://my-search-api.com/content/item/_search?pretty=true'
headers = {'Content-Type': 'application/json'}

query_data = {
    "query": {
        "bool": {
            "must": [
                {"match": {"item.source": "TECH_MAGAZINE"}},
                {"match": {"item.title": "developers"}}
            ],
            "must_not": [],
            "should": []
        }
    },
    "from": 0,
    "size": 25,
    "sort": [],
    "facets": {}
}

response = requests.get(url, headers=headers, json=query_data)
result = response.json()

The json parameter does all the heavy lifting - automatically serializes your dictionary and sets the content type. So much cleaner than wrestling with JSON encoding in urllib2.

3

requests is definitely the way to go - urllib2’s pretty outdated. Just add some error handling since elasticsearch can be finicky. Check if response.status_code is 200 before parsing the json. Trust me, it’ll save you tons of debugging headaches.

4

I’ve been doing Elasticsearch queries with Python for about 3 years and hit the same issue. The requests library works fine, but I’d recommend the official elasticsearch-py client instead. I switched after running into auth issues and connection pooling problems when hitting ES clusters frequently. The client handles retries, connection management, and supports all ES operations out of the box. Your query would look like this:

from elasticsearch import Elasticsearch

es = Elasticsearch(['http://my-search-api.com'])
response = es.search(
    index='content',
    doc_type='item',
    body={
        "query": {
            "bool": {
                "must": [
                    {"match": {"item.source": "TECH_MAGAZINE"}},
                    {"match": {"item.title": "developers"}}
                ]
            }
        },
        "from": 0,
        "size": 25
    }
)

Really helpful if you’re planning more complex ES operations beyond basic searches. Raw requests works fine for simple stuff though.