How To Use Python Requests Module Example

Python requests module is a http client library, which is similar to urllib and urllib2. It is more convenient than urllib and can save us a lot of work.

1. Install Python Requests Module.

Install via pip command like below.

$ pip install requests

You can also install it from source code.

# Get request module source code.
$ wget https://github.com/requests/requests/tarball/master

# Rename the download file to zip file name. 
$ mv master requests.tgz

# Unzip the zip file.
$ tar xzf requests.tgz

# Go to the unzip folder.
$ cd requests-requests-3dc84cd

# Install it.
$ python setup.py install

2. Send Request Use Requests Module.

Use python requests module to send network request is very simple.

# Import requests module.
>>> import requests

# Browse request url and generate response object.
>>> response = requests.get('http://www.code-learner.com')

# Get response status code.
>>> response.status_code 
200

# Get response header value.
>>> response.headers['content-type']
'text/html; charset=utf8'

# Get response encoding value.
>>> response.encoding
'utf-8'

# Get response page content.
>>> response.content

3. Pass Parameters To URL.

You may often want to pass some kind of data to the query string of a URL. If you build the URL manually, the data is placed in the URL as a key/value pair that follow a question mark. For example, www.google.com?q=python3.

Requests module allow you to use params keyword to provide request parameters ( key=value pair ) in a dictionary. For example, if you want to pass version=python3 and keywords=request to www.google.com, you can use following code:

# Create a python dictionary object.
>>> data = {'version': 'python3', 'keywords': 'request'}

# Use request module to get request url and pass above dictionary object as query string.
>>> result = requests.get("http://www.google.com", params=data)

# Print out the returned status code.
>>> print result.status_code
200

4. Get Original Response Content Data.

In rare cases you may want to get the original socket response from the server, so you can use response object’s raw attribute to get it. If you really want to do this, make sure you set stream=True in the initial request. Below is the example.

# Get request url and set stream=True.
>>> result = requests.get('http://www.code-learner.com', stream=True)

# Now get response raw content data by invoke response object's raw attribute. 
>>> result.raw
<requests.packages.urllib3.response.HTTPResponse object at 0x101194810>

# Get first 10 raw characters in unicode encoding.
>>> result.raw.read(10)
'\x1f\x8b\x08\x00\x00\x00\x00\x00\x00\x03'

5. Customize Request Header.

If you want to add http headers to a request, simply pass a dictionary object to the headers parameter. In below example we will specify a http header content-type when send post request to the web server.

# First import python requests module.
>>> import requests

# Declare a url string.
>>> url = 'http://www.code-learner.com'

# Create a dictionary object which holds post data.
>>> post_data = {'version': 'python3', 'keywords': 'python'}

# Create a content-type header, the header value is application/json which told web server the post data is json format.
>>> headers = {'content-type': 'application/json'}

# Send post request to web server with the customized header and post data.
>>> result = requests.post(url, data=json.dumps(post_data), headers=headers)

6. Error & Exception.

All exceptions explicitly thrown by requests module inherit from requests.exceptions.RequestException class. When you encounter network problems (such as failed DNS query, rejected connection, etc.), requests module will throw a ConnectionError exception. When you encounter a rare invalid HTTP response error, requests module will throw a HTTPError exception. If the request times out, a Timeout exception will be thrown. If the request exceeds the configured maximum number of redirections, a TooManyRedirects exception will be thrown.

7. Other Common Methods.

  1. response.content : Get binary response content.
  2. response.json() : Get response data in json format.
  3. response.headers : Get response headers value.
  4. response.cookies : Get cookies value.
  5. response.history : Get request history.