How To Use Python Requests Module Example

Leave a Comment / Python

Python requests module is a http client library, which is similar to urllib and urllib2. It is more convenient than urllib and can save us a lot of work.

1. Install Python Requests Module.

Install via pip command like below.

$ pip install requests

You can also install it from source code.

# Get request module source code.
$ wget https://github.com/requests/requests/tarball/master

# Rename the download file to zip file name. 
$ mv master requests.tgz

# Unzip the zip file.
$ tar xzf requests.tgz

# Go to the unzip folder.
$ cd requests-requests-3dc84cd

# Install it.
$ python setup.py install

2. Send Request Use Python Requests Module.

Use python requests module to send http request is very simple.

# Import requests module.
>>> import requests

# Browse request url and generate response object.
>>> response = requests.get('http://www.code-learner.com')

# Get response status code.
>>> response.status_code 
200

# Get response header value.
>>> response.headers['content-type']
'text/html; charset=utf8'

# Get response encoding value.
>>> response.encoding
'utf-8'

# Get response page content.
>>> response.content

3. Python Requests Get & Python Requests Post Example.

3.1 Python Requests Get Example.

You can use python requests module’s get method to send a http get request like below source code.

response = requests.get("http://www.test-abc.com/")

You may often want to pass some kind of data to the query string of a URL ( request a url with http get method ). If you build the URL manually, the data is placed in the URL as a key/value pair that follows a question mark. For example, www.test-abc.com?q=python3.

Python requests module allow you to use params keyword to provide request parameters ( key=value pair ) in a dictionary. For example, if you want to pass version=python3 and keywords=request to www.test-abc.com, you can use the following code:

# Create a python dictionary object.
>>> data = {'version': 'python3', 'keywords': 'request'}

# Use request module to get request url and pass above dictionary object as query string.
>>> response = requests.get("http://www.test-abc.com", params=data)

# Print out the returned status code.
>>> print(response.status_code)
200

# You can see that the request URL is correct.
>>> print(response.url)
http://www.test-abc.com/?version=python3&keywords=request

# We can also pass a list to a request parameter. In below example, we pass a list contains value ['value2', 'value3'] to key2.
>>> req_params = {'key1': 'value1', 'key2': ['value2', 'value3']}
>>> response = requests.get("http://www.test-abc.com/test", params=req_params)
# You can see the request url contains both key2=value2&key2=value3.
>>> print(response.url)
http://www.test-abc.com/test?key1=value1&key2=value2&key2=value3

3.2 Python Requests Post Example.

You can use python requests module’s post method to send a http post request like below source code.

# Python requests module's post method has a data argument, we can pass post data value to the data argument. 
# In below example, we pass a python dictionary object to the data argument.
response = requests.post("http://www.test-abc.com/", data = {"key":"value"})

In above example, we pass a dictionary object to the data parameter. Besides this, we can also pass JSON format string to the data parameter like below.

# Import json moduel for later use.
>>> import json
>>> import requests
# Create a dictionary object that contains the post data value.
>>> params = {"key":"value"}
# Invoke json.dumps() method to convert the dictionary object to json format string.
>>> json_str = json.dumps(params)
# Invoke requests module's post method and pass the json format string to data argument.
>>> response = requests.post("http://www.test-abc.com/", data = json_str)

Because it is too common to send JSON format data, the keyword parameter json has been added to the higher version of the python requests module. You can directly send the JSON data to the post request instead of using the json module, see below example.

>>> params = {"key":"value"}
>>> 
>>> response = requests.post("http://www.test-abc.com/", json = params)

What if we want to post a file? At this time, you need to use the files parameter.

>>> url = 'http://www.test-abc.com/post_file'

# It is strongly recommended to open the file in binary mode, because if it is opened in text file format, an error may occur due to the header "content length"
>>> files = {'file': open('test_file.xls', 'rb')}

# Post files to the url.
>>> response = requests.post(url, files=files)

>>> response.text

We can also specify additional information such as the file name in the post file.

>>> url = 'http://www.test-abc.com/post_file'

>>> files = {'file': ('test_files.xls', open('test_files.xls', 'rb'), 'application/vnd.ms-excel', {'Expires': '0'})}

>>> response = requests.post(url, files=files)

4. Get Server Response Data.

The python requests module get or post method will return an object. We can get server response data through this object. This object provides some attributes such as text, encoding, content, etc. Now let’s look at how to get server returned data use this object after sending the request.

4.1 Response text Attribute.

The text attribute returns string text data.

>>> import requests
>>> response = requests.get('http://www.test-abc.com')
>>>
# The text parameter save the server returned html page source code. 
>>> response.text
'<!doctype html><html itemscope="" itemtype="http:....)();</script>        </body></html>'
>>>
# Get the returned html page source code text character encoding.
>>> response.encoding
'ISO-8859-1'

4.2 Response content Attribute.

The content attribute returns byte stream, if we request an image and want to save the image, we can use it. Below is an example.

def saveImage( imgUrl,imgName ="default.jpg" ):
    # Get the image stream.
    response = requests.get(imgUrl, stream=True)
    image = response.content

    targetDir="D:\"
    saveImagePath = targetDir+imgName  
    print("Save image in "+saveImagePath+"\n")
    try:
        # Open file and write the returned bytes stream in the file.
        with open(saveImagePath ,"wb") as jpg:
            jpg.write(image)     
            return
    except IOError:
        print("IO Error")
        return
    finally:
        jpg.close

4.3 Response json Attribute.

If the response of the request is a JSON, can I get the data in JSON format directly? Yes, the json attribute is just for this.

>>> response.json

4.4 Response raw Attribute.

In rare cases you may want to get the original socket response from the server, so you can use the response object’s raw attribute to get it. If you really want to do this, make sure you set stream=True in the initial request. Below is an example.

# Get request url and set stream=True.
>>> response = requests.get('http://www.test-abc.com', stream=True)

# Now get response raw content data by invoke response object's raw attribute. 
>>> response.raw
<requests.packages.urllib3.response.HTTPResponse object at 0x101194810>

# Get first 10 raw characters in unicode encoding.
>>> response.raw.read(10)
'\x1f\x8b\x08\x00\x00\x00\x00\x00\x00\x03'

4.5 Response status_code Attribute.

The status_code attribute returns the server response http status code.

>>> import requests
>>> 
>>> response = requests.get('http://www.test-abc.com')
>>>
>>> response.status_code
200

5. Use Python Requests Module To Handle HTTP Headers.

5.1 Get Server Response Headers.

We can get all response headers use the response object’s headers attribute.

>>> import requests
>>> 
>>> response = requests.get('http://www.test-abc.com')
>>>
>>> # The headers attribute is a dictionary object.
>>> response.headers
{'Content-Length': '5725', 'Cache-Control': 'private, max-age=0', 'Connection': 'keep-alive', 'Content-Encoding': 'gzip', 'Content-Type': 'text/html; charset=ISO-8859-1', 'Date': 'Thu, 14 Jan 2021 13:35:59 GMT', 'Expires': '-1', 'Keep-Alive': 'timeout=4', 'P3p': 'CP="This is not a P3P policy! See g.co/p3phelp for more info."', 'Proxy-Connection': 'keep-alive', 'Server': 'gws', 'Set-Cookie': '1P_JAR=2021-01-14-13; expires=Sat, 13-Feb-2021 13:35:59 GMT; path=/; domain=.test-abc.com; Secure, NID=207=qwM9QeM1jGIhlVzH9TZiRR3ccSvFhIg3xKbmlysAYsYwTvicGkETfz2ECsq0lGWL2OJNXsnn_bfEPr9ZlsO2aV8ru5JjvtijIsct8Wpa2LOTLcM9Od29DzYfOONwSxYMjZ3yDJq8C6nyUIWaThHN5BxzXalB19PQQ_9jzdyMTo4; expires=Fri, 16-Jul-2021 13:35:59 GMT; path=/; domain=.test-abc.com; HttpOnly', 'X-Frame-Options': 'SAMEORIGIN', 'X-Xss-Protection': '0'}

We can use the following method to get a special response header.

>>> import requests 
>>> 
>>> response = requests.get('http://www.test-abc.com') 
>>>
>>> response.headers['Content-Type']
>>>
>>> response.headers.get('Content-Type')

If you want to get the request headers (that is, the header we send to the server), you can use response object’s request.headers attribute to get it.

>>> import requests 
>>> 
>>> response = requests.get('http://www.test-abc.com')
>>>
>>> response.request.headers
{'User-Agent': 'python-requests/2.22.0', 'Accept-Encoding': 'gzip, deflate', 'Accept': '*/*', 'Connection': 'keep-alive'}
>>>
>>> response.request.headers['User-Agent']
'python-requests/2.22.0'

5.2 Add Customize Request Header.

If you want to add HTTP headers to a request, you can simply pass a dictionary object to the headers parameter of python requests module’s get or post method. In below example we will specify a http header content-type when send post request to the web server.

# First import python requests module.
>>> import requests

# Declare a url string.
>>> url = 'http://www.code-learner.com'

# Create a dictionary object which holds post data.
>>> post_data = {'version': 'python3', 'keywords': 'python'}

# Create a content-type header, the header value is application/json which told web server the post data is json format.
>>> headers = {'content-type': 'application/json'}

# Send post request to web server with the customized header and post data.
>>> response = requests.post(url, data=json.dumps(post_data), headers=headers)

6. Use Python Requests Module To Handle HTTP Cookies.

If a server response contains cookies, we can use the following method to get them.

>>> url = 'http://www.test-abc.com'
>>> 
>>> response = requests.get(url)
>>> 
>>> response.cookies
<RequestsCookieJar[Cookie(version=0, name='BDORZ', value='27315', port=None, port_specified=False, domain='.test-abc.com', domain_specified=True, domain_initial_dot=True, path='/', path_specified=True, secure=False, expires=1610756378, discard=False, comment=None, comment_url=None, rest={}, rfc2109=False)]>
>>>
>>> cookie = requests.utils.dict_from_cookiejar(result.cookies)
>>> 
>>> cookie
{'BDORZ': '27315'}
>>>
>>> for name, value in result.cookies.items():
...     print(name,' = ', value)
... 
BDORZ  =  27315

We can also send our own cookies (using cookies keyword parameters) to web server.

>>> url = 'http://www.test-abc.com/'
>>> 
>>> cookies={'cookie_1':'value_1'}
>>> 
>>> response = requests.get(url, cookies=cookies)

7. Use Python Requests Module To Handle URL Redirection.

Sometimes when we request a URL, the server will automatically redirect our request. For example, redirect HTTP request to an HTTPS request. We can use the history attribute to see the URL redirection.

# We request a http url.
>>> response = requests.get('http://www.dev2qa.com')
>>> 
# We can see that the request protocol is changed to https.
>>> response.url
'https://www.dev2qa.com/'
>>> 
>>> response.history
[<Response [301]>]

If we do not want the server to redirect HTTP request to HTTPS request, we can set allow_redirects parameter’s value to False as below.

# Set allow_redirects to False.
>>> response = requests.get('http://www.dev2qa.com', allow_redirects=False)
>>> 
>>> response.url
'http://www.dev2qa.com/'
>>> 
>>> response.history
[]

8.Use Python Requests Module To Handle Request Timeout.

We can use the timeout parameter to set the url request timeout time (time unit is seconds).

>>> response = requests.get('http://www.google.com', timeout=1)

9.Use Python Requests Module To Handle Proxy Server.

We can also use proxies in our programs for HTTP or HTTPS access (using the proxies keyword argument) as below.

>>> proxies = {
  "http": "http://10.10.1.10:3128",
  "https": "http://10.10.1.10:1080",
}
>>> requests.get("http://www.test-abc.com", proxies=proxies)

10.Use Python Requests Module To Handle HTTP Session.

Sometimes we need to log in to a website first, and then we can use the relevant URL. In this case, we can first use the login API of the website to log in, then get the session object ( use requests.Session() method ), and finally use the session object to request other URLs.

>>> session = requests.Session()

>>> login_data={'form_username':'jerry','form_password':'haha'}

>>> session.post("http://www.test-abc.com/testLogin",login_data)

>>> response = s.get('http://www.test-abc.com/notification/')

>>> response.text

11. Use Python Requests Module To Download Web Page.

We can also download web page by using the requests module.

>>> response = requests.get("http://www.code-learner.com")

>>> 
>>> with open("test.html","wb") as html:
...     html.write(result.content)
... 
89297
>>> html.close()

12. Error & Exception.

  1. All exceptions explicitly thrown by requests module inherit from requests.exceptions.RequestException class.
  2. When you encounter network problems (such as failed DNS query, rejected connection, etc.), requests module will throw a ConnectionError exception.
  3. When you encounter a rare invalid HTTP response error, requests module will throw a HTTPError exception.
  4. If the request times out, a Timeout exception will be thrown.
  5. If the request exceeds the configured maximum number of redirections, a TooManyRedirects exception will be thrown.

Leave a Comment

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.