5 Best Python Libraries for working with HTTP

You’re asking one question: “What makes Python different?” And the short answer is: “lots of things“. The longer answers starts by stating that there’s lots that’s familiar, too. Python is a lot like any other general-purpose programming language, with statements, expressions, operators, functions, modules, methods, and classes. All the usual stuff, really. And then there’s the other stuff Python provides that makes the programmer’s life your life – that little bit easier.

Just to make sure that everything is set up correctly, and to show the classical first example, create a file called print.py in a plain text editor with the following contents:

#!/usr/bin/env python

print(“Hello”, “World!”)

The first line is a comment. In Python,comments begin with a # and continue to the end of the line.
The second line is blank outside quoted strings, Python ignores blank lines, but they are often useful to humans to break up large blocks of code to make them easier to read.
The third line is Python code. Here, the print() function is called with two arguments, each of type str (string; i.e., a sequence of characters).

Each statement encountered in a .py file is executed in turn, starting with the first one and progressing line by line. This is different from some other languages, for example, C++ and Java, which have a particular function or method with a special name where they start from.

Below you can find out the best python libraries for working with HTTP:

1. Grequests

GRequests allows you to use Requests with Gevent to make asynchronous HTTP Requests easily.

Official Repo on Github – https://github.com/kennethreitz/grequests
Latest Version (As per 4th Feb 2018) – v0.3.0
Direct Download Link – https://pypi.python.org/packages/25/9f/dff6f21676ca5d3e806e60cf0a49c52eb8f8c859310d28b1e740bc49b93e/grequests-0.3.0.tar.gz

Syntax –

import grequests

urls = [
‘http://www.heroku.com’,
‘http://python-tablib.org’,
‘http://httpbin.org’,
‘http://python-requests.org’,
‘https://yeahhub.com/’
]

To install this grequests library with pip, run the following command in your terminal:

Command: pip install grequests

2. httplib2 – Comprehensive HTTP client library.

The Hypertext Transfer Protocol (HTTP) is an application protocol for distributed, collaborative, hypermedia information systems. HTTP is the foundation of data communication for the World Wide Web.

Python httplib2 module provides methods for accessing Web resources via HTTP. It supports many features, such as HTTP and HTTPS, authentication, caching, redirects, and compression. Httplib2 is a comprehensive HTTP client library, httplib2.py supports many features left out of other HTTP libraries. HTTPS support is only available if the socket module was compiled with SSL support.

The following three types of HTTP Authentication are supported. These can be used over both HTTP and HTTPS.

– Digest
– Basic
– WSSE

Official Repo on Github – https://github.com/httplib2/httplib2
Latest Version (As per 4th Feb 2018) – v0.10.3

To install httplib2 with pip, use the following command in your terminal:

Command: pip install httplib2

A simple Code:-

import httplib2
h = httplib2.Http(“.cache”)
(resp_headers, content) = h.request(“http://example.org/”, “GET”)

The ‘content‘ is the content retrieved from the URL. The content is already decompressed or unzipped if necessary.

Here’s the sample code for viewing the source code with httplib2 library.

#!/usr/bin/python3

import httplib2

http = httplib2.Http()
content = http.request(“https://www.yeahhub.com”)[1]

print(content.decode())

Below is the example of httplib2 with HEAD Method –

#!/usr/bin/python3

import httplib2

http = httplib2.Http()

resp = http.request(“https://www.yeahhub.com”, “HEAD”)[0]

print(“Server: ” + resp[‘server’])
print(“Content type: ” + resp[‘content-type’])

3. Requests – HTTP Requests for Humans.

Requests allows you to send organic, grass-fed HTTP/1.1 requests, without the need for manual labor. There’s no need to manually add query strings to your URLs, or to form-encode your POST data. Keep-alive and HTTP connection pooling are 100% automatic.

Official Repo on Github – https://github.com/requests/requests
Latest version (As per 4th Feb 2018) – v2.18.4
Documentation – http://docs.python-requests.org/en/master/

To install this library, you can either use pip command to install or via git clone.

Command: pip install requests
Command: git clone https://github.com/requests/requests.git

Besides, all the cool kids are doing it. Requests is one of the most downloaded Python packages of all time, pulling in over 11,000,000 downloads every month. You don’t want to be left out!

Sample Code:-

#!/usr/bin/python3

import requests

#GET Request
r = requests.get(‘https://api.github.com/events’)

#POST Request
r = requests.post(‘http://httpbin.org/post’, data = {‘key’:’value’})

4. Treq – Python requests like API built on top of Twisted’s HTTP client.

Treq is an HTTP library inspired by requests but written on top of Twisted’s Agents. It provides a simple, higher level API for making HTTP requests when using Twisted.

Official Repo on Github – https://github.com/twisted/treq

To install treq, use the following command:

Command: pip install treq

Basic GET Request Example –

#!/usr/bin/env python

from twisted.internet.task import react
from _utils import print_response

import treq

def main(reactor, *args):
d = treq.get(‘http://httpbin.org/get’)
d.addCallback(print_response)
return d

react(main, [])

Basic POST Request Example –

import json

from twisted.internet.task import react
from _utils import print_response

import treq

def main(reactor, *args):
d = treq.post(‘http://httpbin.org/post’,
json.dumps({“msg”: “Hello!”}).encode(‘ascii’),
headers={b’Content-Type’: [b’application/json’]})
d.addCallback(print_response)
return d

react(main, [])

5. Urllib3 – A HTTP library with thread-safe connection pooling, file post support, sanity friendly.

Urllib3 is a powerful, sanity-friendly HTTP client for Python. Much of the Python ecosystem already uses urllib3 and you should too. urllib3 brings many critical features that are missing from the Python standard libraries:

Thread safety.
Connection pooling.
Client-side SSL/TLS verification.
File uploads with multipart encoding.
Helpers for retrying requests and dealing with HTTP redirects.
Support for gzip and deflate encoding.
Proxy support for HTTP and SOCKS.
100% test coverage.

Official Repo on Github – https://github.com/shazow/urllib3
Documentation Link – https://urllib3.readthedocs.io/en/latest/

urllib3 can be installed with pip:

Command: pip install urllib3

Alternatively, you can also install with Git.

Command: git clone git://github.com/shazow/urllib3.git
Command: python setup.py install

Example:-

# Yeahhub.com

from requests.packages import urllib3

http = urllib3.PoolManager()
r = http.request(‘GET’, ‘https://yeahhub.com’)

print “r.status: “, r.status
print “r.data”, r.data