HTTP requests in Python explained

What is an HTTP request?

If you’re reading this newsletter, I’m sure you already know all the stuff I’m going to explain but let’s make a brief introduction in case someone is approaching web scraping for the first time.

HTTP requests are one of the founding stones of the HTTP protocol and basically, it’s the call a client/browser makes to a server. The outcome of the request is the server response, which contains the content placed at the URL passed in the request.

A request is composed by:

  • Method
  • Url
  • Headers
  • Body


Each request can have only one method between the ones listed in the HTTP Protocol. The most important ones are:

  • GET: to retrieve data from a url
  • POST: to modify data on a server
  • PUT: to substitute data on a server
  • DELETE: to delete data on a server

Basically, in our web scraping projects when we need to read data from a server we’ll use several GET requests, while if we need to send data to an API to query it or to fill a form, we’ll use POST requests.

Continue reading