Use HEAD request when spidering: only fetch content if actually required

Status: Alpha

Brought to you by: esmondpitt, vanrogu

#10 Use HEAD request when spidering: only fetch content if actually required

Milestone: Next Release (example)

Status: accepted

Owner: EJP

Labels: None

Priority: 5

Updated: 2022-02-21

Created: 2021-09-20

Creator: EJP

Private: No

Spidering should be done entirely with the HTTP HEAD request, not a GET request. This will save lots of time and bandwidth when not actually downloading or parsing, e.g. on images.

Discussion

EJP - 2021-10-06

assigned_to: EJP
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

EJP - 2022-02-21

status: open --> accepted
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

Use HEAD request when spidering: only fetch content if actually required

Group

Searches

Help

#10 Use HEAD request when spidering: only fetch content if actually required

Discussion