What is the structure of a crawler?

What is the structure of a crawler?

In software engineering, there are such a few words " high cohesion and low coupling ", which means: large modules are divided into small modules to achieve high independence between each module. Modification of one module will affect other modules. The module or the entire project has less impact.

We take a crawler that downloads pictures as an example to make it clearer to everyone.

Error example

import re
import requests
def Visit(url,regularity,regularity_1): #parameter 1, web page address, parameter 2, regular expression rule parameter 3 secondary web page regular rule parameter 4 page number parameter 5 total page number
    r = requests.get(url)
    r.encoding = r.apparent_encoding
    for i in range(len(regular_ls)):
        url_1="https://www.236www.com/"+regular_ls[i] #Extracted secondary webpage address
        html = requests.get(url_1)
        html.encoding = html.apparent_encoding
        web_source_html = html.text
        regular_ls_1 = re.findall(regularity_1, web_source_html)
        for n in range(len(regular_ls_1)):
                address = "D:\\picture\\"+str(yema[17:])+"--"+str(i)+"--"+str(n)+".jpg" #Picture download local path
                with open(address, "wb") as f:
                    print(address,'download complete')
                print(str(i)+str(n),"Print failed")

def web_batch(The_number_of):
    regularity ='<a href="(.*?)" target="_blank' # First-level webpage regularity rules
    regularity_1 ='<img src="(.*?)"/>' # Second-level webpage regularity rules
    for i in range(The_number_of):
        url = "https://www.0606rr.com/Html/63/index-2" + str(number) + ".html" # Visit web address
        Visit(url, regularity, regularity_1)
        number =number + 1

The above code is a crawler that downloads pictures. I don't know what it feels like at first glance.

Anyway, when I read it at the time, I felt that the readability was so low. Not only was there no key comment, but the modules were all stuffed together. Reading it line by line, I really didn’t understand what this py file was doing .

The above code editor thinks that there are several shortcomings:

1. Either there is no blank line or only one line between the function modules. It is recommended in the Python coding standards that there are two blank lines between each module.

2. The code is not robust and the request has no exception handling. If a request for an image fails, the program will crash.

3. Each function does too many things, which is too different from "high cohesion and low coupling", and later maintenance is inconvenient.

After modification

Modify the above code. I revised it based on the idea of ​​"high cohesion and low coupling" and object-oriented. The amount of code has increased a bit, but the crawler is not only more robust, the function of each module is clear at a glance, and the key comments are also improved.

If crawlers want to be robust and easy to maintain, they are generally written in this structure, generally divided into 5 modules, and large crawler projects are structured in this way. For example, the Scrapy framework is also based on this structure, as follows:


The main logic module, the business logic is implemented here.

from picture. serial crawling. urlManager import urlManager
from picture. serial crawling. htmlDownload import htmlDownload
from picture. serial crawling. parseHtml import parseHtml
from picture. serial crawling. dataOutput import dataOutput
import json
import time

class spipderMan():
    Main logic
    def __init__(self):
        Initialize each module
        self.manager = urlManager()
        self.download = htmlDownload()
        self.parse = parseHtml()
        self.output = dataOutput()

    def get_url(self):
        Get the url of each page
        page_urls = self.manager.create_url()

    def request_url(self,page_urls):
        Request the url of each page
        for page_url in page_urls:
            response = self.download.request_url(page_url)
            # Determine whether the request is successful
            if response == None:
                print('Failed to access the webpage, it may be crawled back or there is a problem with the network~')
            # Determine whether it is the last page
            html = json.loads(response.text)
            if html['end'] != False:
                print('There are no more pictures, the download is complete!')

    def get_img_url(self,html):
        Parse to get the URL of all pictures on each page
        img_urls = self.parse.get_this_page_img_urls(html)

    def get_img(self,img_urls):
        Download image

if __name__ =='__main__':
    # Run the main interface
    start_time = time.time()
    spider = spipderMan()
    end_time = time.time()


The module that controls url scheduling.

class urlManager():
    Management url
    def __init__(self):
        Initialize the url to be spliced
        self.base_url ='http://image.so.com/zj?ch=beauty&sn={}&listtype=new&temp=1'

    def create_url(self):
        Construct the url of each page
        urls = [self.base_url.format(str(i)) for i in range(0,1020,30)]
        return urls


The module requested by the web page to download.

import requests
import random

class htmlDownload():
    Web download
    def __init__(self):
        Initialize the request header
        USER_AGENTS = [
            'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/64.0.3282.204 Safari/537.36',
            'Mozilla/5.0 (Windows NT 10.0; Win64; x64)AppleWebKit/537.36 (KHTML, like Gecko) Chrome/73.0.3683.103 Safari/537.36',
            'Mozilla/5.0 (Windows NT 10.0; …) Gecko/20100101 Firefox/66.0',       
        self.headers = {'User-Agent': random.choice(USER_AGENTS)}

    def request_url(self,url):
        :param url:
        response = requests.get(url,self.headers)
        if response.status_code == 200:
            return response
            return None


Analyze the module that gets the data

class parseHtml():
    Analyze web pages and extract data
    def __init__(self):
        self.img_urls = [] # Store the list of image titles and urls

    def get_this_page_img_urls(self,html):
        Get the url of the picture on this page
        :param html:
        :return: Store the list of image urls
        # Print how many pictures are currently downloaded, first determine whether the access is successful, then print if successful
        if html['list']:
            img_count = html['lastid']-30
            print('Currently downloaded {} sheets, there is an error, the same name will be filtered~'.format(img_count))

        for item in html['list']:
            img_title = item['group_title']
            img_url = item['qhimg_url']
        return self.img_urls


Data processing modules, such as storing in the database, cleaning and other operations.

import os
from picture. serial crawling. htmlDownload import htmlDownload

class dataOutput():
    Data output processing
    def __init__(self):
        Create image save path
        self.root_path = r'C:\Users\13479\Desktop\python project\my crawler\Is there any multi-process, thread image, video download comparison\pictures\serial crawling\images\\'
        # Create if there is no file path
        if not os.path.exists(self.root_path):

        self.download = htmlDownload()

    def download_img(self,img_urls):
        Function to download pictures
        :param img_urls: image name, url corresponding list
        for img_url in img_urls:
            # Construct the complete download route of the picture
            download_path ='{}{}.jpg'.format(self.root_path,img_url[0])
            if not os.path.exists(download_path):
                response = self.download.request_url(img_url[1])
                    with open(download_path,'wb') as f:

The above is a crawler that the editor downloads pictures from the 360 ​​picture website. According to this structure, it will be much easier to use and maintain later.

"High cohesion and low coupling" is a kind of thinking, and there is no fixed coding structure, but if you write the code in this way, it is not only convenient for your own later maintenance, but also very readable for others.

File acquisition

Follow the public account "Kinxia Learning Python" and reply to "360 Image Crawler" to get the source code.


The editor wrote a serial crawler this time. It took 6 minutes to download 1000 like pictures. Next time, I will modify it to multi-process and multi-thread to share with you.

Reference: https://cloud.tencent.com/developer/article/1555324 What is the structure of a crawler? -Cloud + Community-Tencent Cloud