Okay, let's enter the topic below to explain the realization of the program.
1. we must analyze the home page structure of the Movie Paradise website.
From the menu bar above, we can see the overall classification of the entire website resources. It just so happens that we can take advantage of its classification and use each classification address as the starting point of the crawler.
①Analyze the homepage address to extract the classification information
#Analysis Home def CrawIndexPage(starturl): print "Crawling the homepage" page = __getpage(starturl) if page=="error": return page = page.decode('gbk','ignore') tree = etree.HTML(page) Nodes = tree.xpath("//div[@id='menu']//a") print "Home page resolves the address",len(Nodes),"bar" for node in Nodes: CrawledURLs =  CrawledURLs.append(starturl) url=node.xpath("@href") if re.match(r'/html/[A-Za-z0-9_/]+/index.html', url): if __isexit(host + url,CrawledURLs): pass else: try: catalog = node.xpath("text()").encode("utf-8") newdir = "E:/Movie Resources/" + catalog os.makedirs(newdir.decode("utf-8")) print "Create category directory successfully------"+newdir thread = myThread(host + url, newdir,CrawledURLs) thread.start() except: pass
In this function, first download the source code of the web page, and parse the menu classification information through XPath. And create the corresponding file directory. One area that needs attention is the encoding problem, but it has been entangled by this encoding for a long time. By checking the source code of the web page, we can find that the encoding of the web page is GB2312. Here, the text information is needed to construct the Tree object through XPath. Decoding operation, change gb2312 into Unicode encoding, so that the DOM tree structure is correct, otherwise there will be problems in the later parsing.
②Analyze the homepage of each category
# Parse classification file def CrawListPage(indexurl, filedir, CrawledURLs): print "Resolving classified homepage resources" print indexurl page = __getpage(indexurl) if page=="error": return CrawledURLs.append(indexurl) page = page.decode('gbk','ignore') tree = etree.HTML(page) Nodes = tree.xpath("//div[@class='co_content8']//a") for node in Nodes: url=node.xpath("@href") if re.match(r'/', url): # The non-paged address can parse the video resource address from it if __isexit(host + url,CrawledURLs): pass else: #File naming is not allowed to appear the following special symbols filename=node.xpath("text()").encode("utf-8").replace("/"," ")\ .replace("\\"," ")\ .replace(":"," ")\ .replace("*"," ")\ .replace("?"," ")\ .replace("\""," ")\ .replace("<", "")/ .replace(">", "")\ .replace("|", "") CrawlSourcePage(host + url,filedir,filename,CrawledURLs) pass else: # The paging address is parsed again from the nest print "The paging address is parsed again from the nest", url index = indexurl.rfind("/") baseurl = indexurl[0:index + 1] pageurl = baseurl + url if __isexit(pageurl,CrawledURLs): pass else: print "The paging address is parsed again from the nest", pageurl CrawListPage(pageurl, filedir, CrawledURLs) pass pass
Open the homepage of each category and you will find that there is a same structure (click to open the example). 1. the node containing the resource URL is parsed, and then the name and URL are extracted. There are two things to pay attention to in this part. One is because you want to save the resource in a txt file, but some special symbols cannot appear when naming, so you need to deal with it. The second is to deal with the paging. The data in the website is displayed in the form of paging, so how to identify and capture the paging is also very important. Through observation, it is found that there is no "/" in front of the paging address, so it is only necessary to find out the paging address link through regular expressions, and then nested calls can solve the paging problem.
③Analyze the resource address and save it in a file
#Processing resource page crawling resource address def CrawlSourcePage(url,filedir,filename,CrawledURLs): print url page = __getpage(url) if page=="error": return CrawledURLs.append(url) page = page.decode('gbk','ignore') tree = etree.HTML(page) Nodes = tree.xpath("//div[@align='left']//table//a") try: source = filedir + "/" + filename + ".txt" f = open(source.decode("utf-8"),'w') for node in Nodes: sourceurl = node.xpath("text()") f.write(sourceurl.encode("utf-8")+"\n") f.close() except: print "!!!!!!!!!!!!!!!!!"
This paragraph is relatively simple, just write the extracted content into a file
In order to improve the efficiency of the program, multi-threaded crawling is used. Here I have opened a thread for each classified homepage, which greatly speeds up the efficiency of crawlers. At the beginning, I just used a single thread to run, but I waited all afternoon and finally ran away all afternoon because of an exception that was not handled. ! ! ! Tired
class myThread (threading.Thread): #Inherit the parent class threading.Thread def __init__(self, url, newdir,CrawledURLs): threading.Thread.__init__(self) self.url = url self.newdir = newdir self.CrawledURLs=CrawledURLs def run(self): #Write the code to be executed into the run function. The thread will run the run function directly after creation CrawListPage(self.url, self.newdir,self.CrawledURLs)
The final crawling result is as follows.