Different parse function for different start_urls in Scrapy

Question

Can Scrapy set different parse function for every start_urls?

This is the piece of pseudo-code:

    start_urls = [
    "http://111sssssssss.com",
    "http://222sssssssssssss.com",
    "http://333sssssssssss.com",
    "http://444sssssssss.com",
]


def parse_1():
    '''some code, this function will crawl http://111sssssssss.com'''


def parse_2():
    '''some code, this function will crawl http://222sssssssssssss.com'''

Is there any way to do that?

Answer 1

You can override / implement the parse_start_url function and there call parse_1 or parse_2 when the response.url meets your criteria (in this case it is the right URL).

def parse_start_url(response):
    if response.url == 'http://111sssssssss.com':
        parse_1(response)
    if response.url == 'http://222sssssssssssss.com':
        parse_2(response)

For more information about parse_start_url() read the documentation.