How to use middlewares when using scrapy runspider command?

Question

I know that we can configure middlewares in settings.py when we have a scrapy project.

I haven't started a scrapy project, and I use runspider command to run spider, but I want to use some middlewares. How to set it in the spider file?

Answer 1

So, the problem is, when you run a spider using scrapy runspider my_file.py, you can use the -s option to pass only simple scalar spider settings (like strings or integers). The problem is, the SPIDER_MIDDLEWARES setting expects a dictionary, and there isn't a really straight-forward way to pass that through the command-line.

Currently, the only way I know to set SPIDER_MIDDLEWARES settings for a spider without a project is using custom spider settings, which is currently available in Scrapy from the code repo (not officially released yet) since Scrapy 1.0.

If you go that route, you can put your middlewares in a file middlewares.py and do:

import middlewares  # need this, or you get import error

class MySpider(scrapy.Spider):
    name = 'my-spider'

    custom_settings = {
        'SPIDER_MIDDLEWARES': {
            'middlewares.SampleMiddleware': 500,
        }
    }

    ...

Alternatively, if you're putting the middleware class in the same file, you can use:

import scrapy

class SampleMiddleware(object):
    # your middleware code here
    ...


def fullname(o):
    return o.__module__ + "." + o.__name__


class MySpider(scrapy.Spider):
    name = 'my-spider'

    custom_settings = {
        'SPIDER_MIDDLEWARES': {
            fullname(SampleMiddleware): 500,
        }
    }

    ...