Install Selenium Middleware for Scrapy

Scrapy provides the scrapy-selenium library that allows using Selenium to fetch website data before returning it to the spider for processing. However, in some cases, for example, I don’t want to use regular selenium but want to use undetected-chromedriver because it helps bypass Cloudflare on websites, want to add proxy through chrome extensions, want to scroll up and down and perform some actions before entering the spider, etc., using the library will be very limited. Using the library is also very convenient in some simple cases and I will introduce it in a later article. In this article, I will talk about using undetected-chromedriver to build SeleniumMiddleware to bypass Cloudflare websites.

Initialize Project

Initialize a scrapy project:

1

scrapy startproject seleniummiddlewarescrapydemo

Create a spider to crawl data, here I crawl data from the vinpearl.com/ site:

1

scrapy genspider vinpearl_dot_com https://vinpearl.com/vi/news/ha-noi

Install necessary libraries:

1
2

pip install scrapy
pip install scrapy-selenium

SeleniumMiddleware

Create the SeleniumBaseMiddleware class as follows:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39

class SeleniumBaseMiddleWare(object):
    @classmethod
    def from_crawler(cls, crawler):
        middleware = cls()
        crawler.signals.connect(middleware.spider_opened, signals.spider_opened)
        crawler.signals.connect(middleware.spider_closed, signals.spider_closed)
        return middleware

    def get_chrome_options(self):
        options = uc.ChromeOptions()
        return options

    def spider_opened(self, spider):
        try:
            options = self.get_chrome_options()

            self.driver = uc.Chrome(
                options=options,
                version_main=127
            )
        except Exception as e:
            logging.error(f"Failed to open spider and create driver: {e}")
            raise

    def process_request(self, request, spider):
        logging.info('SELENIUM FETCH URL {}'.format(request.url))
        self.driver.get(request.url)

        request.meta.update({'driver': self.driver})
        content = self.driver.page_source
        return HtmlResponse(
            request.url, encoding="utf-8", body=content, request=request
        )

    def process_response(self, request, response, spider):
        return response
    
    def spider_closed(self, spider):
        self.driver.quit()

• spider_opened() initializes driver and necessary parameters, such as options or if there are additional proxies we will also add proxies here.

• spider_closed() closes driver after finishing crawl.

• process_request() handles data fetching, assigns request.meta.update({'driver': self.driver}) to use driver inside spider if needed.

In the vinpearl_dot_com spider, if there are tasks that need to use driver like scrolling up and down, clicking buttons, then extract driver as follows:

1
2
3
4

driver = response.request.meta["driver"]
driver.execute_script(
    "window.scrollTo(0, document.body.scrollHeight);"
)

With the condition that driver has been updated into the request of the previous middleware request.meta.update({'driver': self.driver}) as in SeleniumBaseMiddleware.

In the settings file, add SeleniumBaseMiddleware to DOWNLOADER_MIDDLEWARES:

1
2
3

DOWNLOADER_MIDDLEWARES = {
   "seleniummiddlewarescrapydemo.middlewares.SeleniumBaseMiddleWare": 543,
}

Now crawling websites is very easy when no longer blocked by Cloudflare and has generated all Javascript code of dynamic websites into HTML for easy analysis.

Complete project see at: https://github.com/demanejar/selenium-middleware-scrapy-demo