Crawl Bilingual News with Scrapy and Splash

Learn more about the concepts, advantages and disadvantages of Client Side and Server Side Rendering through the article Client-Side and Server-Side Rendering.

We’ll see that some websites use client-side rendering, their HTML code segments will be generated on the user’s browser side, so when crawling, although F12 shows all the elements of the website completely, the downloaded code is all Javascript code and we can’t find the needed elements anywhere.

The browser works like this because it has a Javascript interpreter. When downloading the website’s code, the interpreter will interpret the Javascript code first so this code generates and downloads the necessary HTML segments and re-renders them for the user. For websites that are rendered on the user side like this, we can use 2 following methods to collect data from the website:

• 1. Use API. Dynamic rendering websites usually use APIs to get data. Press F12 and monitor the network tab to find out the API that the website uses to get data
• 1. We’ll do the same thing that the browser is doing, we’ll have a Javascript code compiler to render HTML code before analyzing. The 2 most popular tools are Selenium and Splash. With Scrapy, it’s usually used together with Splash. To compare the difference between Scrapy and Selenium, you can see at https://webscraping.fyi/lib/compare/python-selenium-vs-python-splash/

In this article, we’ll collect data from some bilingual websites that are dynamically generated by Javascript. The illustrative website is https://toomva.com. You can see the complete project at https://github.com/trannguyenhan/bilingualcrawl-vietnamese-english.

Install scrapy-splash Library

1

pip install scrapy scrapy-splash

Install and Run Splash

1

sudo docker pull scrapinghub/splash

1

sudo docker run -p 8050:8050 scrapinghub/splash

Add Splash Middleware in settings.py

1
2
3
4
5
6
7
8
9
10
11
12

SPIDER_MIDDLEWARES = {
#    'bilingualcrawl.middlewares.BilingualcrawlSpiderMiddleware': 543,
    'scrapy_splash.SplashDeduplicateArgsMiddleware': 100,
}

# Enable or disable downloader middlewares
# See https://docs.scrapy.org/en/latest/topics/downloader-middleware.html
DOWNLOADER_MIDDLEWARES = {
#    'bilingualcrawl.middlewares.BilingualcrawlDownloaderMiddleware': 543,
    'scrapy_splash.SplashCookiesMiddleware': 723,
    'scrapy_splash.SplashMiddleware': 725,
}

Add Required Splash Configuration

1
2
3
4

SPLASH_URL="http://localhost:8050"
DUPEFILTER_CLASS = 'scrapy_splash.SplashAwareDupeFilter'
HTTPCACHE_STORAGE = 'scrapy_splash.SplashAwareFSCacheStorage'
SPLASH_COOKIES_DEBUG = False

Use Scrapy Splash in Spider

In spiders, instead of using scrapy.Request, we’ll use SplashRequest so they can be used with Splash to render HTML code before analysis:

1
2
3
4

def parse_website(self, response):
  lst = response.css('.grid-search-video').css('a').xpath('@href').extract()
  for itm in lst: 
      yield SplashRequest(url=itm, callback=self.parse)

Test Splash Through Postman

1

curl --location --request GET 'http://localhost:8050/render.html?url=https://demanejar.github.io/posts/add-proxy-to-scrapy-project/'

Read more details about Splash at https://splash.readthedocs.io/.