Viewing posts for the category Scrapy

Adapting Scrapy's images pipeline to use Azure Storage

Although it's not documented, the Scrapy images pipeline supports storing your downloaded images in an Amazon S3 bucket. This is very useful especially when scraping is over distributed nodes and you want a centralized image store.

Read more

How does Scrapy react to a blocked pipeline?

These are just some notes on how Scrapy reacts to a blocked pipeline.

Read more