Skip to content
Advertisement

Storing data to SQL not working with my sql connector and scrapy

I am trying to store my scraped data with scrapy to a SQL database but my code does not send anything while no error is mentioned when runned.

I am using my sql connector since I don’t manage to install MySQL-python. My SQL database seems to running well and when I run the code the traffic KB/s raise. Please find below my pipelines.py code.

Advertisement

Answer

You need to add the class in ITEM_PIPELINES first to let the scrapy know i want to use this pipeline.

In your settings.py file Update the lines below with your class name as following.

The numbers 700 and 800 shows in which order the pipelines will process data, it can be any integer between 1-1000. Pipelines will process items in the order based by this number, so pipeline with 700 would process data before the pipeline with 800.

Note: Replace the projectname in 'projectname.pipelines.CleaningPipeline' with your actual projectname.

User contributions licensed under: CC BY-SA
9 People found this is helpful
Advertisement