PHP Scraping Scripts Stop Working and Exhibit High CPU Usage
So i have setup a bunch of php scrapes (all different code), but they all
use PHP, curl and proxy IPs to collect data on different sites each month.
The settings on the server and in the code are set to have no time limit
so I leave these scripts running for days collecting data.
Most of the time the scripts work, but sometimes randomly the scripts will
start to slowly increase in CPU usage going as high as 90% (memory usage,
however, remains very low) and they are then never able to finish and I
have to restart them again and then do start working.
Does anyone know why this is? Could it be the sites I am scraping are
blocking me or my proxy IPs are crashing in the middle of the scrape?
Those were the only two thoughts I had on why this happens randomly.
If you do happen to know the cause, do you know of a solution?
No comments:
Post a Comment