通过python代码调起scrapy程序,优雅的将scrapy嵌套进其他程序中,极其简单,可以多进程调起

#! usr/bin/env python
# coding=utf-8
from scrapy import cmdline
import time

def start():
    cmdline.execute('scrapy crawl spider_name'.split())
    cmdline.execute('scrapy crawl spider_name'.split())
    cmdline.execute('scrapy crawl spider_name'.split())
    cmdline.execute('scrapy crawl spider_name'.split())
    cmdline.execute('scrapy crawl spider_name'.split())
    cmdline.execute('scrapy crawl spider_name'.split())

if '__main__' == __name__:
    start()

Logo

CSDN联合极客时间,共同打造面向开发者的精品内容学习社区,助力成长!

更多推荐