《python爬虫练习一》scrapy框架抓取游戏目录

scrapy框架获取游戏目录

1.安装环境

#安装库文件
pip install scrapy

2.Scrapy指令新建一个工程Os_GAME

scrapy startproject Os_GAME

1.1目录结构中常需要操作的

items：用于定义您要提取的数据项（即所谓的item）的Python模块。
pipelines.py：用于定义数据的处理方式，如存储到数据库、输出到文件等。
settings.py：用于存储Scrapy项目的各种配置信息。

2.新建一个主体.py文件在spiders路径下

cd Os_GAME
#这个地方的链接不代表最终页面的链接，后续需要覆写
scrapy genspider pull_game https://www.oschina.net

2.1我主体的代码

import scrapy
from ..items import OsGameItem

class PullGameSpider(scrapy.Spider):
    name = "pull_game"
    allowed_domains = ["www.oschina.net/group/fishfish"]
    start_urls = ["https://www.oschina.net/group/fishfish?circle=&tab=project&p=1&_pjax=.entry-container__inner"]
    def start_requests(self):
        for i in range(1,11):
            base_url = f"https://www.oschina.net/group/fishfish?circle=&tab=project&p={i}&_pjax=.entry-container__inner"
            yield  scrapy.Request(url=base_url,callback=self.parse)
    def parse(self, response):
        items = OsGameItem()
        data01 = response.xpath("//*[@id=\"mainScreen\"]/div/div/div[2]/div/div[1]/div/div[2]/div//*[@class=\"project-item__title\"]/a/text()")
        data02 = response.xpath("//*[@id=\"mainScreen\"]/div/div/div[2]/div/div[1]/div/div[2]/div//*[@class=\"project-item__title\"]/a/@href")
        # print(data01)
        for i,j in zip(data01,data02):
            items["title"] = i.extract()
            items["url"] = j.extract()
            yield items
        pass

2.2中间变量我设置了一个标题一个链接

# Define here the models for your scraped items
#
# See documentation in:
# https://docs.scrapy.org/en/latest/topics/items.html

import scrapy


class OsGameItem(scrapy.Item):
    # define the fields for your item here like:
    # name = scrapy.Field()
    title = scrapy.Field()
    url = scrapy.Field()
    pass

2.3设计存储方案，存储到csv文件中

# Define your item pipelines here
#
# Don't forget to add your pipeline to the ITEM_PIPELINES setting
# See: https://docs.scrapy.org/en/latest/topics/item-pipeline.html


# useful for handling different item types with a single interface
from itemadapter import ItemAdapter
import csv

class OsGamePipeline:
    def open_spider(self, spider):
        # 打开CSV文件，并将标题写入文件
        self.file = open('Os_game.csv', 'w', newline='')
        self.writer = csv.writer(self.file)
        self.writer.writerow(['Title', 'Url'])

    def close_spider(self, spider):
        self.file.close()

    def process_item(self, item, spider):
        # 将数据写入文件
        self.writer.writerow([item['title'], item['url']])
        return item

2.4默认设置

```python
#文件名：setting.py
#增加一个USER_AGENT,
USER_AGENT = 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.110 Safari/537.3'
# Obey robots.txt rules
#机器人选项为False
ROBOTSTXT_OBEY = False
#设置管道优先级
ITEM_PIPELINES = {
    "baidu.pipelines.BaiduPipeline": 300,
}

结语：我的作业顺序一般是先用基本库进行代码的实现，然后再用框架进行优化，主要是框架里面还是有很多东西不是非常的熟悉。