makescraper

makescraper is a simple API for scraping website data. Just pass in the url and keywords you are looking for and have the scraper return you json.

📚 Table of Contents

1. Project Structure
2. Installation
3. Deliverables
4. Resources

Project Structure

📂 makescraper
├── LICENSE
├── README.md
├── go.mod
├── go.sum
├── server.go
├── assets
│   ├── css
│   │   └── styles.css
│   ├── fonts
│   ├── img
│   │   └── thumbnail.jpg
│   ├── js
│   │   └── scripts.min.js
│   └── svg
├── config
├── controllers
│   └── url.go
├── crawler
│   └── scrape.go
├── docs
│   └── media
│       ├── form.png
│       └── logo.jpg
├── json
│   └── serilization.go
├── models
│   └── site.go
├── parser
└── templates
    └── index.html

🚀 Installation

Homebrew

Deliverables

Scraping

• IMPORTANT: Complete the Web Scraper Workflow worksheet distributed in class.
• Create a struct to store your data.
• Refactor the c.OnHTML callback on line 16 to use the selector(s) you tested while completing the worksheet.
• Print the data you scraped to stdout.

Stretch Challenges

• Add more fields to your struct. Extract multiple data points from the website. Print them to stdout in a readable format.

Serializing & Saving

• Serialize the struct you created to JSON. Print the JSON to stdout to validate it.
• Write scraped data to a file named output.json.
• Add, commit, and push to GitHub.

Stretch Challenges

• TBA 02/10!

Resources

Lesson Plans

• BEW 2.5 - Scraping the Web: Concepts and examples covered in class related to web scraping and crawling.

Example Code

Scraping

• Colly - Docs: Check out the sidebar for 20+ examples!
• Ali Shalabi - Syntax-Helper: Command line interface to help generate proper code syntax, pulled from the Golang documentation.

Serializing & Saving

• JSON to Struct: Paste any JSON data and convert it into a Go structure that will support storing that data.
• GoByExample - JSON: Covers Go's built-in support for JSON encoding and decoding to and from built-in and custom data types (structs).
• GoByExample - Writing Files: Covers creating new files and writing to them.