README
¶
scavenge
A modern, batteries-included, modular scraping framework for Go.
Installation
go get github.com/LQR471814/scavenge
Features
- Minimal footprint: ~1k LOC with minimal reliance on third-party libraries.
- Batteries-included: Domain whitelisting, automatic throttling, retries, replaying responses, etc... all included.
- Flexible: Near-full control over the details of scraping, but no need to concern yourself with the details if you don't need to.
- Extensible: Easily add features you need with simple middleware interfaces.
Example
Here's an example that scrapes ~1000 wikipedia pages in ~20 seconds.
To run it:
cd examples/wikipedia
go run .
Dependencies
golang.org/x/net- Used indownloader.Response.github.com/PuerkitoBio/purell- Used only inmiddleware.Dedupeandmiddleware.Replay.github.com/gobwas/glob- Used only inmiddleware.AllowedDomains.github.com/zeebo/xxh3- Used only inmiddleware.FSReplayStore.- All the other dependencies are only used in examples.
Credits
- Python's scrapy is a massive influence on this library, many of the design choices in this library are taken straight from here.
Documentation
¶
Index ¶
- func ShortUrl(u *url.URL) string
- func WithOnItemProcessorFail(callback func(i items.Item, err error)) option
- func WithOnRequestFail(fn func(req *downloader.Request, err error)) option
- func WithOnSpiderHandlerFail(callback func(res *downloader.Response, err error)) option
- func WithParallelDownloads(count int) option
- func WithParallelItems(count int) option
- func WithRetryDelayBounds(minDelay, maxDelay time.Duration) option
- type Logger
- type Navigator
- func (n Navigator) AnchorRequest(a *html.Node) (*downloader.Request, error)
- func (n Navigator) AnchorUrl(a *html.Node) (*url.URL, error)
- func (n Navigator) Context() context.Context
- func (n Navigator) CurrentUrl() *url.URL
- func (n Navigator) Request(req *downloader.Request)
- func (n Navigator) SaveItem(value any)
- type Perf
- type Scavenger
- type SlogLogger
- type Spider
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
func WithOnItemProcessorFail ¶
WithOnItemProcessorFail sets the error handler for item processing errors.
func WithOnRequestFail ¶
func WithOnRequestFail(fn func(req *downloader.Request, err error)) option
WithOnRequestFail sets the error handler for requests that:
- fail before sending, either due to a malformed request or failed middleware.
- fail while sending, due to some http transport error.
- fail after receiving a response that causes an error in the downloader's middleware.
func WithOnSpiderHandlerFail ¶
func WithOnSpiderHandlerFail(callback func(res *downloader.Response, err error)) option
WithOnSpiderHandlerFail sets the error handler for errors that occur within Spider.HandleResponse
func WithParallelDownloads ¶
func WithParallelDownloads(count int) option
WithParallelDownloads sets the amount of requests and responses that can be processed in parallel.
func WithParallelItems ¶
func WithParallelItems(count int) option
WithParallelItems sets the amount of items that can be processed in parallel.
func WithRetryDelayBounds ¶
WithRetryDelayBounds sets the bounds for retry delay.
Types ¶
type Logger ¶
type Logger interface {
Debug(component, msg string, args ...any)
Info(component, msg string, args ...any)
Warn(component, msg string, args ...any)
Error(component, msg string, args ...any)
}
Logger is an interface containing logging reporters.
Note: the args of any logging methods (Debug, Info, Warn, Error) are a list of keys and values in alternating order like it is in log/slog(https://pkg.go.dev/log/slog).
func LoggerFromContext ¶
LoggerFromContext retrieves a Logger from the given context, it will panic if the Logger is not there.
type Navigator ¶
type Navigator struct {
// contains filtered or unexported fields
}
Navigator contains methods for saving items and following urls typically used in a spider's response handler.
func (Navigator) AnchorRequest ¶
AnchorRequest returns a GET request to the given url created by resolving the href of the given a-tag with the current url as the base url.
func (Navigator) AnchorUrl ¶
AnchorUrl returns the absolute url referenced by an anchor tag (created by resolving the href with the current url as the base url).
func (Navigator) CurrentUrl ¶
CurrentUrl returns the current url of the navigator.
func (Navigator) Request ¶
func (n Navigator) Request(req *downloader.Request)
Request queues the given request.
type Scavenger ¶
type Scavenger struct {
// contains filtered or unexported fields
}
Scavenger is the main component that schedules requests and processes items concurrently, it handles retry logic and pausing/resuming scraping.
func NewScavenger ¶
func NewScavenger( dl downloader.Downloader, iproc items.Processor, logger Logger, options ...option, ) *Scavenger
func ScavengerFromContext ¶
ScavengerFromContext returns Scavenger from a given context.
This will only return a scavenger if called within a spider's HandleResponse, item pipeline, or middleware.
func (*Scavenger) QueueRequest ¶
func (s *Scavenger) QueueRequest(req *downloader.Request, referer *url.URL)
type SlogLogger ¶
type SlogLogger struct {
// contains filtered or unexported fields
}
SlogLogger implements Logger using a log/slog(https://pkg.go.dev/log/slog) logger.
func NewSlogLogger ¶
func NewSlogLogger(log *slog.Logger, enablePerf bool) SlogLogger
NewSlogLogger creates a new SlogLogger using a given slog.Logger
If enablePerf is false, all calls to Elapsed will be a no-op, otherwise, the elapsed statistics will be periodically printed to the console as DEBUG-level messages.
func (SlogLogger) Debug ¶
func (t SlogLogger) Debug(component, msg string, args ...any)
func (SlogLogger) Error ¶
func (t SlogLogger) Error(component, msg string, args ...any)
func (SlogLogger) Info ¶
func (t SlogLogger) Info(component, msg string, args ...any)
func (SlogLogger) Warn ¶
func (t SlogLogger) Warn(component, msg string, args ...any)
type Spider ¶
type Spider interface {
StartingRequests() []*downloader.Request
HandleResponse(nav Navigator, res *downloader.Response) error
}
Spider contains the business logic of navigating to different links and storing structured data from the unstructured page.
Note: HandleResponse should be safe to be called concurrently, if it is too hard to make it so, make sure that any scavenger that uses this spider has the option WithDownloadWorkerCount set to 1.
Source Files
Directories