scrapper

package

v0.0.0-...-871180b Latest Latest Go to latest Published: Jun 26, 2022 License: GPL-3.0 Imports: 11 Imported by: 0

Details

Valid go.mod file
Redistributable license
Tagged version
Stable version
Learn more about best practices

Repository

github.com/Gealber/calvopro-botv2

Links

Open Source Insights

Documentation ¶

Index ¶

Constants
Variables
func GetVideoURL(video *Video)
func PrepareQuery(rawQuery string) string
func Serialize(videos []*Video) []byte
type Video
- func Deserialize(data []byte) []*Video
- func InfoVideos(query string) []*Video

Constants ¶

View Source

const BASE_URL = "https://xnxx.com"

View Source

const (
	XNXX_CACHE = "./xnxx-cache"
)

Variables ¶

View Source

var USER_AGENTS = []string{
	"Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:47.0) Gecko/20100101 Firefox/47.0",
	"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_5) AppleWebKit/600.8.9 (KHTML, like Gecko) Version/8.0.8 Safari/600.8.9",
	"Mozilla/5.0 (iPad; CPU OS 8_4_1 like Mac OS X) AppleWebKit/600.1.4 (KHTML, like Gecko) Version/8.0 Mobile/12H321 Safari/600.1.4",
	"Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/45.0.2454.85 Safari/537.36",
	"Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/45.0.2454.85 Safari/537.36",
	"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/42.0.2311.135 Safari/537.36 Edge/12.10240",
	"Mozilla/5.0 (Windows NT 6.3; WOW64; rv:40.0) Gecko/20100101 Firefox/40.0",
	"Mozilla/5.0 (Windows NT 6.3; WOW64; Trident/7.0; rv:11.0) like Gecko",
	"Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/45.0.2454.85 Safari/537.36",
	"Mozilla/5.0 (Windows NT 6.1; Trident/7.0; rv:11.0) like Gecko",
	"Mozilla/5.0 (Windows NT 10.0; WOW64; rv:40.0) Gecko/20100101 Firefox/40.0",
}

Functions ¶

func GetVideoURL ¶

func GetVideoURL(video *Video)

GetVideoURL retrieve video url and add it to the struct

func PrepareQuery ¶

func PrepareQuery(rawQuery string) string

PrepateQuery ...

func Serialize ¶

func Serialize(videos []*Video) []byte

Serialize the videos to be passed to redis

Types ¶

type Video ¶

type Video struct {
	Title       string
	URL         string
	PageURL     string
	ImageURL    string
	Description string
	Size        int64
	Format      string
	Duration    string
}

Video ...

func Deserialize ¶

func Deserialize(data []byte) []*Video

Deserialize the bytes into a slice of videos

func InfoVideos ¶

func InfoVideos(query string) []*Video

InfoVideos basic info of the videos for that query

Source Files ¶

View all Source files

?	: This menu
/	: Search site
f or F	: Jump to
y or Y	: Canonical URL