wgetpipe

wgetpipe

Wgetpipe takes a list of fully-qualified URLs over STDIN and Gets them, outputting the code, url and elapsed fetch time.

It scans STDIN, spawning up to -max getters at a time, which stream their responses back to the collator to format the output. This tool was generated to aid in seeding pull-through caches, but has utility in other areas as well

Usage

      --bar                Use progress bar instead of printing lines, can still use -stats
      --debug              Enable debug output
      --errorsonly         Only output errors (HTTP Codes >= 400)
      --guess int          Rough guess of how many GETs will be coming for -bar to start at. It will adjust
      --max int            Maximium in-flight GET requests at a time (default 5)
      --nocolor            Don't colorize the output
      --nodnscache         Disable DNS caching
      --responsedebug      Enable full response output if debugging is on
      --save               Save the content of the files. Into hostname/folders/file.ext files
      --size string        Size of chunks to download (whole-numbers with suffixes of B,KB,MB,GB,PB)
      --sleep duration     Amount of time to sleep between spawning a GETter (e.g. 1ms, 10s)
      --stats              Output stats at the end
      --timeout duration   Amount of time to allow each GET request (e.g. 30s, 5m)

Licensing

MIT

Thank you to SpatialKey for allowing this tool to be released, as it was initially developed for them.