mixpanels3

package module

v0.0.0-...-2cc843a Latest Latest Go to latest Published: Mar 29, 2015 License: GPL-2.0 Imports: 19 Imported by: 0

Details

Valid go.mod file
Redistributable license
Tagged version
Stable version
Learn more about best practices

Repository

github.com/pior/mixpanel-s3

Links

Open Source Insights

README ¶

mixpanel-s3

Extract Mixpanel raw events and upload to S3

Features

Static binary (Go)
Download raw events from the Mixpanel data api
Upload on S3
On the fly Gzip compression
Event selection
Split by events
Retry download (5 times, wait 60s before retrying)

Install

go install github.com/pior/mixpanel-s3/mixpanel-s3

Usage

usage: mixpanel-s3 --key=XXXXXX --secret=XXXXXX --bucket=BUCKET [<flags>]

Extract Mixpanel raw events and upload to S3

Flags:
  --help               Show help.
  -f, --from="2015-03-25"  
                       Extract from this date
  -t, --to="2015-03-25"  
                       Extract to this date
  -e, --event=EVENT    Extract only this event
  -k, --key=XXXXXX     Mixpanel api key
  -s, --secret=XXXXXX  Mixpanel secret key
  -b, --bucket=BUCKET  S3 bucket name
  -p, --prefix=PREFIX  S3 key prefix
  --split              Split by event name
  --version            Show application version.

Note: use the TMPDIR environment variable to control where mixpanel-s3 will stage the raw events

Configuration

Mixpanel and AWS credentials can be provided as command line arguments or environment variables

Environment Variables

AWS_ACCESS_KEY_ID
AWS_SECRET_ACCESS_KEY
MIXPANEL_API_KEY
MIXPANEL_SECRET_KEY
S3_BUCKET
S3_PREFIX
TMPDIR

Documentation ¶

Index ¶

func RunConfig(c *Config) (err error)
type Config
- func (c *Config) GetEffectiveS3Prefix() string
- func (c *Config) GetTmpFilename() string
type Event
type EventBuffer
- func SplitEvents(input io.Reader) (events []*EventBuffer, err error)
type EventRecord
type MixpanelAPI
- func (m *MixpanelAPI) RawEvents(file *os.File, from string, to string, events []string) (err error)
type S3Loader
- func NewS3Loader(bucket string, prefix string) *S3Loader

Constants ¶

This section is empty.

Variables ¶

This section is empty.

Functions ¶

func RunConfig ¶

func RunConfig(c *Config) (err error)

Types ¶

type Config ¶

type Config struct {
	From   string
	To     string
	Events []string
	Key    string
	Secret string
	Bucket string
	Prefix string
	Split  bool
}

func (*Config) GetEffectiveS3Prefix ¶

func (c *Config) GetEffectiveS3Prefix() string

func (*Config) GetTmpFilename ¶

func (c *Config) GetTmpFilename() string

type Event ¶

type Event struct {
	// contains filtered or unexported fields
}

type EventBuffer ¶

type EventBuffer struct {
	Name string
	Key  string
	File *os.File
	// contains filtered or unexported fields
}

func SplitEvents ¶

func SplitEvents(input io.Reader) (events []*EventBuffer, err error)

type EventRecord ¶

type EventRecord struct {
	Event string `json:"event"`
}

type MixpanelAPI ¶

type MixpanelAPI struct {
	ApiKey    string
	ApiSecret string
}

func (*MixpanelAPI) RawEvents ¶

func (m *MixpanelAPI) RawEvents(file *os.File, from string, to string, events []string) (err error)

type S3Loader ¶

type S3Loader struct {
	Bucket string
	Prefix string
	// contains filtered or unexported fields
}

func NewS3Loader ¶

func NewS3Loader(bucket string, prefix string) *S3Loader

func (*S3Loader) Emit ¶

func (s *S3Loader) Emit(f *os.File, key string) (err error)

func (*S3Loader) EmitGzip ¶

func (s *S3Loader) EmitGzip(r io.Reader, key string) (err error)

func (*S3Loader) GetFullKeyForKey ¶

func (s *S3Loader) GetFullKeyForKey(key string) string

func (*S3Loader) GetUrlForKey ¶

func (s *S3Loader) GetUrlForKey(key string) string

func (*S3Loader) Init ¶

func (s *S3Loader) Init() (err error)

Source Files ¶

View all Source files

Directories ¶

Path	Synopsis
mixpanel-s3

?	: This menu
/	: Search site
f or F	: Jump to
y or Y	: Canonical URL