README
¶
WolfX
A batch framework for golang.
Overview
Architecture
There is Job as the top-level concept and WolfX instance bundles multiple jobs.
Each Job has a unique job name and target job can be executed by passing the job name to WolfX instance.
One or more Steps can be registered for a Job and executed in concurrently with each other.
Each Step has one reader and one writer.
The Reader and Writer runs on separate goroutines and they are connected by a channel.
Getting started
Installing
Install the library first.
go get github.com/yackrru/wolfx
Usage
Note that the following sample code is incomplete.
DBToFileJob just works as a Job and it has one step, ReadAndOutputStep, which is set to execute sequentially.
The ReadAndOutputStep is incompletely configured,
but it can be read that the Reader is set to read data from the database and the Writer is set to write data to a file.
Finally, you can execute this job by passing DBToFileJob as an argument to Execute.
package main
import (
"context"
"github.com/yackrru/wolfx"
"github.com/yackrru/wolfx/integration/database"
"github.com/yackrru/wolfx/integration/file"
"github.com/yackrru/wolfx/middleware"
)
func Execute(jobName string) int {
wx := wolfx.New()
wx.Add(new(DBToFileJob))
if err := wx.Run(jobName); err != nil {
middleware.Logger.Fatal(err)
}
return 0
}
type DBToFileJob struct {}
func (j *DBToFileJob) Name() string {
return "DBToFileJob"
}
func (j *DBToFileJob) Run() error {
return wolfx.NewJobBuilder().
Single(j.ReadAndOutputStep).
Build()
}
func (j *DBToFileJob) ReadAndOutputStep(ctx context.Context) error {
return wolfx.NewStepBuilder(ctx).
SetReader(database.NewReader(&database.ReaderConfig{})).
SetWriter(file.NewWriter(&file.WriterConfig{})).
Build()
}
Built-in integrations
The following can be used as Reader or Writer in Step.
| Type | Package.Name | Description |
|---|---|---|
| Reader | file.Reader | Reads a file using the file.CSVReader interface, which is satisfied by the standard package csv/Reader. |
| Writer | file.Writer | Writes a file using the file.CSVWriter interface, which is satisfied by the standard package csv/Writer. |
| Reader | database.Reader | Use sql/DB to load data with cursor from database. |
| Writer | database.Writer | Use sql/DB to import data to database. |
Sample codes
Documentation
¶
Index ¶
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
This section is empty.
Types ¶
type ConcurrentJob ¶
type ConcurrentJob interface {
Concurrent(s ...Step) *JobBuilder
}
ConcurrentJob is the interface that wraps the method of Concurrent.
Concurrent invokes multiple Step concurrently.
type JobBuilder ¶
type JobBuilder struct {
Flows []Flow
}
JobBuilder implements JobBuilderAPI.
func NewJobBuilder ¶
func NewJobBuilder() *JobBuilder
func (*JobBuilder) Build ¶
func (b *JobBuilder) Build() error
func (*JobBuilder) Concurrent ¶
func (b *JobBuilder) Concurrent(steps ...Step) *JobBuilder
func (*JobBuilder) Single ¶
func (b *JobBuilder) Single(s Step) *JobBuilder
type JobBuilderAPI ¶
type JobBuilderAPI interface {
// Build invokes multiple Step.
Build() error
SingleJob
ConcurrentJob
}
JobBuilderAPI is the builder interface for bootstrap and is designed to be invoked by JobExecutor.Run.
type JobExecutor ¶
type JobExecutor interface {
// Name returns the unique job name.
Name() string
// Run invokes bootstrap.
Run() error
}
JobExecutor is the top-level batch job.
A job has a unique name and a single bootstrap. It is designed so that the bootstrap is invoked by passing the unique name to the WolfX instance.
type ReaderSetter ¶
type ReaderSetter interface {
SetReader(r middleware.Reader) *StepBuilder
}
ReaderSetter sets Reader to StepBuilder.
type SingleJob ¶
type SingleJob interface {
Single(s Step) *JobBuilder
}
SingleJob is the interface that wraps the method of Single.
Single invokes a Step.
type StepBuilder ¶
type StepBuilder struct {
Reader middleware.Reader
Writer middleware.Writer
// contains filtered or unexported fields
}
StepBuilder implements StepBuilderAPI.
func NewStepBuilder ¶
func NewStepBuilder(ctx context.Context) *StepBuilder
func (*StepBuilder) Build ¶
func (b *StepBuilder) Build() error
func (*StepBuilder) SetReader ¶
func (b *StepBuilder) SetReader(r middleware.Reader) *StepBuilder
func (*StepBuilder) SetWriter ¶
func (b *StepBuilder) SetWriter(w middleware.Writer) *StepBuilder
type StepBuilderAPI ¶
type StepBuilderAPI interface {
// Build invokes user's settings.
Build() error
StepMiddlewareSetter
}
StepBuilderAPI is the builder interface for bootstrap and is designed to be invoked by each step definition.
type StepMiddlewareSetter ¶
type StepMiddlewareSetter interface {
ReaderSetter
WriterSetter
}
StepMiddlewareSetter is the interface that wraps methods of SetReader and SetWriter.
type WolfX ¶
type WolfX struct {
JobExecutors []JobExecutor
// If ArtOFF is true, ASCII art will not be displayed at launching.
ArtOFF bool
// LogLevel is gogger's LogLevel.
LogLevel gogger.LogLevel
}
WolfX is the top-level framework instance. It is the collection of batch jobs.
func (*WolfX) Add ¶
func (wx *WolfX) Add(e JobExecutor) *WolfX
Add adds JobExecutor to the WolfX instance.
type WriterSetter ¶
type WriterSetter interface {
SetWriter(r middleware.Writer) *StepBuilder
}
WriterSetter is sets Writer to StepBuilder.
Source Files
Directories