README
¶
gocrud
Go framework to simplify creating, reading, updating, and deleting arbitrary depth structured data — to make building REST services fast and easy.
Releases
Note that the master branch always refers to latest version of Gocrud, which would contain breaking changes. To use stable version of Gocrud APIs, please use the packages released via gopkg.in.
| Gocrud version | Install instructions | Godoc | Source |
|---|---|---|---|
| v1 (stable) | go get -v gopkg.in/manishrjain/gocrud.v1/... |
godoc | source |
| master (dev) | go get -v github.com/manishrjain/gocrud/... |
godoc | source |
Questions / Support
I primarily use IRC on freenode network. Channel is #gocrud. I'm mrjn on the network.
#gocrud on freenode
I also hang out at the gophers.slack.com, at #gocrud channel.
Although, Slack doesn't have a linux client which is what my workstation runs, and hence, I prefer IRC over Slack.
You can get an invitation to join Slack via this link:
Gopher Slack Signup.
You can also direct message me, my user id is @manishrjain.
Why?
Courtesy: Monish, co-founder karma.wiki
Having built over 3 different startup backends, I think a lot of time is wasted figuring out and coding CRUD for data structures. In addition, the choice of database has to be made up front, which causes a lot of headache for startup founders. Gocrud was written with the aim to make CRUD easy, and provide the flexibility to switch out both the underlying storage and search engines at any stage of development.
Data stores
| Datastore | Driver Available | Status |
|---|---|---|
| LevelDB | Yes | Ready |
| MySQL | Yes | Needs to implement Iterate func |
| PostgreSQL | Yes | Needs to implement Iterate func |
| Cassandra | Yes | Ready |
| MongoDB | Yes | Needs to implement Iterate func |
| Google Datastore | Yes | Needs to implement Iterate func |
| RethinkDB | Yes | Needs to implement Iterate func |
| Amazon DynamoDB | No | Needs work |
| Datastore usage shows how to use and initialize various datastores. One can add support for more by implementing this interface: |
type Store interface {
Init(args ...string)
Commit(its []*x.Instruction) error
IsNew(subject string) bool
GetEntity(subject string) ([]x.Instruction, error)
Iterate(fromId string, num int, ch chan x.Entity) (int, error)
}
Search engines
| Search Engine | Drive Available |
|---|---|
| Elastic Search | Yes |
| Solr | No |
Can be added by implementing these interfaces:
type Engine interface {
Init(args ...string)
Update(x.Doc) error
NewQuery(kind string) Query
}
type Query interface {
Limit(num int) Query
Order(field string) Query
Run() ([]x.Doc, error)
// and few others
}
Framework
This framework is built to follow these principles:
- Versioning: Keep track of all edits to the data, including deletion operations.
- Authorship: Be able to track who edited (/deleted) what.
- Retention: On deletion, only mark it as deleted. Never actually delete any data.
The framework makes it easy to have Parent-Child relationships, quite common in today’s CRUD operations. For e.g.
- Posts created by User (User -> Post)
- Comments on Posts (Post -> Comment)
- Likes on Posts (Post -> Like)
- Likes on Comments (Comment -> Like)
And be able to traverse these relationships and retrieve all of the children, grandchildren etc. For e.g. (User -> Post -> [(Comment -> Like), Like])
The framework does this by utilizing Graph operations, but without using a Graph database. This means the framework can be used to quickly build a Go backend to serve arbitrarily complex data, while still using your database of choice. See example usage
Dependency management
Users who import Gocrud into their packages are responsible to organize and maintain all of their dependencies to ensure code compatibility and build reproducibility. Gocrud makes no direct use of dependency management tools like Godep.
Performance considerations
For the example, this is what gets stored in the database:
mysql> select * from instructions;
+------------+--------------+-----------+--------------------------------------+-----------+---------------------+---------+----+
| subject_id | subject_type | predicate | object | object_id | nano_ts | source | id |
+------------+--------------+-----------+--------------------------------------+-----------+---------------------+---------+----+
| uid_oNM | User | Post | NULL | wClGp | 1435408916326573229 | uid_oNM | 1 |
| wClGp | Post | body | "You can search for cat videos here" | | 1435408916326573229 | uid_oNM | 2 |
| wClGp | Post | tags | ["search","cat","videos"] | | 1435408916326573229 | uid_oNM | 3 |
| wClGp | Post | url | "www.google.com" | | 1435408916326573229 | uid_oNM | 4 |
| wClGp | Post | Like | NULL | kStx9 | 1435408916341828408 | uid_qB3 | 5 |
| kStx9 | Like | thumb | 1 | | 1435408916341828408 | uid_qB3 | 6 |
| wClGp | Post | Comment | NULL | 8f78r | 1435408916341828408 | uid_qB3 | 7 |
| 8f78r | Comment | body | "Comment by on the post" | | 1435408916341828408 | uid_qB3 | 8 |
| wClGp | Post | Like | NULL | Gyd7G | 1435408916352622582 | uid_a30 | 9 |
| Gyd7G | Like | thumb | 1 | | 1435408916352622582 | uid_a30 | 10 |
| 8f78r | Comment | Like | NULL | q2IKK | 1435408916357443075 | uid_I5u | 11 |
| q2IKK | Like | thumb | 1 | | 1435408916357443075 | uid_I5u | 12 |
| 8f78r | Comment | Comment | NULL | g8llL | 1435408916357443075 | uid_I5u | 13 |
| g8llL | Comment | body | "Comment xv on comment" | | 1435408916357443075 | uid_I5u | 14 |
| q2IKK | Like | Comment | NULL | oaztb | 1435408916368908590 | uid_SPX | 15 |
| oaztb | Comment | body | "Comment kL on Like" | | 1435408916368908590 | uid_SPX | 16 |
| 8f78r | Comment | censored | true | | 1435408916377065650 | uid_D2g | 17 |
| kStx9 | Like | _delete_ | true | | 1435408916384422689 | uid_2a5 | 18 |
+------------+--------------+-----------+--------------------------------------+-----------+---------------------+---------+----+
18 rows in set (0.00 sec)
The writes are in constant time, where each (entity,predicate) constitutes one row. As the properties per entity grow, more rows need to be read (1 row = 1 edge/predicate) to get the entity, it's predicates and it's children. This however, shouldn't be much of a concern for any standard data, which has limited number of predicates/properties per entity. Gocrud in addition, retrieves all children in parallel via goroutines, instead of retrieving them one by one.
Property value filtering, sorting, full and partial text matching are now being made available via various search engines. Gocrud provides a search interface, which provides the most common search functionality right out of the box. Thus, there's a clear distinction between data store and search right from the beginning.
Reserved keywords
The following predicates are reserved by the framework, and shouldn't be used by the caller. Currently, this guideline isn't being hardly enforced by the framework.
| Predicate | Meaning |
|---|---|
_parent_ |
Stores an edge from child -> parent entity. |
_delete_ |
Marks a particular entity as deleted. |
Contact
Feel free to contact me at my Twitter handle @manishrjain for any discussions related to this framework. Also, feel free to send pull requests, they're welcome!
Documentation
¶
Overview ¶
Usage is a dummy package to showcase how to use gocrud with various examples.
Index ¶
Examples ¶
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
func Search ¶
func Search()
Example ¶
package main
import (
"fmt"
"io/ioutil"
"time"
_ "github.com/manishrjain/gocrud/drivers/leveldb"
_ "github.com/manishrjain/gocrud/drivers/memsearch"
"github.com/manishrjain/gocrud/indexer"
"github.com/manishrjain/gocrud/req"
"github.com/manishrjain/gocrud/search"
"github.com/manishrjain/gocrud/store"
"github.com/manishrjain/gocrud/x"
)
var log = x.Log("usage")
type SimpleIndexer struct {
}
func (si SimpleIndexer) OnUpdate(e x.Entity) (result []x.Entity) {
result = append(result, e)
return
}
func (si SimpleIndexer) Regenerate(e x.Entity) (rdoc x.Doc) {
rdoc.Id = e.Id
rdoc.Kind = e.Kind
rdoc.NanoTs = time.Now().UnixNano()
result, err := store.NewQuery(e.Id).Run()
if err != nil {
x.LogErr(log, err).Fatal("While querying store")
return
}
data := result.ToMap()
rdoc.Data = data
return
}
var particles = [...]string{
"up", "charm", "top", "gluon", "down", "strange",
"bottom", "photon", "boson", "higgs boson",
}
func main() {
path, err := ioutil.TempDir("", "gocrudldb_")
if err != nil {
x.LogErr(log, err).Fatal("Opening file")
return
}
store.Get().Init(path) // leveldb
search.Get().Init() // memsearch
// Run indexer to update entities in search engine in real time.
c := req.NewContextWithUpdates(10, 100)
indexer.Register("Child", SimpleIndexer{})
indexer.Run(c, 2)
u := store.NewUpdate("Root", "bigbang").SetSource("author")
for i := 0; i < 10; i++ {
child := u.AddChild("Child").Set("pos", i).Set("particle", particles[i])
if i == 5 {
child.MarkDeleted() // This shouldn't be retrieved anymore.
}
}
if err = u.Execute(c); err != nil {
x.LogErr(log, err).Fatal("While updating")
return
}
indexer.WaitForDone(c) // Block until indexing is done.
docs, err := search.Get().NewQuery("Child").Order("-data.pos").Run()
if err != nil {
x.LogErr(log, err).Fatal("While searching")
return
}
fmt.Println("docs:", len(docs))
for _, doc := range docs {
m := doc.Data.(map[string]interface{})
fmt.Println(m["pos"], m["particle"])
}
}
Output: docs: 9 9 higgs boson 8 boson 7 photon 6 bottom 4 down 3 gluon 2 top 1 charm 0 up
func Store ¶
func Store()
Example ¶
package main
import (
"fmt"
"io/ioutil"
_ "github.com/manishrjain/gocrud/drivers/leveldb"
_ "github.com/manishrjain/gocrud/drivers/memsearch"
"github.com/manishrjain/gocrud/req"
"github.com/manishrjain/gocrud/store"
"github.com/manishrjain/gocrud/x"
)
var log = x.Log("usage")
func main() {
path, err := ioutil.TempDir("", "gocrudldb_")
if err != nil {
x.LogErr(log, err).Fatal("Opening file")
return
}
store.Get().Init(path) // leveldb
// Update some data.
c := req.NewContext(10) // 62^10 permutations
err = store.NewUpdate("Root", "bigbang").SetSource("author").
Set("when", "13.8 billion years ago").Set("explosive", true).Execute(c)
if err != nil {
x.LogErr(log, err).Fatal("Commiting update")
return
}
// Retrieve that data
result, err := store.NewQuery("bigbang").Run()
if err != nil {
x.LogErr(log, err).Fatal("While querying store")
return
}
fmt.Println(result.Kind) // Root
fmt.Println(result.Id) // bigbang
data := result.ToMap()
{
val, ok := data["explosive"]
if !ok {
log.Fatal("creator should be set")
return
}
fmt.Println(val) // true
}
{
val, ok := data["when"]
if !ok {
log.Fatal("creator should be set")
return
}
fmt.Println(val)
}
}
Output: Root bigbang true 13.8 billion years ago
Types ¶
This section is empty.
Source Files
Directories
¶
| Path | Synopsis |
|---|---|
|
drivers
|
|
|
cassandra
Package cassandra contains Cassandra driver for Gocrud.
|
Package cassandra contains Cassandra driver for Gocrud. |
|
elastic
command
|
|
|
examples
|
|
|
social
command
Package main to demonstrate usage of the gocrud apis, using a social network as an example.
|
Package main to demonstrate usage of the gocrud apis, using a social network as an example. |
|
Package indexer provides two different methods to run real time incremental indexing for entities.
|
Package indexer provides two different methods to run real time incremental indexing for entities. |
|
This package is the initialization point for the api.
|
This package is the initialization point for the api. |
|
Package search provides a way to index entities and run relatively complicated search queries, best served outside of data stores, and by specialized search engines like ElasticSearch or Solr etc.
|
Package search provides a way to index entities and run relatively complicated search queries, best served outside of data stores, and by specialized search engines like ElasticSearch or Solr etc. |
|
Package store provides an interface for data store operations, to allow for easy extensibility to support various datastores.
|
Package store provides an interface for data store operations, to allow for easy extensibility to support various datastores. |
|
Package x stores utility functions, mostly for internal usage.
|
Package x stores utility functions, mostly for internal usage. |