sumjson

package module
v0.0.0-...-2a1cfb4 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: May 31, 2023 License: MIT Imports: 4 Imported by: 0

README

sumjson

Summarizes the content of multiple similarly shaped JSON objects.

usage

The objects must be one after another, not part of an overlapping array.

Correct:

{}
{}
{}
{}

Incorrect:

[
    {},
    {},
    {},
    {}
]

Although the giant array would work, the result would be an empty summary.

limitations

At this time the algorithm runs using O(n) amount of memory with regard to the input. Processing a 21MiB JSON file takes about 7s on my 14" 2023 Apple MacBook Pro, M2 Pro CPU with 32GB of RAM.

license

MIT.

Documentation

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

This section is empty.

Types

type ArraySummaryNode

type ArraySummaryNode struct {
	Freq int `json:"freq"`
}

type BoolSummaryNode

type BoolSummaryNode struct {
	Freq      int `json:"freq"`
	TrueFreq  int `json:"trues"`
	FalseFreq int `json:"falses"`
}

type BucketRange

type BucketRange struct {
	From float64 `json:"from"`
	To   float64 `json:"to"`
	Freq int     `json:"freq"`
}

type Key

type Key struct {
	Name string `json:"key"`
	Freq int    `json:"freq"`
}

type Node

type Node struct {
	Key      string  `json:"key,omitempty"`
	Children []*Node `json:"objects,omitempty"`
	Elems    []*Node `json:"arrays,omitempty"`

	ScalarNumbers []float64 `json:"numbers,omitempty"`
	ScalarStrings []string  `json:"strings,omitempty"`
	ScalarBools   []bool    `json:"bools,omitempty"`
	ScalarNulls   int       `json:"nulls,omitempty"`

	// stats
	Freq       int          `json:"freq"`
	Summarized *SummaryNode `json:"summary,omitempty"`
}

type NullSummaryNode

type NullSummaryNode struct {
	Freq int `json:"freq"`
}

type NumberSummaryNode

type NumberSummaryNode struct {
	Freq    int     `json:"freq"`
	Unique  int     `json:"unique"`
	AllInts bool    `json:"all_ints"`
	Min     float64 `json:"min"`
	Max     float64 `json:"max"`

	Distribution []BucketRange `json:"distribution"`
}

type ObjectSummaryNode

type ObjectSummaryNode struct {
	Keys []Key `json:"keys"`
}

type Reporter

type Reporter interface {
	ObjectRead(from, to, total int)
	Summarized(done, total int)
}

type StringSample

type StringSample struct {
	Value string `json:"val"`
	Freq  int    `json:"freq"`
}

type StringSummaryNode

type StringSummaryNode struct {
	Freq   int            `json:"freq"`
	Unique int            `json:"unique"`
	MinLen int            `json:"min_len"`
	MaxLen int            `json:"max_len"`
	Top    []StringSample `json:"top"`
}

type Summary

type Summary struct {
	Root *Node `json:"root"`
}

func Summarize

func Summarize(data []byte, reporter Reporter) (*Summary, error)

type SummaryNode

type SummaryNode struct {
	Objects *ObjectSummaryNode `json:"object,omitempty"`
	Arrays  *ArraySummaryNode  `json:"array,omitempty"`
	Numbers *NumberSummaryNode `json:"number,omitempty"`
	Strings *StringSummaryNode `json:"string,omitempty"`
	Bools   *BoolSummaryNode   `json:"bool,omitempty"`
	Nulls   *NullSummaryNode   `json:"null,omitempty"`
}

Directories

Path Synopsis
cmd
sumjson command

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL