README
¶
avro-sqlite
avro-sqlite is a Go package that converts SQLite database schemas and data to Apache Avro format and vice versa. It allows for exporting SQLite tables to Avro Object Container Files (OCF) and importing Avro data back into SQLite tables.
Features
- SQLite to Avro Conversion: Export SQLite table schemas and data to Avro Object Container Files (OCF).
- Schema Management: Convert SQLite schemas to Avro schemas and export them as JSON.
- Data Import: Load Avro data back into SQLite tables.
- Flexible Configuration: Customize export options, including table selection and file naming.
- Database Introspection: List and analyze tables in a SQLite database.
Installation
To install avro-sqlite, use the go get command:
go get -u github.com/britt/avro-sqlite
Usage
Here are some examples of how to use the avro-sqlite package:
Exporting SQLite to Avro
package main
import (
"database/sql"
"log"
"github.com/britt/avro-sqlite"
_ "github.com/mattn/go-sqlite3"
)
func main() {
// Open SQLite database
db, err := sql.Open("sqlite3", "path/to/your/database.sqlite")
if err != nil {
log.Fatal(err)
}
defer db.Close()
// Export all tables to Avro OCF files
files, err := avrosqlite.SqliteToAvro(db, "output_directory", "prefix_", true, nil)
if err != nil {
log.Fatal(err)
}
log.Printf("Exported files: %v", files)
}
This example demonstrates how to export all tables from a SQLite database to Avro OCF files, including JSON schema files for each table.
Reading Schema and Data
// Read schema for a specific table
schema, err := avrosqlite.ReadSchema(db, "table_name")
if err != nil {
log.Fatal(err)
}
// Load data from a table
data, err := avrosqlite.LoadData(db, "table_name")
if err != nil {
log.Fatal(err)
}
Converting SQLite Schema to Avro Schema
sqliteSchema, err := avrosqlite.ReadSchema(db, "table_name")
if err != nil {
log.Fatal(err)
}
avroSchema, err := sqliteSchema.ToAvro()
if err != nil {
log.Fatal(err)
}
// Use the Avro schema...
Loading Avro Data into SQLite
import (
"os"
"github.com/hamba/avro"
)
// Open the Avro file
file, err := os.Open("path/to/avro/file.avro")
if err != nil {
log.Fatal(err)
}
defer file.Close()
// Read the SQLite schema (assuming you have it)
schema, err := avrosqlite.ReadSchema(db, "table_name")
if err != nil {
log.Fatal(err)
}
// Load the Avro data into SQLite
count, err := avrosqlite.LoadAvro(db, schema, file)
if err != nil {
log.Fatal(err)
}
log.Printf("Inserted %d records", count)
These examples provide a more accurate representation of how to use the avro-sqlite package based on the actual implementation in the provided source files.
Contributing
Contributions to avro-sqlite are welcome! Here's how you can contribute:
- Fork the repository
- Create your feature branch (
git checkout -b feature/AmazingFeature) - Commit your changes (
git commit -m 'Add some AmazingFeature') - Push to the branch (
git push origin feature/AmazingFeature) - Open a Pull Request
Please ensure your code adheres to the project's coding standards and includes appropriate tests.
License
This project is licensed under the MIT License - see the LICENSE file for details.
Acknowledgments
- Thanks to all contributors who have helped shape
avro-sqlite. - Special thanks to the Go community for providing excellent tools and libraries.
Support
If you encounter any issues or have questions, please file an issue on the GitHub issue tracker.
Happy coding with avro-sqlite!
Documentation
¶
Overview ¶
Package avrosqlite provides reads the schema and data from a sqlite database and converts it to [Apache Avro](https://avro.apache.org/) format. It also does the reverse. It can read an Avro schema and data and convert it to a sqlite database.
Index ¶
- Variables
- func ListTables(db *sql.DB) ([]string, error)
- func LoadAvro(db *sql.DB, schema *SqliteSchema, r io.Reader) (int64, error)
- func LoadData(db *sql.DB, table string) ([]map[string]any, error)
- func ReadAvro(schema avro.Schema, r io.Reader) ([]map[string]any, error)
- func SqliteToAvro(db *sql.DB, path, prefix string, includeJSON bool, enhancer Enhancer) ([]string, error)
- func TableToJSON(db *sql.DB, table, fileName string, enhancer Enhancer) error
- func TableToOCF(db *sql.DB, table, fileName string, enhancer Enhancer) error
- type Enhancer
- type SchemaField
- type SqliteSchema
- type SqliteType
Constants ¶
This section is empty.
Variables ¶
var SqliteBlobDefault = []byte{}
SqliteBlobDefault represents the default value for BLOB type.
Functions ¶
func ListTables ¶
ListTables returns a list of user-defined tables in the SQLite database. It excludes system tables listed in sqliteSpecialTables.
func LoadAvro ¶ added in v0.0.2
LoadAvro loads Avro data into a SQLite database.
Parameters:
- db: A pointer to the sql.DB representing the SQLite database connection.
- schema: A pointer to the SqliteSchema containing the table structure.
- r: An io.Reader providing the Avro data to be loaded.
Returns:
- int64: The number of records successfully inserted into the database.
- error: An error if any occurred during the process, nil otherwise.
If the specified table does not exist in the database, it will be created. If the table already exists, it will be truncated before inserting new data.
func LoadData ¶ added in v0.0.4
LoadData retrieves all data from the specified SQLite table. It returns a slice of maps, where each map represents a row in the table.
func ReadAvro ¶
ReadAvro reads Avro records from an io.Reader and returns them as a slice of maps.
Parameters:
- schema: The Avro schema used to decode the data.
- r: An io.Reader providing the Avro data to be read.
Returns:
- []map[string]any: A slice of maps, where each map represents an Avro record. The keys are field names, and the values are the corresponding field values.
- error: An error if any occurred during the reading process, nil otherwise.
This function decodes Avro records until it reaches the end of the input or encounters an error.
func SqliteToAvro ¶ added in v0.1.0
func SqliteToAvro(db *sql.DB, path, prefix string, includeJSON bool, enhancer Enhancer) ([]string, error)
SqliteToAvro exports data from a SQLite database to a set of OCF (Object Container File) files.
Parameters:
- db: A pointer to the sql.DB representing the SQLite database connection.
- path: The directory path where the OCF files will be saved.
- prefix: A string to be prepended to each table name in the output file names.
- includeJSON: If true, also saves a JSON version of each table's schema.
- enhancer: An Enhancer interface for modifying schemas and data (can be nil).
Returns:
- []string: A slice of strings containing the paths of all created files.
- error: An error if any occurred during the process, nil otherwise.
This function exports all tables from the SQLite database to individual OCF files. It optionally includes JSON schema files. The function is not atomic, and errors may result in incomplete sets of files.
func TableToJSON ¶ added in v0.2.0
TableToJSON writes the schema of a specified table to a JSON file.
Parameters:
- db: A pointer to the sql.DB representing the SQLite database connection.
- table: The name of the table whose schema is to be exported.
- fileName: The path and name of the JSON file to be created.
- enhancer: An Enhancer interface for modifying the schema (can be nil).
Returns:
- error: An error if any occurred during the process, nil otherwise.
This function reads the schema from the specified table, applies any enhancements, and writes the resulting schema to a JSON file.
func TableToOCF ¶ added in v0.1.0
TableToOCF writes the data from a specified table to an OCF (Object Container File) file.
Parameters:
- db: A pointer to the sql.DB representing the SQLite database connection.
- table: The name of the table to export.
- fileName: The path and name of the OCF file to be created.
- enhancer: An Enhancer interface for modifying the schema and data (can be nil).
Returns:
- error: An error if any occurred during the process, nil otherwise.
This function reads the schema and data from the specified table, applies any enhancements, and writes the result to an OCF file.
Types ¶
type Enhancer ¶ added in v0.2.0
type Enhancer interface {
// Schema modifies the schema in place before rows are read.
// It receives a pointer to the SqliteSchema and can make modifications to it.
// Returns an error if any issues occur during schema enhancement.
Schema(*SqliteSchema) error
// Row modifies each row in place before it is written.
// It receives a map representing a single row of data and can modify its contents.
// Returns an error if any issues occur during row enhancement.
Row(map[string]any) error
}
Enhancer is an interface for augmenting the schema and the data with additional information or computed values.
type SchemaField ¶
type SchemaField struct {
Name string `json:"name"`
Type SqliteType `json:"type"`
Nullable bool `json:"nullable"`
Default any `json:"default,omitempty"`
}
SchemaField represents a single field in a SQLite table schema.
func (SchemaField) AvroDefault ¶
func (s SchemaField) AvroDefault() interface{}
AvroDefault returns the default value for a field in the Avro schema.
type SqliteSchema ¶
type SqliteSchema struct {
Table string `json:"table"`
Fields []SchemaField `json:"fields"`
Sql string `json:"sql"`
}
SqliteSchema represents the schema of a SQLite table.
func ReadSchema ¶
func ReadSchema(db *sql.DB, tableName string) (*SqliteSchema, error)
ReadSchema retrieves the schema of a specified SQLite table. It returns a SqliteSchema struct containing table name, fields, and creation SQL.
type SqliteType ¶ added in v0.2.1
type SqliteType string
SqliteType represents the data type of a SQLite column.
const ( SqliteNull SqliteType = "null" SqliteInteger SqliteType = "integer" SqliteReal SqliteType = "real" SqliteText SqliteType = "text" SqliteBlob SqliteType = "blob" SqliteBoolean SqliteType = "boolean" SqliteIntegerDefault = 0 SqliteRealDefault = 0.0 SqliteTextDefault = "" )
Constants for SQLite data types and default values.
Source Files