ipdb

package module
v1.0.1 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Nov 13, 2025 License: MIT Imports: 20 Imported by: 0

README

go-ipdb

Self-contained IP database library for Go.

See also: go-domaindb

Automatically downloads and caches various types of IP databases for local in-process querying without reaching out to external APIs.

Supports the following operations:

  • Get ISO 3166 country code for IP address (geoIP)
  • Check if IP address is a range database (such as Tor exit nodes or known VPNs)
  • Optional bring-your-own download logic
  • Optional bring-your-own caching logic

Requests for IP data do not leave your process, and you can aggregate multiple data sources into a single database for better accuracy.

Even if remote lists go down, the library can still function by using cached data. If you specify multiple data sources, the failing sources will be skipped and the remaining sources will be used.

Use Cases

This library is useful if:

  • You need to block non-human requests to parts of your application
  • You don't want to rely on an external service for IP data
  • You have strict privacy requirements that prevent using an external service for IP data
  • You need to aggregate multiple IP data sources into a single database

Download

Add to your project by running:

go get github.com/termermc/go-ipdb

Examples

See examples for more usage examples.

Below are some simpler examples demonstrating the basic usage of the library.

IP to Country

The IP to country system uses the MaxMindDB (.mmdb) format. By specifying a URL to a mirror of the MaxMind GeoLite2 database, your application can resolve the ISO 3166 country code of visitor IP addresses. A default can be provided if the IP address's country is not known.

ip := netip.MustParseAddr("13.107.246.40")

cc, err := ipdb.ResolveIpIsoCountry(ip)
if err != nil {
	panic(err)
}

println(cc) // US

IP Range Database Check

The IP range database check system uses newline-separated list(s) of IP ranges in CIDR notation or bare IP addresses. Any bare IP addresses will be assumed to be /32 for IPv4 and /128 for IPv6. Bare IPs and CIDR ranges can be mixed in the same list.

All lists will be loaded into the in-memory database and IPs can be checked against them. Multiple lists can be combined for better detection.

// Assuming you have a DB named "tor-exit" containing Tor exit node IPs
const DbTorExit = "tor-exit"

ip := netip.MustParseAddr("13.107.246.40")

isTor, err := ipdb.IsIpInRangeDb(DbTorExit, ip)
if err != nil {
     panic(err)
}

if isTor {
	println("Tor exit node IP detected")
}

Obtaining Database Files

You can find many different IP lists online in .txt format for datacenter and VPN ranges. A few list URLs are included in the examples directory. Googling will yield more results. You should avoid any lists that are not updated frequently.

For Tor exit nodes, you can use Dan's exit node list. Keep in mind that this list rate limits you to 1 request per 30 minutes, so ensure you configure the Ipdb not to download faster than that. In general, you should be updating your databases every hour, or even less frequently.

MaxMind provides the free GeoLite2 database on their website, but you must agree to their terms of use and sign up before downloading it. There are public mirrors without this limitation online. You can Google "geolite2 country mmdb" to find them. A mirror URL from an NPM package is included in the examples directory.

Please keep in mind that the more lists you use, the more memory your process will consume. The in-memory representation of a database is larger than the database file itself.

Documentation

Index

Constants

View Source
const DbNameGeoIpv4 = "__ipdb_geo_ipv4"

DbNameGeoIpv4 is the database name used for the GeoIP database for IPv4 addresses. Do not name any database with this name.

View Source
const DbNameGeoIpv6 = "__ipdb_geo_ipv6"

DbNameGeoIpv6 is the database name used for the GeoIP database for IPv6 addresses. Do not name any database with this name.

View Source
const DbNameMaxSize = 64

DbNameMaxSize is the max size of the database name, in bytes.

Variables

View Source
var ErrAllUrlsFailed = errors.New("all URLs in data source failed")

ErrAllUrlsFailed is returned when all URLs in a data source failed.

View Source
var ErrDataSourceNoSource = errors.New("data source has no sources: len(Urls) == 0 or Get method is nil")

ErrDataSourceNoSource is returned when a data source has no sources. "No sources" means that the data source has no URLs and the Get method is nil.

View Source
var ErrDbNameTooLong = fmt.Errorf("database name too long, must be at most %d bytes long", DbNameMaxSize)

ErrDbNameTooLong is returned when a database name exceeds DbNameMaxSize bytes.

View Source
var ErrIpdbClosed = errors.New("ipdb closed")

ErrIpdbClosed is returned when an operation is attempted on a closed IP database.

View Source
var ErrNoCacheAndNoDownload = errors.New("no cached copy of database existed, and downloading is disabled")

ErrNoCacheAndNoDownload is returned when there is no cached database, and downloading is disabled so there is no way to get the database.

Functions

This section is empty.

Types

type AllCheckpoints

type AllCheckpoints struct {
	// All checkpoints.
	// Key is the database name, value is the checkpoint.
	Checkpoints map[string]Checkpoint `json:"checkpoints"`
}

AllCheckpoints is information for all database checkpoints.

type Checkpoint

type Checkpoint struct {
	// When the database was last updated from source.
	// Unix epoch second timestamp.
	LastUpdatedUnix int64 `json:"last_updated_unix"`
}

Checkpoint is checkpoint information for a database.

type DataSource

type DataSource struct {
	// Urls are the URLs where the IP data is located.
	// Either Get or Urls must be provided; Get takes precedence over Urls.
	// Any URLs that cannot be fetched will result in an error log and be skipped.
	Urls []*url.URL

	// Get is a function to get the IP data.
	// Either Get or Url must be provided; Get takes precedence over Url.1
	Get func() (io.ReadCloser, error)

	// RefreshInterval is the interval between updating the data from the source.
	RefreshInterval time.Duration
}

DataSource stores source information for IP data. It includes the URL the data can be fetched from and the time to wait between updating the data from the URL.

type FsStorageDriver

type FsStorageDriver struct {
	// contains filtered or unexported fields
}

FsStorageDriver implements StorageDriver by storing databases and checkpoints inside a data directory. Use NewFsStorageDriver to create an instance.

func NewFsStorageDriver

func NewFsStorageDriver(dataDir string) (*FsStorageDriver, error)

NewFsStorageDriver creates a new instance of StorageDriver. The specified directory must exist and be readable and writable by the current user. If the directory does not exist, returns a wrapped syscall.ENOENT. If the path is not a directory, returns a wrapped syscall.ENOTDIR.

func (*FsStorageDriver) ReadCheckpoints

func (s *FsStorageDriver) ReadCheckpoints() (*AllCheckpoints, error)

func (*FsStorageDriver) ReadDatabase

func (s *FsStorageDriver) ReadDatabase(name string) (io.ReadCloser, error)

func (*FsStorageDriver) WriteCheckpoints

func (s *FsStorageDriver) WriteCheckpoints(checkpoints *AllCheckpoints) error

func (*FsStorageDriver) WriteDatabase

func (s *FsStorageDriver) WriteDatabase(name string, input io.ReadCloser) error

type Ipdb

type Ipdb struct {
	// contains filtered or unexported fields
}

Ipdb stores and updates IP databases.

Includes functionality to:

  • Resolve ISO 3166 country codes for an IP address
  • Check if an IP address is a Tor exit node
  • Check if an IP address appears in a specific list of ranges.

Databases are cached on disk and updated periodically from data sources. At runtime, databases are stored in-memory.

Caches are not aware of which data sources were used to create them, so adding, removing or changing data source URLs or Get method implementations should be followed by clearing the cache.

Create an instance with NewIpdb; do not create an empty Ipdb struct and attempt to use it.

There should only be one instance of Ipdb per storage driver or storage location, and ideally only one per process. It is safe to use a single instance of Ipdb across multiple goroutines.

func NewIpdb

func NewIpdb(options Options) (*Ipdb, error)

NewIpdb creates a new Ipdb instance. Blocks until the databases are loaded, unless Options.LoadDatabasesInBackground is true. There should only be one instance of Ipdb per storage driver or storage location, and ideally only one per process. If error is nil, the returned Ipdb instance will never be nil.

func (*Ipdb) Close

func (s *Ipdb) Close() error

func (*Ipdb) DownloadAndLoadGeoIpv4Mmdb

func (s *Ipdb) DownloadAndLoadGeoIpv4Mmdb() error

DownloadAndLoadGeoIpv4Mmdb downloads the GeoIPv4 MMDB and loads it into memory. You most likely do not need to call this function, as loading databases is handled automatically by the Ipdb instance.

func (*Ipdb) DownloadAndLoadGeoIpv6Mmdb

func (s *Ipdb) DownloadAndLoadGeoIpv6Mmdb() error

DownloadAndLoadGeoIpv6Mmdb downloads the GeoIPv6 MMDB and loads it into memory. You most likely do not need to call this function, as loading databases is handled automatically by the Ipdb instance.

func (*Ipdb) DownloadAndLoadRangeDatabase added in v1.0.0

func (s *Ipdb) DownloadAndLoadRangeDatabase(name string) error

DownloadAndLoadRangeDatabase downloads the database with the specified name and loads it into memory. You most likely do not need to call this function, as loading databases is handled automatically by the Ipdb instance.

func (*Ipdb) IsIpInRangeDb added in v1.0.0

func (s *Ipdb) IsIpInRangeDb(dbName string, ip netip.Addr) (bool, error)

IsIpInRangeDb returns whether an IP was found in the specified IP range database. If the database does not exist, returns a NoSuchDatabaseError. If the database has not been initialized, returns a NotInitializedError. If the Ipdb instance has been closed, returns ErrIpdbClosed.

func (*Ipdb) ResolveIpIsoCountry

func (s *Ipdb) ResolveIpIsoCountry(ip netip.Addr, defaultCc string) (string, error)

ResolveIpIsoCountry resolves the ISO 3166 country code for the specified IP address. The country code is uppercase. If no country can be found, defaults to defaultCc. If the required database is not initialized, returns a NotInitializedError.

To avoid returning an invalid ISO 3166 country code, the defaultCc parameter should ideally be a valid ISO 3166 country code. A good placeholder value is "AQ", which is the ISO 3166 country code for Antarctica.

type NoSuchDatabaseError added in v1.0.0

type NoSuchDatabaseError struct {
	// The name of database that did not exist.
	Name string
}

NoSuchDatabaseError is returned when trying to access a IP range database that does not exist. Includes the requested database name that did not exist.

func NewNoSuchDatabaseError added in v1.0.0

func NewNoSuchDatabaseError(name string) *NoSuchDatabaseError

NewNoSuchDatabaseError creates a new NoSuchDatabaseError instance with the specified database name.

func (*NoSuchDatabaseError) Error added in v1.0.0

func (err *NoSuchDatabaseError) Error() string

type NotInitializedError

type NotInitializedError struct {
	// The type of database that was not initialized.
	Name string

	// The message.
	Msg string
}

NotInitializedError is returned when a function is run that required an IP database to be initialized, but it was not initialized. Includes the IP database type that was required but not initialized.

func NewNotInitializedError

func NewNotInitializedError(name string, msg string) *NotInitializedError

NewNotInitializedError creates a new NewNotInitializedError instance with the specified database name and message.

func (*NotInitializedError) Error

func (err *NotInitializedError) Error() string

type Options

type Options struct {
	// The storage driver used to store cached databases and checkpoint information.
	// Unless you have a custom driver you want to use, you should most likely use FsStorageDriver.
	// Required.
	StorageDriver StorageDriver

	// By default, Ipdb uses slog.Default.
	// If Logger is specified, it will use it instead.
	Logger *slog.Logger

	// Overrides the default HTTP client if not nil.
	// If nil, uses a default HTTP client with a 10-second timeout.
	HttpClient *http.Client

	// If true, disables downloading from sources and only uses cached database files.
	//
	// Important: You must still provide sources for the databases you want to use, regardless of whether download is disabled.
	DisableDownload bool

	// If true, the Ipdb instance will be created without waiting for databases to be loaded.
	// The databases will be loaded asynchronously in the background.
	// This can be useful if you're developing and don't want database loading to block startup.
	// It is NOT recommended for production.
	//
	// Important: Any methods on Ipdb that require databases to be initialized will fail until the databases have loaded.
	LoadDatabasesInBackground bool

	// A mapping of database names to their underlying IP range sources.
	// Each source's URL must point to a file containing a newline-separated list of IP addresses and/or CIDR ranges.
	// Empty lines are ignored.
	RangeSources map[string]*DataSource

	// The source for the IPv4 country database.
	// The URL must point to a MaxMindDB (.mmdb) file, containing at least the fields provided by MaxMind's GeoLite2 database.
	// See: https://dev.maxmind.com/geoip/geolite2-free-geolocation-data/
	//
	// Resolving IPv4 country codes will be disabled if this field is nil.
	GeoIpv4Source *DataSource

	// The source for the IPv6 country database.
	// The URL must point to a MaxMindDB (.mmdb) file, containing at least the fields provided by MaxMind's GeoLite2 database.
	// See: https://dev.maxmind.com/geoip/geolite2-free-geolocation-data/
	//
	// Resolving IPv6 country codes will be disabled if this field is nil.
	GeoIpv6Source *DataSource
}

Options are options for creating an Ipdb instance. Any omitted DataSource fields will be disabled and unavailable, even if cached files for them exist.

type StorageDriver

type StorageDriver interface {
	// WriteDatabase opens the database file with the specified name for writing.
	// The reader will be closed by the function regardless of whether an error occurs.
	WriteDatabase(name string, input io.ReadCloser) error

	// ReadDatabase opens the database file with the specified name for reading.
	// The caller is expected to close the reader.
	// If there is no cached database with the specified name, the function will return syscall.ENOENT.
	ReadDatabase(name string) (io.ReadCloser, error)

	// WriteCheckpoints writes all checkpoints.
	// Checkpoints must not be nil.
	WriteCheckpoints(checkpoints *AllCheckpoints) error

	// ReadCheckpoints reads and returns all checkpoints.
	// The returned checkpoints will never be nil if there is no error.
	// If checkpoints have not been saved yet, the function will return syscall.ENOENT.
	ReadCheckpoints() (*AllCheckpoints, error)
}

StorageDriver is an interface that stores IP databases and checkpoint data.

Directories

Path Synopsis
examples
simple command

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL