README
¶
Warning
Though functional, this software is still in an alpha state. Though written with compatibility in mind, this software has not been tested with IPv6.
Description
This CNI plugin is for container runtime operators who wish to use subnets as another administrative boundary for applications running on their cluster. When using this plugin, every cluster node becomes a router/gateway to any number of layer 2 virtual broadcast domains which are spanned across the entire cluster. This allows you to logically separate applications by network, regardless of which nodes the containers making up the application run on.
The network to which to connect (and optionally with a specific address) is passed in as part of the configuration. Optionally, the plugin can look in Kubernetes for pod annotations for the necessary information, since kubernetes does not yet support passing per pod annotations into a CNI plugin.
This plugin will utilize an external CNI IPAM plugin, but it requires that the IPAM plugin is aware of all addresses cluster-wide. If you utilize the corresponding routetable-ipam plugin, and a routing protocol, you can get efficient routing directly to a node running the destination container, without proxying through some other random node.
These distributed layer 2 networks are accomplished using a combination of the linux kernel's built in vxlan and macvlan drivers. When a container is started, the plugin will create a macvlan interface bridged with the hosts macvlan interface, both as slave devices to the vxlan interface, and then move the new macvlan interface into the container namespace. The container's default route is set to the nodes macvlan address, and traffic originating to/from the container is routed through the node.
Caveats:
- Every node in the cluster will require an address on the macvlan to route for containers that it hosts. In large clusters running IPv4, this could consume a lot of address space.
- Currently requires all cluster nodes to participate in the same layer 2 network as the underlay. In theory this could be built to work on an NBMA, but some work would need to be done to accomplish that.
- The host subnet routes create some interesting assymetric routing patterns that must be accounted for. Sometimes you can disable rp_filter. The plugin can optionally install a "bypass route" which sets up a custom rule to ensure that directly connected networks are routed out of the connected interface, instead of the more specific route being chosen.
- If running in k8s, it is highly recommended that the DNS services be isolated on their own network. When pods communicate with the DNS service address, dns responses may not be un-natted by the kube-proxy iptables rules because there is a direct connection to the requesting container. This causes failures in DNS resolution.
Features:
- Hosts will dynamically connect to a given vxlan, only when starting a container on that network (Dynamic disconnect when all containers on a vxlan are gone is still a TODO item).
- You can specify a "default" network, where containers will be placed when the network is not specified.
The networking concepts and some of this code were inspired by and are originally from here
Documentation
¶
Index ¶
- Constants
- type Config
- type HostInterface
- func (hi *HostInterface) AddContainerLink(namespace, ifname string, addr *net.IPNet) (int, error)
- func (hi *HostInterface) Delete() error
- func (hi *HostInterface) DeleteContainerLink(namespace, name string) error
- func (hi *HostInterface) GetGateway() *net.IPNet
- func (hi *HostInterface) GetOption(opt string) (string, bool)
- type Lock
- type Vxlan
Constants ¶
const ( //DefaultIPAMTimeout is how long to wait for the IPAM plugin DefaultIPAMTimeout = 10 //DefaultLockPath is the default path to store vxlan locks DefaultLockPath = "/tmp" //DefaultLockExt is the default extension of the lock file DefaultLockExt = ".lock" //DefaultVxlanRouteTable is the route table number used to store routes that override the /32 routes DefaultVxlanRouteTable = 192 //NetworkAnnotation is the string key where we search for the name of the vxlan to join NetworkAnnotation = "vxlan-cni.phdata.io/NetworkName" //AddressAnnotation is the string key where we search for the IP address requested AddressAnnotation = "vxlan-cni.phdata.io/RequestedAddress" )
Variables ¶
This section is empty.
Functions ¶
This section is empty.
Types ¶
type Config ¶
type Config struct {
*cni.Config
DefaultNetwork string `json:"defaultNetwork"`
K8sNetworkFromNamespace bool `json:"k8sNetworkFromNamespace"`
K8sReadAnnotations bool `json:"k8sReadAnnotations"`
K8sConfigPath string `json:"k8sConfigPath"`
Vxlans []*Vxlan `json:"vxlans"`
}
Config is the cni config extended with our required attributes
type HostInterface ¶
type HostInterface struct {
VxlanParams *Vxlan
// contains filtered or unexported fields
}
HostInterface represents the host's connection to the vxlan It must be made up of both a "vxlan" interface which participates in the cluster's vxlan and a "macvlan" slave of the vxlan interface which acts as the hosts connection to that vxlan the hosts address gets isntalled on the macvlan
func GetOrCreateHostInterface ¶
func GetOrCreateHostInterface(vxlan *Vxlan) (*HostInterface, error)
GetOrCreateHostInterface creates required host interfaces if they don't exist, or gets them if they already do
func (*HostInterface) AddContainerLink ¶
AddContainerLink adds a new macvlan link to the vxlan link, adds an IP, and puts it in the requested namespace.
func (*HostInterface) Delete ¶
func (hi *HostInterface) Delete() error
Delete removes the components of the host interface from the host
func (*HostInterface) DeleteContainerLink ¶
func (hi *HostInterface) DeleteContainerLink(namespace, name string) error
DeleteContainerLink deletes the containers interface
func (*HostInterface) GetGateway ¶
func (hi *HostInterface) GetGateway() *net.IPNet
GetGateway gets the gateway address and subnet from the vxlan config
type Lock ¶
type Lock struct {
Name string
// contains filtered or unexported fields
}
Lock represents a filesystem based mutex on a whole vxlan this allows us to effectively serialize any accesses to an individual network's interfaces
type Vxlan ¶
type Vxlan struct {
ID int `json:"id"`
Name string `json:"name"`
Cidr string `json:"cidr"`
ExcludeFirst int `json:"excludeFirst"`
ExcludeLast int `json:"excludeLast"`
Options map[string]string `json:"options"`
MTU int `json:"mtu"`
}
Vxlan represents the configuration for an overlay broadcast domain
Source Files
Directories