controller

controller是libnetwork的核心，本次将对controller展开分析。分析将从controller的功能展开：

  1. 管理driver；
  2. 管理network；
  3. 管理sandbox；
  4. 管理store；
  5. 监听unix socket(sandbox_externalkey使用)。

先来看controller的定义，在/libnetwork/controller.go中：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
type controller struct {
	id                     string
	drvRegistry            *drvregistry.DrvRegistry
	sandboxes              sandboxTable
	cfg                    *config.Config
	stores                 []datastore.DataStore
	discovery              hostdiscovery.HostDiscovery
	extKeyListener         net.Listener
	watchCh                chan *endpoint
	unWatchCh              chan *endpoint
	svcRecords             map[string]svcInfo
	nmap                   map[string]*netWatch
	serviceBindings        map[serviceKey]*service
	defOsSbox              osl.Sandbox
	ingressSandbox         *sandbox
	sboxOnce               sync.Once
	agent                  *agent
	networkLocker          *locker.Locker
	agentInitDone          chan struct{}
	keys                   []*types.EncryptionKey
	clusterConfigAvailable bool
	sync.Mutex
}

主要的字段有：

  • id: controller的标识，随机生成；
  • drvRegistry: 负责管理driver；
  • sandboxes: 负责管理sandbox；
  • stores: 负责管理store；
  • extKeyListener: 负责处理externalkey。

注：service, ingress目前还不大了解，本次暂不分析，以后补充。

再来看controller的生成函数：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
//***生成controller***//
func New(cfgOptions ...config.Option) (NetworkController, error) {
	c := &controller{
		id:              stringid.GenerateRandomID(),
		cfg:             config.ParseConfigOptions(cfgOptions...),
		sandboxes:       sandboxTable{},
		svcRecords:      make(map[string]svcInfo),
		serviceBindings: make(map[serviceKey]*service),
		agentInitDone:   make(chan struct{}),
		networkLocker:   locker.New(),
	}
	//***初始化stores***//
	if err := c.initStores(); err != nil {
		return nil, err
	}
	//***初始化drvRegistry***//
	drvRegistry, err := drvregistry.New(c.getStore(datastore.LocalScope), c.getStore(datastore.GlobalScope), c.RegisterDriver, nil)
	if err != nil {
		return nil, err
	}
	for _, i := range getInitializers() {
		var dcfg map[string]interface{}
		// External plugins don't need config passed through daemon. They can
		// bootstrap themselves
		if i.ntype != "remote" {
			dcfg = c.makeDriverConfig(i.ntype)
		}
		if err := drvRegistry.AddDriver(i.ntype, i.fn, dcfg); err != nil {
			return nil, err
		}
	}
	if err = initIPAMDrivers(drvRegistry, nil, c.getStore(datastore.GlobalScope)); err != nil {
		return nil, err
	}
	c.drvRegistry = drvRegistry
	if c.cfg != nil && c.cfg.Cluster.Watcher != nil {
		if err := c.initDiscovery(c.cfg.Cluster.Watcher); err != nil {
			// Failing to initialize discovery is a bad situation to be in.
			// But it cannot fail creating the Controller
			log.Errorf("Failed to Initialize Discovery : %v", err)
		}
	}
	c.WalkNetworks(populateSpecial)
	// Reserve pools first before doing cleanup. Otherwise the
	// cleanups of endpoint/network and sandbox below will
	// generate many unnecessary warnings
	c.reservePools()
	// Cleanup resources
	c.sandboxCleanup(c.cfg.ActiveSandboxes)
	c.cleanupLocalEndpoints()
	c.networkCleanup()
	//***启动ExternalKeyListener***//
	if err := c.startExternalKeyListener(); err != nil {
		return nil, err
	}
	return c, nil
}

New()的流程如下：

  1. 生成controller；
  2. 调用initStores()初始化stores；
  3. 初始化drvRegistry；
  4. 清理store中sandbox及newwork资源；
  5. 启动ExternalKeyListener监听unix socket请求。

(一) 管理driver

controller管理driver是通过DrvRegistry进行的。所以我们就先分析DrvRegistry，然后再看初始化过程所做的工作。还有一点需要注意，libnetwork中的driver就是指network driver。

(1) DrvRegistry

DrvRegistry定义在/libnetwork/drvregistry/drvregistry.go中：

1
2
3
4
5
6
7
8
9
10
11
type driverTable map[string]*driverData
type ipamTable map[string]*ipamData
// DrvRegistry holds the registry of all network drivers and IPAM drivers that it knows about.
type DrvRegistry struct {
	sync.Mutex
	drivers     driverTable
	ipamDrivers ipamTable
	dfn         DriverNotifyFunc
	ifn         IPAMNotifyFunc
}

其中，drivers存储了driver，ipamDrivers存储IPAM driver。可以使用DrvRegistry包下的New()生成DrvRegistry：

1
2
3
4
5
6
7
8
9
10
11
//***生成DrvRegistry***//
func New(lDs, gDs interface{}, dfn DriverNotifyFunc, ifn IPAMNotifyFunc) (*DrvRegistry, error) {
	r := &DrvRegistry{
		drivers:     make(driverTable),
		ipamDrivers: make(ipamTable),
		dfn:         dfn,
		ifn:         ifn,
	}
	return r, nil
}

DrvRegistry::AddDriver()

AddDriver()可以向DrvRegistry中添加driver。

1
2
3
4
//***向DrvRegistry添加driver***//
func (r *DrvRegistry) AddDriver(ntype string, fn InitFunc, config map[string]interface{}) error {
	return fn(r, config)
}

添加过程是靠传入的参数fn完成的。

DrvRegistry::WalkIPAMs()

WalkIPAMs()可以把函数作用在DrvRegistry中的所有IPAM driver上。

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
//***类似目录的Walk，调用ifn函数作用在每一个IPAM driver上，直到ifn()为true***//
func (r *DrvRegistry) WalkIPAMs(ifn IPAMWalkFunc) {
	type ipamVal struct {
		name string
		data *ipamData
	}
	r.Lock()
	ivl := make([]ipamVal, 0, len(r.ipamDrivers))
	for k, v := range r.ipamDrivers {
		ivl = append(ivl, ipamVal{name: k, data: v})
	}
	r.Unlock()
	for _, iv := range ivl {
		if ifn(iv.name, iv.data.driver, iv.data.capability) {
			break
		}
	}
}

DrvRegistry::WalkDrivers()

WalkDrivers()可以把函数作用在DrvRegistry中的所有driver上。

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
//***类似目录的Walk，调用dfn函数作用在每一个network driver上，直到dfn()为true***//
func (r *DrvRegistry) WalkDrivers(dfn DriverWalkFunc) {
	type driverVal struct {
		name string
		data *driverData
	}
	r.Lock()
	dvl := make([]driverVal, 0, len(r.drivers))
	for k, v := range r.drivers {
		dvl = append(dvl, driverVal{name: k, data: v})
	}
	r.Unlock()
	for _, dv := range dvl {
		if dfn(dv.name, dv.data.driver, dv.data.capability) {
			break
		}
	}
}

DrvRegistry::Driver()

Driver()可以依据name获取network driver。

1
2
3
4
5
6
7
8
9
10
11
12
//***获取DrvRegistry中名为name的network driver***//
func (r *DrvRegistry) Driver(name string) (driverapi.Driver, *driverapi.Capability) {
	r.Lock()
	defer r.Unlock()
	d, ok := r.drivers[name]
	if !ok {
		return nil, nil
	}
	return d.driver, &d.capability
}

DrvRegistry::IPAM()

IPAM()可以通过name获取IPAM driver。

1
2
3
4
5
6
7
8
9
10
11
12
//***获取DrvRegistry中名为name的IPAM driver***//
func (r *DrvRegistry) IPAM(name string) (ipamapi.Ipam, *ipamapi.Capability) {
	r.Lock()
	defer r.Unlock()
	i, ok := r.ipamDrivers[name]
	if !ok {
		return nil, nil
	}
	return i.driver, i.capability
}

DrvRegistry::RegisterDriver()

RegisterDriver()可以注册driver。

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
//***注册network driver***//
func (r *DrvRegistry) RegisterDriver(ntype string, driver driverapi.Driver, capability driverapi.Capability) error {
	if strings.TrimSpace(ntype) == "" {
		return fmt.Errorf("network type string cannot be empty")
	}
	r.Lock()
	_, ok := r.drivers[ntype]
	r.Unlock()
	if ok {
		return driverapi.ErrActiveRegistration(ntype)
	}
	if r.dfn != nil {
		if err := r.dfn(ntype, driver, capability); err != nil {
			return err
		}
	}
	dData := &driverData{driver, capability}
	r.Lock()
	r.drivers[ntype] = dData
	r.Unlock()
	return nil
}

DrvRegistry::RegisterIpamDriver()

RegisterIpamDriver()可以注册IPAM driver。

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
//***注册IPAM driver***//
func (r *DrvRegistry) RegisterIpamDriver(name string, driver ipamapi.Ipam) error {
	return r.registerIpamDriver(name, driver, &ipamapi.Capability{})
}
// RegisterIpamDriverWithCapabilities registers the IPAM driver discovered with specified capabilities.
func (r *DrvRegistry) RegisterIpamDriverWithCapabilities(name string, driver ipamapi.Ipam, caps *ipamapi.Capability) error {
	return r.registerIpamDriver(name, driver, caps)
}
func (r *DrvRegistry) registerIpamDriver(name string, driver ipamapi.Ipam, caps *ipamapi.Capability) error {
	if strings.TrimSpace(name) == "" {
		return fmt.Errorf("ipam driver name string cannot be empty")
	}
	r.Lock()
	_, ok := r.ipamDrivers[name]
	r.Unlock()
	if ok {
		return types.ForbiddenErrorf("ipam driver %q already registered", name)
	}
	locAS, glbAS, err := driver.GetDefaultAddressSpaces()
	if err != nil {
		return types.InternalErrorf("ipam driver %q failed to return default address spaces: %v", name, err)
	}
	if r.ifn != nil {
		if err := r.ifn(name, driver, caps); err != nil {
			return err
		}
	}
	r.Lock()
	r.ipamDrivers[name] = &ipamData{driver: driver, defaultLocalAddressSpace: locAS, defaultGlobalAddressSpace: glbAS, capability: caps}
	r.Unlock()
	return nil
}

(2) 初始化DrvRegistry

再来看libnetwork是如何初始化DrvRegistry的。初始化代码在/libnetwork/controller.go的New()中：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
//***初始化drvRegistry***//
drvRegistry, err := drvregistry.New(c.getStore(datastore.LocalScope), c.getStore(datastore.GlobalScope), c.RegisterDriver, nil)
if err != nil {
	return nil, err
}
for _, i := range getInitializers() {
	var dcfg map[string]interface{}
	// External plugins don't need config passed through daemon. They can
	// bootstrap themselves
	if i.ntype != "remote" {
		dcfg = c.makeDriverConfig(i.ntype)
	}
	if err := drvRegistry.AddDriver(i.ntype, i.fn, dcfg); err != nil {
		return nil, err
	}
}
if err = initIPAMDrivers(drvRegistry, nil, c.getStore(datastore.GlobalScope)); err != nil {
	return nil, err
}
c.drvRegistry = drvRegistry

这段代码的逻辑如下：

  1. 生成drvRegistry；
  2. 调用getInitializers()获取需要注册的network driver的注册函数；
  3. 通过调用drvRegistry.AddDriver()添加network driver；
  4. 通过调用initIPAMDrivers()添加IPAM driver。可以使用DrvRegistry包下的New

我们先来看driver的初始化，主要涉及两个函数：getInitailizers()和drvRegistry.AddDriver()。其中drvRegistry.AddDriver()我们已经分析过。
getInitailizers()定义在/docker/libnetwork/drivers_linux.go中：

1
2
3
4
5
6
7
8
9
10
11
12
13
func getInitializers() []initializer {
	in := []initializer{
		{bridge.Init, "bridge"},
		{host.Init, "host"},
		{macvlan.Init, "macvlan"},
		{null.Init, "null"},
		{remote.Init, "remote"},
		{overlay.Init, "overlay"},
	}
	in = append(in, additionalDrivers()...)
	return in
}

可以看到getInitializers()返回了”bridge”, “host”, “macvlan”, “null”, “remote”, “overlay”这些drivers的初始化函数及名称。

以bridge driver为例，看下Init()是如何实现的，bridge的Init()定义在/libnetwork/drivers/bridge/bridge.go中：

1
2
3
4
5
6
7
8
9
10
11
12
13
// Init registers a new instance of bridge driver
func Init(dc driverapi.DriverCallback, config map[string]interface{}) error {
	d := newDriver()
	if err := d.configure(config); err != nil {
		return err
	}
	c := driverapi.Capability{
		DataScope: datastore.LocalScope,
	}
	//***注册network driver***//
	return dc.RegisterDriver(networkType, d, c)
}

这里调用的RegisterDriver()就是DrvRegistry中的RegisterDriver()。

(3) docker plugin

在controller中，还实现了loadDriver()和loadIPAMDriver()，定义在/libnetwork/controller.go中：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
func (c *controller) loadDriver(networkType string) error {
	// Plugins pkg performs lazy loading of plugins that acts as remote drivers.
	// As per the design, this Get call will result in remote driver discovery if there is a corresponding plugin available.
	_, err := plugins.Get(networkType, driverapi.NetworkPluginEndpointType)
	if err != nil {
		if err == plugins.ErrNotFound {
			return types.NotFoundErrorf(err.Error())
		}
		return err
	}
	return nil
}
func (c *controller) loadIPAMDriver(name string) error {
	if _, err := plugins.Get(name, ipamapi.PluginEndpointType); err != nil {
		if err == plugins.ErrNotFound {
			return types.NotFoundErrorf(err.Error())
		}
		return err
	}
	return nil
}

这两个方法是从调用/docker/pkg/plugins包获取对应的driver。
所以这里提下Docker的plugins discovery机制。Docker在启动时会去扫描/run/docker/plugins目录下的sock文件，并初始化成driver。这里只是猜测这是一种第三方driver注册的渠道，因为现在还没遇到过类似的driver，所以暂不展开分析。

小结

管理driver的分析结束，现在可以通过controller.drvRegistry.Driver(name)获取driver，通过controller.drvRegistry.IPAM(name)获取IPAM driver了。下一篇将介绍”管理network”。