controller

controller是libnetwork的核心,本次将对controller展开分析。分析将从controller的功能展开:

  1. 管理driver;
  2. 管理network;
  3. 管理sandbox;
  4. 管理store;
  5. 监听unix socket(sandbox_externalkey使用)。

先来看controller的定义,在/libnetwork/controller.go中:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
type controller struct {
id string
drvRegistry *drvregistry.DrvRegistry
sandboxes sandboxTable
cfg *config.Config
stores []datastore.DataStore
discovery hostdiscovery.HostDiscovery
extKeyListener net.Listener
watchCh chan *endpoint
unWatchCh chan *endpoint
svcRecords map[string]svcInfo
nmap map[string]*netWatch
serviceBindings map[serviceKey]*service
defOsSbox osl.Sandbox
ingressSandbox *sandbox
sboxOnce sync.Once
agent *agent
networkLocker *locker.Locker
agentInitDone chan struct{}
keys []*types.EncryptionKey
clusterConfigAvailable bool
sync.Mutex
}

主要的字段有:

  • id: controller的标识,随机生成;
  • drvRegistry: 负责管理driver;
  • sandboxes: 负责管理sandbox;
  • stores: 负责管理store;
  • extKeyListener: 负责处理externalkey。

注:service, ingress目前还不大了解,本次暂不分析,以后补充。

再来看controller的生成函数:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
//***生成controller***//
func New(cfgOptions ...config.Option) (NetworkController, error) {
c := &controller{
id: stringid.GenerateRandomID(),
cfg: config.ParseConfigOptions(cfgOptions...),
sandboxes: sandboxTable{},
svcRecords: make(map[string]svcInfo),
serviceBindings: make(map[serviceKey]*service),
agentInitDone: make(chan struct{}),
networkLocker: locker.New(),
}
//***初始化stores***//
if err := c.initStores(); err != nil {
return nil, err
}
//***初始化drvRegistry***//
drvRegistry, err := drvregistry.New(c.getStore(datastore.LocalScope), c.getStore(datastore.GlobalScope), c.RegisterDriver, nil)
if err != nil {
return nil, err
}
for _, i := range getInitializers() {
var dcfg map[string]interface{}
// External plugins don't need config passed through daemon. They can
// bootstrap themselves
if i.ntype != "remote" {
dcfg = c.makeDriverConfig(i.ntype)
}
if err := drvRegistry.AddDriver(i.ntype, i.fn, dcfg); err != nil {
return nil, err
}
}
if err = initIPAMDrivers(drvRegistry, nil, c.getStore(datastore.GlobalScope)); err != nil {
return nil, err
}
c.drvRegistry = drvRegistry
if c.cfg != nil && c.cfg.Cluster.Watcher != nil {
if err := c.initDiscovery(c.cfg.Cluster.Watcher); err != nil {
// Failing to initialize discovery is a bad situation to be in.
// But it cannot fail creating the Controller
log.Errorf("Failed to Initialize Discovery : %v", err)
}
}
c.WalkNetworks(populateSpecial)
// Reserve pools first before doing cleanup. Otherwise the
// cleanups of endpoint/network and sandbox below will
// generate many unnecessary warnings
c.reservePools()
// Cleanup resources
c.sandboxCleanup(c.cfg.ActiveSandboxes)
c.cleanupLocalEndpoints()
c.networkCleanup()
//***启动ExternalKeyListener***//
if err := c.startExternalKeyListener(); err != nil {
return nil, err
}
return c, nil
}

New()的流程如下:

  1. 生成controller;
  2. 调用initStores()初始化stores;
  3. 初始化drvRegistry;
  4. 清理store中sandbox及newwork资源;
  5. 启动ExternalKeyListener监听unix socket请求。

(一) 管理driver

controller管理driver是通过DrvRegistry进行的。所以我们就先分析DrvRegistry,然后再看初始化过程所做的工作。还有一点需要注意,libnetwork中的driver就是指network driver。

(1) DrvRegistry

DrvRegistry定义在/libnetwork/drvregistry/drvregistry.go中:

1
2
3
4
5
6
7
8
9
10
11
type driverTable map[string]*driverData
type ipamTable map[string]*ipamData
// DrvRegistry holds the registry of all network drivers and IPAM drivers that it knows about.
type DrvRegistry struct {
sync.Mutex
drivers driverTable
ipamDrivers ipamTable
dfn DriverNotifyFunc
ifn IPAMNotifyFunc
}

其中,drivers存储了driver,ipamDrivers存储IPAM driver。可以使用DrvRegistry包下的New()生成DrvRegistry:

1
2
3
4
5
6
7
8
9
10
11
//***生成DrvRegistry***//
func New(lDs, gDs interface{}, dfn DriverNotifyFunc, ifn IPAMNotifyFunc) (*DrvRegistry, error) {
r := &DrvRegistry{
drivers: make(driverTable),
ipamDrivers: make(ipamTable),
dfn: dfn,
ifn: ifn,
}
return r, nil
}

DrvRegistry::AddDriver()

AddDriver()可以向DrvRegistry中添加driver。

1
2
3
4
//***向DrvRegistry添加driver***//
func (r *DrvRegistry) AddDriver(ntype string, fn InitFunc, config map[string]interface{}) error {
return fn(r, config)
}

添加过程是靠传入的参数fn完成的。

DrvRegistry::WalkIPAMs()

WalkIPAMs()可以把函数作用在DrvRegistry中的所有IPAM driver上。

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
//***类似目录的Walk,调用ifn函数作用在每一个IPAM driver上,直到ifn()为true***//
func (r *DrvRegistry) WalkIPAMs(ifn IPAMWalkFunc) {
type ipamVal struct {
name string
data *ipamData
}
r.Lock()
ivl := make([]ipamVal, 0, len(r.ipamDrivers))
for k, v := range r.ipamDrivers {
ivl = append(ivl, ipamVal{name: k, data: v})
}
r.Unlock()
for _, iv := range ivl {
if ifn(iv.name, iv.data.driver, iv.data.capability) {
break
}
}
}

DrvRegistry::WalkDrivers()

WalkDrivers()可以把函数作用在DrvRegistry中的所有driver上。

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
//***类似目录的Walk,调用dfn函数作用在每一个network driver上,直到dfn()为true***//
func (r *DrvRegistry) WalkDrivers(dfn DriverWalkFunc) {
type driverVal struct {
name string
data *driverData
}
r.Lock()
dvl := make([]driverVal, 0, len(r.drivers))
for k, v := range r.drivers {
dvl = append(dvl, driverVal{name: k, data: v})
}
r.Unlock()
for _, dv := range dvl {
if dfn(dv.name, dv.data.driver, dv.data.capability) {
break
}
}
}

DrvRegistry::Driver()

Driver()可以依据name获取network driver。

1
2
3
4
5
6
7
8
9
10
11
12
//***获取DrvRegistry中名为name的network driver***//
func (r *DrvRegistry) Driver(name string) (driverapi.Driver, *driverapi.Capability) {
r.Lock()
defer r.Unlock()
d, ok := r.drivers[name]
if !ok {
return nil, nil
}
return d.driver, &d.capability
}

DrvRegistry::IPAM()

IPAM()可以通过name获取IPAM driver。

1
2
3
4
5
6
7
8
9
10
11
12
//***获取DrvRegistry中名为name的IPAM driver***//
func (r *DrvRegistry) IPAM(name string) (ipamapi.Ipam, *ipamapi.Capability) {
r.Lock()
defer r.Unlock()
i, ok := r.ipamDrivers[name]
if !ok {
return nil, nil
}
return i.driver, i.capability
}

DrvRegistry::RegisterDriver()

RegisterDriver()可以注册driver。

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
//***注册network driver***//
func (r *DrvRegistry) RegisterDriver(ntype string, driver driverapi.Driver, capability driverapi.Capability) error {
if strings.TrimSpace(ntype) == "" {
return fmt.Errorf("network type string cannot be empty")
}
r.Lock()
_, ok := r.drivers[ntype]
r.Unlock()
if ok {
return driverapi.ErrActiveRegistration(ntype)
}
if r.dfn != nil {
if err := r.dfn(ntype, driver, capability); err != nil {
return err
}
}
dData := &driverData{driver, capability}
r.Lock()
r.drivers[ntype] = dData
r.Unlock()
return nil
}

DrvRegistry::RegisterIpamDriver()

RegisterIpamDriver()可以注册IPAM driver。

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
//***注册IPAM driver***//
func (r *DrvRegistry) RegisterIpamDriver(name string, driver ipamapi.Ipam) error {
return r.registerIpamDriver(name, driver, &ipamapi.Capability{})
}
// RegisterIpamDriverWithCapabilities registers the IPAM driver discovered with specified capabilities.
func (r *DrvRegistry) RegisterIpamDriverWithCapabilities(name string, driver ipamapi.Ipam, caps *ipamapi.Capability) error {
return r.registerIpamDriver(name, driver, caps)
}
func (r *DrvRegistry) registerIpamDriver(name string, driver ipamapi.Ipam, caps *ipamapi.Capability) error {
if strings.TrimSpace(name) == "" {
return fmt.Errorf("ipam driver name string cannot be empty")
}
r.Lock()
_, ok := r.ipamDrivers[name]
r.Unlock()
if ok {
return types.ForbiddenErrorf("ipam driver %q already registered", name)
}
locAS, glbAS, err := driver.GetDefaultAddressSpaces()
if err != nil {
return types.InternalErrorf("ipam driver %q failed to return default address spaces: %v", name, err)
}
if r.ifn != nil {
if err := r.ifn(name, driver, caps); err != nil {
return err
}
}
r.Lock()
r.ipamDrivers[name] = &ipamData{driver: driver, defaultLocalAddressSpace: locAS, defaultGlobalAddressSpace: glbAS, capability: caps}
r.Unlock()
return nil
}

(2) 初始化DrvRegistry

再来看libnetwork是如何初始化DrvRegistry的。初始化代码在/libnetwork/controller.go的New()中:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
//***初始化drvRegistry***//
drvRegistry, err := drvregistry.New(c.getStore(datastore.LocalScope), c.getStore(datastore.GlobalScope), c.RegisterDriver, nil)
if err != nil {
return nil, err
}
for _, i := range getInitializers() {
var dcfg map[string]interface{}
// External plugins don't need config passed through daemon. They can
// bootstrap themselves
if i.ntype != "remote" {
dcfg = c.makeDriverConfig(i.ntype)
}
if err := drvRegistry.AddDriver(i.ntype, i.fn, dcfg); err != nil {
return nil, err
}
}
if err = initIPAMDrivers(drvRegistry, nil, c.getStore(datastore.GlobalScope)); err != nil {
return nil, err
}
c.drvRegistry = drvRegistry

这段代码的逻辑如下:

  1. 生成drvRegistry;
  2. 调用getInitializers()获取需要注册的network driver的注册函数;
  3. 通过调用drvRegistry.AddDriver()添加network driver;
  4. 通过调用initIPAMDrivers()添加IPAM driver。可以使用DrvRegistry包下的New

我们先来看driver的初始化,主要涉及两个函数:getInitailizers()和drvRegistry.AddDriver()。其中drvRegistry.AddDriver()我们已经分析过。
getInitailizers()定义在/docker/libnetwork/drivers_linux.go中:

1
2
3
4
5
6
7
8
9
10
11
12
13
func getInitializers() []initializer {
in := []initializer{
{bridge.Init, "bridge"},
{host.Init, "host"},
{macvlan.Init, "macvlan"},
{null.Init, "null"},
{remote.Init, "remote"},
{overlay.Init, "overlay"},
}
in = append(in, additionalDrivers()...)
return in
}

可以看到getInitializers()返回了”bridge”, “host”, “macvlan”, “null”, “remote”, “overlay”这些drivers的初始化函数及名称。

以bridge driver为例,看下Init()是如何实现的,bridge的Init()定义在/libnetwork/drivers/bridge/bridge.go中:

1
2
3
4
5
6
7
8
9
10
11
12
13
// Init registers a new instance of bridge driver
func Init(dc driverapi.DriverCallback, config map[string]interface{}) error {
d := newDriver()
if err := d.configure(config); err != nil {
return err
}
c := driverapi.Capability{
DataScope: datastore.LocalScope,
}
//***注册network driver***//
return dc.RegisterDriver(networkType, d, c)
}

这里调用的RegisterDriver()就是DrvRegistry中的RegisterDriver()。

(3) docker plugin

在controller中,还实现了loadDriver()和loadIPAMDriver(),定义在/libnetwork/controller.go中:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
func (c *controller) loadDriver(networkType string) error {
// Plugins pkg performs lazy loading of plugins that acts as remote drivers.
// As per the design, this Get call will result in remote driver discovery if there is a corresponding plugin available.
_, err := plugins.Get(networkType, driverapi.NetworkPluginEndpointType)
if err != nil {
if err == plugins.ErrNotFound {
return types.NotFoundErrorf(err.Error())
}
return err
}
return nil
}
func (c *controller) loadIPAMDriver(name string) error {
if _, err := plugins.Get(name, ipamapi.PluginEndpointType); err != nil {
if err == plugins.ErrNotFound {
return types.NotFoundErrorf(err.Error())
}
return err
}
return nil
}

这两个方法是从调用/docker/pkg/plugins包获取对应的driver。
所以这里提下Docker的plugins discovery机制。Docker在启动时会去扫描/run/docker/plugins目录下的sock文件,并初始化成driver。这里只是猜测这是一种第三方driver注册的渠道,因为现在还没遇到过类似的driver,所以暂不展开分析。

小结

管理driver的分析结束,现在可以通过controller.drvRegistry.Driver(name)获取driver,通过controller.drvRegistry.IPAM(name)获取IPAM driver了。下一篇将介绍”管理network”。