本次分析将介绍如何在kubernetes v1.5.2中通过代码添加一种资源。我们假设要添加的资源为”Application”，只有一个”replicas”属性，即如下所示：

1
2
3
4
5
6
7
8
9
10
11
apiVersion: extensions/v1beta1
kind: Application
metadata:
  creationTimestamp: 2017-12-06T11:37:22Z
  generation: 1
  name: ubuntu
  namespace: default
  resourceVersion: "448526"
  selfLink: /apis/extensions/v1beta1/namespaces/default/applications/ubuntu
  uid: d2fb278c-da79-11e7-adb3-0800274a4ec3
replicas: 20

因为这个demo是参考Deployment写的，所以，我们把Application也放在了extensions/v1beta1中。自己另起一个group添加资源待以后弄清楚Kubernetes的自动生成代码机制再实验，这部分可以参考帮助文件adding-an-APIGroup.md。

添加Application的定义

首先，我们要在/pkg/apis/extensions/types.go中添加Application及ApplicationList的定义：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
// +genclient=true
type Application struct {
        unversioned.TypeMeta `json:",inline"`
        // +optional
        api.ObjectMeta `json:"metadata,omitempty"`
        // +optional
        Replicas int32 `json:"replicas,omitempty"`
}
type ApplicationList struct {
        unversioned.TypeMeta `json:",inline"`
        // +optional
        unversioned.ListMeta `json:"metadata,omitempty"`
        // Items is the list of applications.
        Items []Application `json:"items"`
}

其中，// +genclient=true估计是和代码自动生成机制有关，以后研究。

同样，在/pkg/extensions/v1beta1/types.go中添加Application及ApplicationList的定义：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
// +genclient=true
type Application struct {
        unversioned.TypeMeta `json:",inline"`
        // +optional
        v1.ObjectMeta `json:"metadata,omitempty"`
        // +optional
        Replicas int32 `json:"replicas,omitempty"`
}
type ApplicationList struct {
        unversioned.TypeMeta `json:",inline"`
        // +optional
        unversioned.ListMeta `json:"metadata,omitempty"`
        // Items is the list of applications.
        Items []Application `json:"items"`
}

添加完成之后，执行hack/update-all.sh -v，执行该脚本需要下载gcr.io/google_containers/kube-cross:v1.7.4-0。然后就等，需要很长时间完成，我这边执行到特定阶段时会报错，但此时zz_generated.deepcopy.go等文件已有Application的相关转换代码，ClientSet中也已经有了Application的定义，所以主目标已经达到。至于为什么出错，出错带来的影响，还有待研究完代码自动生成机制后才能给出结论。
报错信息如下：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
......
make[1]: Leaving directory '/home/fankang/kubernetes/kubernetes-1.5.2'
+++ [1210 14:31:32] Building the toolchain targets:
    k8s.io/kubernetes/hack/cmd/teststale
    k8s.io/kubernetes/vendor/github.com/jteeuwen/go-bindata/go-bindata
+++ [1210 14:31:32] Generating bindata:
    test/e2e/framework/gobindata_util.go
+++ [1210 14:31:33] Building go targets for linux/amd64:
    cmd/gendocs
    cmd/genkubedocs
    cmd/genman
    cmd/genyaml
    federation/cmd/genfeddocs
make: Leaving directory '/home/fankang/kubernetes/kubernetes-1.5.2'
Generated docs have been placed in the repository tree. Running hack/update-munge-docs.sh.
fatal: Not a git repository (or any of the parent directories): .git
!!! Error in /home/fankang/kubernetes/kubernetes-1.5.2/hack/update-munge-docs.sh:24
  'git_upstream=$(kube::util::git_upstream_remote_name)' exited with status 1
Call stack:
  1: /home/fankang/kubernetes/kubernetes-1.5.2/hack/update-munge-docs.sh:24 main(...)
Exiting with status 1
!!! Error in hack/../hack/update-generated-docs.sh:57
  '"${KUBE_ROOT}/hack/update-munge-docs.sh"' exited with status 1
Call stack:
  1: hack/../hack/update-generated-docs.sh:57 main(...)
Exiting with status 1
Updating generated-docs FAILED

现在修改/pkg/apis/extensions/register.go，添加Application和ApplicationList的注册：

1
2
3
4
5
6
7
8
func addKnownTypes(scheme *runtime.Scheme) error {
        // TODO this gets cleaned up when the types are fixed
        scheme.AddKnownTypes(SchemeGroupVersion,
                &Application{},
                &ApplicationList{},
                &Deployment{},
                &DeploymentList{},
				......

同样，修改/pkg/apis/extensions/v1beta1/register.go，添加Application和ApplicationList的注册：

1
2
3
4
5
6
7
func addKnownTypes(scheme *runtime.Scheme) error {
        scheme.AddKnownTypes(SchemeGroupVersion,
                &Application{},
                &ApplicationList{},
                &Deployment{},
                &DeploymentList{},
				......

至此，已经完成apiserver侧的资源添加，但此时，资源对应的storage还未定义，所以资源还处在不可用状态。

添加Storage

首先在/pkg/registry/extensions目录下添加application目录。

在application目录下添加doc.go:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
/*
Copyright 2015 The Kubernetes Authors.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
    http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
package application // import "k8s.io/kubernetes/pkg/registry/extensions/application"

然后添加etcd目录，并在etcd目录下，即/pkg/registry/extensions/application/etcd下添加etcd.go：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
/*
Copyright 2015 The Kubernetes Authors.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
    http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
package etcd
import (
        "k8s.io/kubernetes/pkg/api"
        "k8s.io/kubernetes/pkg/apis/extensions"
        "k8s.io/kubernetes/pkg/registry/cachesize"
        "k8s.io/kubernetes/pkg/registry/extensions/application"
        "k8s.io/kubernetes/pkg/registry/generic"
        "k8s.io/kubernetes/pkg/registry/generic/registry"
        "k8s.io/kubernetes/pkg/runtime"
        "k8s.io/kubernetes/pkg/storage"
)
// ApplicationStorage
type ApplicationStorage struct {
	Application *REST
}
func NewStorage(opts generic.RESTOptions) ApplicationStorage {
	applicationRest := NewREST(opts)
	return ApplicationStorage{
		Application: applicationRest,
	}
}
type REST struct {
	*registry.Store
}
// NewREST returns a RESTStorage object that will work against applications.
func NewREST(opts generic.RESTOptions) (*REST) {
	prefix := "/" + opts.ResourcePrefix
	newListFunc := func() runtime.Object { return &extensions.ApplicationList{} }
	storageInterface, dFunc := opts.Decorator(
		opts.StorageConfig,
		cachesize.GetWatchCacheSizeByResource(cachesize.Applications),
		&extensions.Application{},
		prefix,
		application.Strategy,
		newListFunc,
		storage.NoTriggerPublisher,
	)
	store := &registry.Store{
		NewFunc: func() runtime.Object { return &extensions.Application{} },
		// NewListFunc returns an object capable of storing results of an etcd list.
		NewListFunc: newListFunc,
		// Produces a path that etcd understands, to the root of the resource
		// by combining the namespace in the context with the given prefix.
		KeyRootFunc: func(ctx api.Context) string {
			return registry.NamespaceKeyRootFunc(ctx, prefix)
		},
		// Produces a path that etcd understands, to the resource by combining
		// the namespace in the context with the given prefix.
		KeyFunc: func(ctx api.Context, name string) (string, error) {
			return registry.NamespaceKeyFunc(ctx, prefix, name)
		},
		// Retrieve the name field of an application.
		ObjectNameFunc: func(obj runtime.Object) (string, error) {
			return obj.(*extensions.Application).Name, nil
		},
		// Used to match objects based on labels/fields for list.
		PredicateFunc:           application.MatchApplication,
		QualifiedResource:       extensions.Resource("applications"),
		EnableGarbageCollection: opts.EnableGarbageCollection,
		DeleteCollectionWorkers: opts.DeleteCollectionWorkers,
		// Used to validate application creation.
		CreateStrategy: application.Strategy,
		// Used to validate application updates.
		UpdateStrategy: application.Strategy,
		DeleteStrategy: application.Strategy,
		Storage:     storageInterface,
		DestroyFunc: dFunc,
	}
	return &REST{store}
}

现在我们可以看到，在NewREST()函数中，我们还缺cachesize.Applications及application.Strategy还未定义。

所以修改/pkg/registry/cachesize/cachesize.go，添加Application的定义：

1
2
3
4
5
const (
        Applications               Resource = "applications"
        CertificateSigningRequests Resource = "certificatesigningrequests"
        ClusterRoles               Resource = "clusterroles"
		......

再来定义application.Strategy，在/pkg/registry/extensions/application/目录下添加strategy.go：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
/*
Copyright 2015 The Kubernetes Authors.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
    http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
package application
import (
	"fmt"
	"k8s.io/kubernetes/pkg/api"
	"k8s.io/kubernetes/pkg/apis/extensions"
	"k8s.io/kubernetes/pkg/apis/extensions/validation"
	"k8s.io/kubernetes/pkg/fields"
	"k8s.io/kubernetes/pkg/labels"
	"k8s.io/kubernetes/pkg/registry/generic"
	"k8s.io/kubernetes/pkg/runtime"
	apistorage "k8s.io/kubernetes/pkg/storage"
	"k8s.io/kubernetes/pkg/util/validation/field"
)
// applicationStrategy implements behavior for Deployments.
type applicationStrategy struct {
	runtime.ObjectTyper
	api.NameGenerator
}
// Strategy is the default logic that applies when creating and updating Application
// objects via the REST API.
var Strategy = applicationStrategy{api.Scheme, api.SimpleNameGenerator}
// NamespaceScoped is true for application.
func (applicationStrategy) NamespaceScoped() bool {
	return true
}
// PrepareForCreate clears fields that are not allowed to be set by end users on creation.
func (applicationStrategy) PrepareForCreate(ctx api.Context, obj runtime.Object) {
	application := obj.(*extensions.Application)
	application.Generation = 1
}
// Validate validates a new application.
func (applicationStrategy) Validate(ctx api.Context, obj runtime.Object) field.ErrorList {
	application := obj.(*extensions.Application)
	return validation.ValidateApplication(application)
}
// Canonicalize normalizes the object after validation.
func (applicationStrategy) Canonicalize(obj runtime.Object) {
}
// AllowCreateOnUpdate is false for application.
func (applicationStrategy) AllowCreateOnUpdate() bool {
	return false
}
// PrepareForUpdate clears fields that are not allowed to be set by end users on update.
func (applicationStrategy) PrepareForUpdate(ctx api.Context, obj, old runtime.Object) {
}
// ValidateUpdate is the default update validation for an end user.
func (applicationStrategy) ValidateUpdate(ctx api.Context, obj, old runtime.Object) field.ErrorList {
	return validation.ValidateApplicationUpdate(obj.(*extensions.Application), old.(*extensions.Application))
}
// AllowUnconditionalUpdate is the default update policy for application objects.
func (applicationStrategy) AllowUnconditionalUpdate() bool {
	return true
}
// ApplicationToSelectableFields returns a field set that represents the object.
func ApplicationToSelectableFields(application *extensions.Application) fields.Set {
	return generic.ObjectMetaFieldsSet(&application.ObjectMeta, true)
}
// MatchApplication is the filter used by the generic etcd backend to route
// watch events from etcd to clients of the apiserver only interested in specific
// labels/fields.
func MatchApplication(label labels.Selector, field fields.Selector) apistorage.SelectionPredicate {
	return apistorage.SelectionPredicate{
		Label: label,
		Field: field,
		GetAttrs: func(obj runtime.Object) (labels.Set, fields.Set, error) {
			application, ok := obj.(*extensions.Application)
			if !ok {
				return nil, nil, fmt.Errorf("given object is not a application.")
			}
			return labels.Set(application.ObjectMeta.Labels), ApplicationToSelectableFields(application), nil
		},
	}
}

strategy.go中的这些方法或函数必须定义，至于要实现的功能，还要看具体的需求。当然，由于参考了Deployment的实现，applicationStrategy的方法Validate()调用了validate.ValidateApplication()，所以我们要去定义ValidateApplication()，定义在/pkg/apis/extensions/validation/validation.go中：

1
2
3
4
5
6
7
func ValidateApplication(obj *extensions.Application) field.ErrorList {
        return apivalidation.ValidateObjectMeta(&obj.ObjectMeta, true, ValidateDeploymentName, field.NewPath("metadata"))
}
func ValidateApplicationUpdate(update, old *extensions.Application) field.ErrorList {
        return apivalidation.ValidateObjectMeta(&update.ObjectMeta, true, ValidateDeploymentName, field.NewPath("metadata"))
}

至此我们有了Application storage，现在可以在RESTStorageProvider的v1beta1Storage()中添加该storage了，在/pkg/registry/extensions/rest/storage_extensions.go中：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
import (
        ......
        applicationetcd "k8s.io/kubernetes/pkg/registry/extensions/application/etcd"
        ......
)
func (p RESTStorageProvider) v1beta1Storage(apiResourceConfigSource genericapiserver.APIResourceConfigSource, restOptionsGetter genericapiserver.RESTOptionsGetter) map[string]rest.Storage {
        version := extensionsapiv1beta1.SchemeGroupVersion
        storage := map[string]rest.Storage{}
		......
		if apiResourceConfigSource.ResourceEnabled(version.WithResource("applications")) {
                applicationStorage := applicationetcd.NewStorage(restOptionsGetter(extensions.Resource("applications")))
                storage["applications"] = applicationStorage.Application
        }
		......

这里可以看到，apiResourceConfigSource.ResourceEnabled(version.WithResource(“applications”))的结果是false，因为我们还没有在apiResourceConfigSource中注册Application。
修改/pkg/master/master.go的DefaultAPIResourceConfigSource():

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
func DefaultAPIResourceConfigSource() *genericapiserver.ResourceConfig {
	......
	// all extensions resources except these are disabled by default
	ret.EnableResources(
		extensionsapiv1beta1.SchemeGroupVersion.WithResource("applications"),
		extensionsapiv1beta1.SchemeGroupVersion.WithResource("daemonsets"),
		extensionsapiv1beta1.SchemeGroupVersion.WithResource("deployments"),
		extensionsapiv1beta1.SchemeGroupVersion.WithResource("horizontalpodautoscalers"),
		extensionsapiv1beta1.SchemeGroupVersion.WithResource("ingresses"),
		extensionsapiv1beta1.SchemeGroupVersion.WithResource("jobs"),
		extensionsapiv1beta1.SchemeGroupVersion.WithResource("networkpolicies"),
		extensionsapiv1beta1.SchemeGroupVersion.WithResource("replicasets"),
		extensionsapiv1beta1.SchemeGroupVersion.WithResource("thirdpartyresources"),
	)
	return ret
}

添加HumanPrinter

之前分析过，kubectl get pod这类命令调用的是HumanPrinter。现在我们就需要在HumanPrinter中添加Application的打印函数。

修改/pkg/kubectl/resource_printer.go中HumanReadablePrinter的addDefaultHandlers()：

1
2
3
4
5
func (h *HumanReadablePrinter) addDefaultHandlers() {
	......
	h.Handler(applicationColumns, printApplication)
    h.Handler(applicationColumns, printApplicationList)
	......

可以看到，我们现在缺applicationColumns，printApplication和printApplicationList。

先来定义applicationColumns，在同文件下的var xxxColumns列表定义中添加applicationColumns:

1
2
3
4
withNamespacePrefixColumns       = []string{"NAMESPACE"} // TODO(erictune): print cluster name too.
deploymentColumns                = []string{"NAME", "DESIRED", "CURRENT", "UP-TO-DATE", "AVAILABLE", "AGE"}
applicationColumns                = []string{"NAME", "DESIRED", "AGE"}
configMapColumns                 = []string{"NAME", "DATA", "AGE"}

再来定义printApplication和printApplicationList，在相同文件下，添加：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
func printApplication(application *extensions.Application, w io.Writer, options PrintOptions) error {
        name := formatResourceName(options.Kind, application.Name, options.WithKind)
        if options.WithNamespace {
                if _, err := fmt.Fprintf(w, "%s\t", application.Namespace); err != nil {
                        return err
                }
        }
        desiredReplicas := application.Replicas
        age := translateTimestamp(application.CreationTimestamp)
        if _, err := fmt.Fprintf(w, "%s\t%d\t%s", name, desiredReplicas, age); err != nil {
                return err
        }
        if _, err := fmt.Fprint(w, AppendLabels(application.Labels, options.ColumnLabels)); err != nil {
                return err
        }
        _, err := fmt.Fprint(w, AppendAllLabels(options.ShowLabels, application.Labels))
        return err
}
func printApplicationList(list *extensions.ApplicationList, w io.Writer, options PrintOptions) error {
        for _, item := range list.Items {
                if err := printApplication(&item, w, options); err != nil {
                        return err
                }
        }
        return nil
}

现在，我们可以执行make编译了。

验证

可以使用下面文件创建Application：

1
2
3
4
5
6
7
8
9
{
  "apiVersion": "extensions/v1beta1",
  "kind": "Application",
  "metadata": {
    "name": "ubuntu1",
    "namespace": "default"
  },
  "replicas": 20
}

可以使用kubectl get application获取application列表：

1
2
3
4
root@fankang:/home/fankang/kubernetes/kubernetes-1.5.2# ./_output/bin/kubectl get application
NAME      DESIRED   AGE
ubuntu    20        4d
ubuntu1   20        3d

也可以查看具体的application：

1
2
3
4
5
6
7
8
9
10
11
12
root@fankang:/home/fankang/kubernetes/kubernetes-1.5.2# ./_output/bin/kubectl get application ubuntu -o yaml
apiVersion: extensions/v1beta1
kind: Application
metadata:
  creationTimestamp: 2017-12-06T11:37:22Z
  generation: 1
  name: ubuntu
  namespace: default
  resourceVersion: "448526"
  selfLink: /apis/extensions/v1beta1/namespaces/default/applications/ubuntu
  uid: d2fb278c-da79-11e7-adb3-0800274a4ec3
replicas: 20

资源添加完成。