tar包
docker load/save命令可以从tar包导入镜像或把镜像压缩成tar包(一般使用tar格式即可),其中tar包中可以包含多个镜像。我们以命令行docker save nginx:v1 golint:v1 > test.tar
生成test.tar包,然后分析test.tar包中的内容。
repositories
repositories文件中记录了该tar包包含的镜像及镜像最上层的layerID。
1 2 3 4 5 6 7 8
| { "golint": { "v1": "47483a1b2c5e1cb2381d9ed73358753264063e70f329d3f8fffb51b4c861ab5e" }, "nginx": { "v1": "77b9c04c054e859ffc5d0bb6d89084dd543ffc2386ffbe17da7bc43147c10d49" } }
|
manifest.json
manifest.json中记录了镜像的信息,包括:
- Config: 镜像的基本信息;
- RepoTags: 镜像的tag名称;
- Layers: 镜像中包含的层,为layerID/layer.tar。
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38
| [ { "Config": "3034d86f1f0b0379092bb99565d72c1dd4a94b41c1e83bb2d0714bb98720ac16.json", "RepoTags": [ "nginx:v1" ], "Layers": [ "707f1f25f845bd495c21c368a57666f6c4e1b0358868e43e0376fd55512f8734/layer.tar", "3fbeaf15931df30802b43e1147f6c40d3cbe2839bb91e49cf068e58dadb3ec90/layer.tar", "a599739916ea730466ba3a21e5493faf34808cbddc8c3570933048d604cde37d/layer.tar", "0cfdb29ee5612340e908c304954decd7a2c001405abeb475d8e773f66071211b/layer.tar", "aa1b48ac3dd21b33515f37763a1f4cafce662496c6ed3c13326a818d8669f06a/layer.tar", "90627bd0fa0408d7488e79525d6b2810987c5ed579f7f649993ca29a1d819a69/layer.tar", "2a9bb8e69f5a34ead65daecf9ff156975e13c5a29bb27b0c93de879c2787d616/layer.tar", "01a90859e4ca4e3138adbadb3a76be9014b0ea5b89b6ef24988e70e7d7bfe12b/layer.tar", "e27a3a829a7d9d07bec30f000c722c47e428f9c4cc975dbe2b0dbffda189a8f7/layer.tar", "77b9c04c054e859ffc5d0bb6d89084dd543ffc2386ffbe17da7bc43147c10d49/layer.tar" ] }, { "Config": "8b3c2dabe960cf6ed115d1fbf4b93cc46e4e9823f3cfc1f430c96944024c7fa1.json", "RepoTags": [ "golint:v1" ], "Layers": [ "707f1f25f845bd495c21c368a57666f6c4e1b0358868e43e0376fd55512f8734/layer.tar", "3fbeaf15931df30802b43e1147f6c40d3cbe2839bb91e49cf068e58dadb3ec90/layer.tar", "a599739916ea730466ba3a21e5493faf34808cbddc8c3570933048d604cde37d/layer.tar", "0cfdb29ee5612340e908c304954decd7a2c001405abeb475d8e773f66071211b/layer.tar", "aa1b48ac3dd21b33515f37763a1f4cafce662496c6ed3c13326a818d8669f06a/layer.tar", "90627bd0fa0408d7488e79525d6b2810987c5ed579f7f649993ca29a1d819a69/layer.tar", "2f96598b6febc88b8aa7fb649d8fef1ad08c3bc08e32866b0ca882ecd54608e3/layer.tar", "8a4a76326100661bebca4c18a8465d2ab1c19f31083175345ccac623fd8e5810/layer.tar", "9f10635765dbd0d56823ff9b303472a341e163af102f71c5f9abd9ee219a34d9/layer.tar", "47483a1b2c5e1cb2381d9ed73358753264063e70f329d3f8fffb51b4c861ab5e/layer.tar" ] } ]
|
image-id.json
image-id.json中记录了镜像相关的信息,包括,镜像的config信息,history信息,diffID信息等。
Load流程
接下来来看docker load的流程。
client端
先来看docker load命令的定义,在/api/client/image/load.go中:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20
| func NewLoadCommand(dockerCli *client.DockerCli) *cobra.Command { var opts loadOptions cmd := &cobra.Command{ Use: "load [OPTIONS]", Short: "Load an image from a tar archive or STDIN", Args: cli.NoArgs, RunE: func(cmd *cobra.Command, args []string) error { return runLoad(dockerCli, opts) }, } flags := cmd.Flags() flags.StringVarP(&opts.input, "input", "i", "", "Read from tar archive file, instead of STDIN") flags.BoolVarP(&opts.quiet, "quiet", "q", false, "Suppress the load output") return cmd }
|
runLoad()定义如下:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27
| func runLoad(dockerCli *client.DockerCli, opts loadOptions) error { var input io.Reader = dockerCli.In() if opts.input != "" { file, err := os.Open(opts.input) if err != nil { return err } defer file.Close() input = file } if !dockerCli.IsTerminalOut() { opts.quiet = true } response, err := dockerCli.Client().ImageLoad(context.Background(), input, opts.quiet) if err != nil { return err } defer response.Body.Close() if response.Body != nil && response.JSON { return jsonmessage.DisplayJSONMessagesStream(response.Body, dockerCli.Out(), dockerCli.OutFd(), dockerCli.IsTerminalOut(), nil) } _, err = io.Copy(dockerCli.Out(), response.Body) return err }
|
可以看到,runLoad()主要调用client的ImageLoad()方法。
client的ImageLoad()方法定义在第三方库/engine-api/client/image_load.go中:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16
| func (cli *Client) ImageLoad(ctx context.Context, input io.Reader, quiet bool) (types.ImageLoadResponse, error) { v := url.Values{} v.Set("quiet", "0") if quiet { v.Set("quiet", "1") } headers := map[string][]string{"Content-Type": {"application/x-tar"}} resp, err := cli.postRaw(ctx, "/images/load", v, input, headers) if err != nil { return types.ImageLoadResponse{}, err } return types.ImageLoadResponse{ Body: resp.body, JSON: resp.header.Get("Content-Type") == "application/json", }, nil }
|
所以,client的ImageLoad()方法是使用post方法请求/images/load。
server端
server端使用postImagesLoad()方法处理/images/load的post请求,定义在/api/server/router/image/image_routes.go中:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18
| func (s *imageRouter) postImagesLoad(ctx context.Context, w http.ResponseWriter, r *http.Request, vars map[string]string) error { if err := httputils.ParseForm(r); err != nil { return err } quiet := httputils.BoolValueOrDefault(r, "quiet", true) if !quiet { w.Header().Set("Content-Type", "application/json") output := ioutils.NewWriteFlusher(w) defer output.Close() if err := s.backend.LoadImage(r.Body, output, quiet); err != nil { output.Write(streamformatter.NewJSONStreamFormatter().FormatError(err)) } return nil } return s.backend.LoadImage(r.Body, w, quiet) }
|
可以看到postImagesLoad()方法调用了daemon的LoadImage()方法。
Daemon的LoadImage()方法定义在/daemon/image_exporter.go中:
1 2 3 4
| func (daemon *Daemon) LoadImage(inTar io.ReadCloser, outStream io.Writer, quiet bool) error { imageExporter := tarexport.NewTarExporter(daemon.imageStore, daemon.layerStore, daemon.referenceStore, daemon) return imageExporter.Load(inTar, outStream, quiet) }
|
LoadImage()先生成一个imageExporter,然后调用imageExporter.Load()完成
imageExporter
我们来看imageExporter的Load()方法,定义在/image/tarexport/load.go中:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139
| func (l *tarexporter) Load(inTar io.ReadCloser, outStream io.Writer, quiet bool) error { var ( sf = streamformatter.NewJSONStreamFormatter() progressOutput progress.Output ) if !quiet { progressOutput = sf.NewProgressOutput(outStream, false) outStream = &streamformatter.StdoutFormatter{Writer: outStream, StreamFormatter: streamformatter.NewJSONStreamFormatter()} } tmpDir, err := ioutil.TempDir("", "docker-import-") if err != nil { return err } defer os.RemoveAll(tmpDir) if err := chrootarchive.Untar(inTar, tmpDir, nil); err != nil { return err } manifestPath, err := safePath(tmpDir, manifestFileName) if err != nil { return err } manifestFile, err := os.Open(manifestPath) if err != nil { if os.IsNotExist(err) { return l.legacyLoad(tmpDir, outStream, progressOutput) } return manifestFile.Close() } defer manifestFile.Close() var manifest []manifestItem if err := json.NewDecoder(manifestFile).Decode(&manifest); err != nil { return err } var parentLinks []parentLink var imageIDsStr string var imageRefCount int for _, m := range manifest { configPath, err := safePath(tmpDir, m.Config) if err != nil { return err } config, err := ioutil.ReadFile(configPath) if err != nil { return err } img, err := image.NewFromJSON(config) if err != nil { return err } var rootFS image.RootFS rootFS = *img.RootFS rootFS.DiffIDs = nil if expected, actual := len(m.Layers), len(img.RootFS.DiffIDs); expected != actual { return fmt.Errorf("invalid manifest, layers length mismatch: expected %q, got %q", expected, actual) } for i, diffID := range img.RootFS.DiffIDs { layerPath, err := safePath(tmpDir, m.Layers[i]) if err != nil { return err } r := rootFS r.Append(diffID) newLayer, err := l.ls.Get(r.ChainID()) if err != nil { newLayer, err = l.loadLayer(layerPath, rootFS, diffID.String(), m.LayerSources[diffID], progressOutput) if err != nil { return err } } defer layer.ReleaseAndLog(l.ls, newLayer) if expected, actual := diffID, newLayer.DiffID(); expected != actual { return fmt.Errorf("invalid diffID for layer %d: expected %q, got %q", i, expected, actual) } rootFS.Append(diffID) } imgID, err := l.is.Create(config) if err != nil { return err } imageIDsStr += fmt.Sprintf("Loaded image ID: %s\n", imgID) imageRefCount = 0 for _, repoTag := range m.RepoTags { named, err := reference.ParseNamed(repoTag) if err != nil { return err } ref, ok := named.(reference.NamedTagged) if !ok { return fmt.Errorf("invalid tag %q", repoTag) } l.setLoadedTag(ref, imgID, outStream) outStream.Write([]byte(fmt.Sprintf("Loaded image: %s\n", ref))) imageRefCount++ } parentLinks = append(parentLinks, parentLink{imgID, m.Parent}) l.loggerImgEvent.LogImageEvent(imgID.String(), imgID.String(), "load") } for _, p := range validatedParentLinks(parentLinks) { if p.parentID != "" { if err := l.setParentID(p.id, p.parentID); err != nil { return err } } } if imageRefCount == 0 { outStream.Write([]byte(imageIDsStr)) } return nil }
|
Load()的流程如下:
- 建立tmpDir,把tar包解压到该目录中(通过chrootarchive包完成);
- 读取manifest.json,分别按下面流程处理每一个镜像;
- 读取镜像的config文件,并在生成image对象;
- 调用LoadLayer()方法处理镜像每一层;
- 调用imageStore的Create()在imageStore中创建该镜像;
- 处理manifest.json中的tag名称;
- 处理parent关系。
这里具体每个细节的实现就不再展开分析,基本上调用了iamgeStore, layerStore, referenceStore这些Store中定义的方法。
Save流程
接下来来看docker save的流程。
client端
docker save定义在/api/client/image/save.go中:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20
| func NewSaveCommand(dockerCli *client.DockerCli) *cobra.Command { var opts saveOptions cmd := &cobra.Command{ Use: "save [OPTIONS] IMAGE [IMAGE...]", Short: "Save one or more images to a tar archive (streamed to STDOUT by default)", Args: cli.RequiresMinArgs(1), RunE: func(cmd *cobra.Command, args []string) error { opts.images = args return runSave(dockerCli, opts) }, } flags := cmd.Flags() flags.StringVarP(&opts.output, "output", "o", "", "Write to a file, instead of STDOUT") return cmd }
|
可以看到,NewSaveCommand()会把args赋值给opts.images。
runSave()定义如下:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18
| func runSave(dockerCli *client.DockerCli, opts saveOptions) error { if opts.output == "" && dockerCli.IsTerminalOut() { return errors.New("Cowardly refusing to save to a terminal. Use the -o flag or redirect.") } responseBody, err := dockerCli.Client().ImageSave(context.Background(), opts.images) if err != nil { return err } defer responseBody.Close() if opts.output == "" { _, err := io.Copy(dockerCli.Out(), responseBody) return err } return client.CopyToFile(opts.output, responseBody) }
|
runSave()主要调用client的ImageSave()方法。
client的ImageSave()方法定义在第三方库/engine-api/client/image_save.go中:
1 2 3 4 5 6 7 8 9 10 11
| func (cli *Client) ImageSave(ctx context.Context, imageIDs []string) (io.ReadCloser, error) { query := url.Values{ "names": imageIDs, } resp, err := cli.get(ctx, "/images/get", query, nil) if err != nil { return nil, err } return resp.body, nil }
|
ImageSave()把images封装在names中,然后使用GET方法请求/images/get路径。
server端
server端通过getImageGet()方法处理/images/get的GET请求,定义在/api/server/router/image/image_routes.go中:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25
| func (s *imageRouter) getImagesGet(ctx context.Context, w http.ResponseWriter, r *http.Request, vars map[string]string) error { if err := httputils.ParseForm(r); err != nil { return err } w.Header().Set("Content-Type", "application/x-tar") output := ioutils.NewWriteFlusher(w) defer output.Close() var names []string if name, ok := vars["name"]; ok { names = []string{name} } else { names = r.Form["names"] } if err := s.backend.ExportImage(names, output); err != nil { if !output.Flushed() { return err } sf := streamformatter.NewJSONStreamFormatter() output.Write(sf.FormatError(err)) } return nil }
|
getImagesGet()先从请求中解析出names,然后调用daemon的ExportImage()来导出所有镜像。
daemon的ExportImage()定义在/daemon/image_exporter.go中:
1 2 3 4
| func (daemon *Daemon) ExportImage(names []string, outStream io.Writer) error { imageExporter := tarexport.NewTarExporter(daemon.imageStore, daemon.layerStore, daemon.referenceStore, daemon) return imageExporter.Save(names, outStream) }
|
ExportImage()先生成imageExporter,然后调用imageExporter.Save()完成镜像导出。
imageExporter
来看Save()方法,定义在/image/tarexport/save.go中:
1 2 3 4 5 6 7 8 9 10
| func (l *tarexporter) Save(names []string, outStream io.Writer) error { images, err := l.parseNames(names) if err != nil { return err } return (&saveSession{tarexporter: l, images: images}).save(outStream) }
|
可以看到,最后调用的是saveSession的save()方法,定义如下:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107
| func (s *saveSession) save(outStream io.Writer) error { s.savedLayers = make(map[string]struct{}) s.diffIDPaths = make(map[layer.DiffID]string) tempDir, err := ioutil.TempDir("", "docker-export-") if err != nil { return err } defer os.RemoveAll(tempDir) s.outDir = tempDir reposLegacy := make(map[string]map[string]string) var manifest []manifestItem var parentLinks []parentLink for id, imageDescr := range s.images { foreignSrcs, err := s.saveImage(id) if err != nil { return err } var repoTags []string var layers []string for _, ref := range imageDescr.refs { if _, ok := reposLegacy[ref.Name()]; !ok { reposLegacy[ref.Name()] = make(map[string]string) } reposLegacy[ref.Name()][ref.Tag()] = imageDescr.layers[len(imageDescr.layers)-1] repoTags = append(repoTags, ref.String()) } for _, l := range imageDescr.layers { layers = append(layers, filepath.Join(l, legacyLayerFileName)) } manifest = append(manifest, manifestItem{ Config: digest.Digest(id).Hex() + ".json", RepoTags: repoTags, Layers: layers, LayerSources: foreignSrcs, }) parentID, _ := s.is.GetParent(id) parentLinks = append(parentLinks, parentLink{id, parentID}) s.tarexporter.loggerImgEvent.LogImageEvent(id.String(), id.String(), "save") } for i, p := range validatedParentLinks(parentLinks) { if p.parentID != "" { manifest[i].Parent = p.parentID } } if len(reposLegacy) > 0 { reposFile := filepath.Join(tempDir, legacyRepositoriesFileName) f, err := os.OpenFile(reposFile, os.O_RDWR|os.O_CREATE|os.O_TRUNC, 0644) if err != nil { f.Close() return err } if err := json.NewEncoder(f).Encode(reposLegacy); err != nil { return err } if err := f.Close(); err != nil { return err } if err := system.Chtimes(reposFile, time.Unix(0, 0), time.Unix(0, 0)); err != nil { return err } } manifestFileName := filepath.Join(tempDir, manifestFileName) f, err := os.OpenFile(manifestFileName, os.O_RDWR|os.O_CREATE|os.O_TRUNC, 0644) if err != nil { f.Close() return err } if err := json.NewEncoder(f).Encode(manifest); err != nil { return err } if err := f.Close(); err != nil { return err } if err := system.Chtimes(manifestFileName, time.Unix(0, 0), time.Unix(0, 0)); err != nil { return err } fs, err := archive.Tar(tempDir, archive.Uncompressed) if err != nil { return err } defer fs.Close() if _, err := io.Copy(outStream, fs); err != nil { return err } return nil }
|
save()方法调用saveImage()把镜像进行导出,然后处理manifest.json文件,这些,都在一个临时文件中,最后把临时文件中的数据打包到输出。
saveImage()定义如下:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62
| func (s *saveSession) saveImage(id image.ID) (map[layer.DiffID]distribution.Descriptor, error) { img, err := s.is.Get(id) if err != nil { return nil, err } if len(img.RootFS.DiffIDs) == 0 { return nil, fmt.Errorf("empty export - not implemented") } var parent digest.Digest var layers []string var foreignSrcs map[layer.DiffID]distribution.Descriptor for i := range img.RootFS.DiffIDs { v1Img := image.V1Image{} if i == len(img.RootFS.DiffIDs)-1 { v1Img = img.V1Image } rootFS := *img.RootFS rootFS.DiffIDs = rootFS.DiffIDs[:i+1] v1ID, err := v1.CreateID(v1Img, rootFS.ChainID(), parent) if err != nil { return nil, err } v1Img.ID = v1ID.Hex() if parent != "" { v1Img.Parent = parent.Hex() } src, err := s.saveLayer(rootFS.ChainID(), v1Img, img.Created) if err != nil { return nil, err } layers = append(layers, v1Img.ID) parent = v1ID if src.Digest != "" { if foreignSrcs == nil { foreignSrcs = make(map[layer.DiffID]distribution.Descriptor) } foreignSrcs[img.RootFS.DiffIDs[i]] = src } } configFile := filepath.Join(s.outDir, digest.Digest(id).Hex()+".json") if err := ioutil.WriteFile(configFile, img.RawJSON(), 0644); err != nil { return nil, err } if err := system.Chtimes(configFile, img.Created, img.Created); err != nil { return nil, err } s.images[id].layers = layers return foreignSrcs, nil }
|
saveImage()会调用saveLayer()方法导出layer,最后生成镜像的config文件。
saveLayer()定义如下:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77
| func (s *saveSession) saveLayer(id layer.ChainID, legacyImg image.V1Image, createdTime time.Time) (distribution.Descriptor, error) { if _, exists := s.savedLayers[legacyImg.ID]; exists { return distribution.Descriptor{}, nil } outDir := filepath.Join(s.outDir, legacyImg.ID) if err := os.Mkdir(outDir, 0755); err != nil { return distribution.Descriptor{}, err } if err := ioutil.WriteFile(filepath.Join(outDir, legacyVersionFileName), []byte("1.0"), 0644); err != nil { return distribution.Descriptor{}, err } imageConfig, err := json.Marshal(legacyImg) if err != nil { return distribution.Descriptor{}, err } if err := ioutil.WriteFile(filepath.Join(outDir, legacyConfigFileName), imageConfig, 0644); err != nil { return distribution.Descriptor{}, err } layerPath := filepath.Join(outDir, legacyLayerFileName) l, err := s.ls.Get(id) if err != nil { return distribution.Descriptor{}, err } defer layer.ReleaseAndLog(s.ls, l) if oldPath, exists := s.diffIDPaths[l.DiffID()]; exists { relPath, err := filepath.Rel(outDir, oldPath) if err != nil { return distribution.Descriptor{}, err } os.Symlink(relPath, layerPath) } else { tarFile, err := os.Create(layerPath) if err != nil { return distribution.Descriptor{}, err } defer tarFile.Close() arch, err := l.TarStream() if err != nil { return distribution.Descriptor{}, err } defer arch.Close() if _, err := io.Copy(tarFile, arch); err != nil { return distribution.Descriptor{}, err } for _, fname := range []string{"", legacyVersionFileName, legacyConfigFileName, legacyLayerFileName} { if err := system.Chtimes(filepath.Join(outDir, fname), createdTime, createdTime); err != nil { return distribution.Descriptor{}, err } } s.diffIDPaths[l.DiffID()] = layerPath } s.savedLayers[legacyImg.ID] = struct{}{} var src distribution.Descriptor if fs, ok := l.(distribution.Describable); ok { src = fs.Descriptor() } return src, nil }
|
saveLayer()主要是生成layer中的tar文件,version文件,config文件,具体就不再展开分析。
总结
docker load和docker save命令都会使用临时目录做为中间过渡,其本质是把tar包中的内容写到Docker镜像库中,或从Docker镜像库中提取信息生成tar包中的内容。至于打包或解包操作,已在之前”Docker工具包分析-archive-v1.12.3”中分析过。