Android编译原理之make编译过程

cover

一、make

以高通的代码为例，高通在编译时使用如下的指令进行编译，实质是对make的封装

1
2
./build.sh --target_only

在执行一系列check的函数后，最终调用make执行执行编译。而make函数是在build/envsetup.sh时声明。

从get_make_command函数可知，make后，真正然后执行编译的入口是：build/soong/soong_ui.bash

二、soong_ui.bash

  1. source microfactory.bash，得到一些函数命令， 例如：soong_build_go
  2. 编译/build/soong/cmd/soong_ui/main.go，生成 out/soong_ui这个可执行程序
  3. 执行命令：out/soong_ui –make-mode ，执行了make命令，会把”build/make/core/main.mk” 加到构建环境中，同时启动kati、blueprint-soong、ninja的编译。
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
function gettop
{
    local TOPFILE=build/soong/root.bp
    if [ -n "${TOP-}" -a -f "${TOP-}/${TOPFILE}" ] ; then
        # The following circumlocution ensures we remove symlinks from TOP.
        (cd $TOP; PWD= /bin/pwd)
    else
        if [ -f $TOPFILE ] ; then
            # The following circumlocution (repeated below as well) ensures
            # that we record the true directory name and not one that is
            # faked up with symlink names.
            PWD= /bin/pwd
        else
            local HERE=$PWD
            T=
            while [ \\( ! \\( -f $TOPFILE \\) \\) -a \\( $PWD != "/" \\) ]; do
                \\cd ..
                T=`PWD= /bin/pwd -P`
            done
            \\cd $HERE
            if [ -f "$T/$TOPFILE" ]; then
                echo $T
            fi
        fi
    fi
}

# Save the current PWD for use in soong_ui
export ORIGINAL_PWD=${PWD}
export TOP=$(gettop)
source ${TOP}/build/soong/scripts/microfactory.bash

soong_build_go soong_ui android/soong/cmd/soong_ui
soong_build_go mk2rbc android/soong/mk2rbc/cmd
soong_build_go rbcrun rbcrun/cmd

cd ${TOP}
exec "$(getoutdir)/soong_ui" "$@"

2.1 build/soong/scripts/microfactory.bash

build/soong/scripts/microfactory.bash

microfactory.bash得到build_go的函数命令,并提供 soong_build_go的函数执行方法并继续调用

build/blueprint/microfactory/microfactory.bash

GOROOT=prebuilts/go/linux-x86

2.2 build/blueprint/microfactory/microfactory.bash

build_go函数做了以下事情：

  1. 第一次执行编译时调用go run out/.microfactory_Linux/intermediates/src/microfactory.go编译出

microfactory_linux，第二次执行会执行检查，如果已编译则跳过go run

  1. 通过microfactory_Linux编译出soong_ui

soong_build_go soong_ui android/soong/cmd/soong_ui

  • build_go soong_ui android/soong/cmd/soong_ui

实际调用逻辑为：

1
2
3
4
5
6
7
8
GOROOT=$(cd /prebuilts/go/linux-x86/; pwd) /out/microfactory_Linux
  -b "/out/microfactory_Linux" \\
  -pkg-path "github.com/google/blueprint=/build/blueprint" \\
  -trimpath "./" \\
  -pkg-path android/soong=/build/soong
  -pkg-path github.com/golang/protobuf=/external/golang-protobuf} \\
  -o "out/soong_ui" android/soong/cmd/soong_ui

从这儿可以知道soong_build_go soong_ui android/soong/cmd/soong_ui的意思就是从soong/cmd/soong_ui/下的go文件编译生成soong_ui

2.3 soong_ui的源文件main.go

soong_ui 是通过编译 build/soong/cmd/soong_ui/main.go得来

主要执行soong/ui/build/build.go，从build.go就可以看到执行soong的大体流程。

main.go中配置的toBuild为 BuildProductConfig | BuildSoong | BuildKati | BuildNinja，支持productconfig\soong\kati\ninja的构建

2.4 build.go流程

  1. runMakeProductConfig 主要配置编译参数
  2. runSoong 对工具进行编译，编译出blueprint等编译工具, 把*.bp 编译成 out/soong/build.ninja
  3. runKatiBuild, 加载 build/make/core/main.mk， 搜集所有的Android.mk文件生成ninja文件:out/build-${product}.ninja
  4. runKatiPackage, 加载build/make/packaging/main.mk, 编译生成out/build-aosp_arm-package.ninja
  5. createCombinedBuildNinjaFile,将out/soong/build.ninja 、out/build-aosp_arm.ninja和out/build-aosp_arm-package.ninja， 合成为out/combined-aosp_arm.ninja
  6. runNinja，运行Ninja命令， 解析combined-aosp_arm.ninja，执行编译过程

2.4.1 runMakeProductConfig

build/soong/ui/build/dumpvars.go

只是配置编译需要的参数

2.4.2 runSoong

主要功能：

  1. 对工具进行编译，编译出blueprint等编译工具
  2. 把*.bp 编译成 out/soong/build.ninja

2.4.2.1 Android R版本

runSoong()的执行过程：

  1. 执行build/blueprint/bootstrap.bash 生成.minibootstrap/build.ninja 和.bootstrap/build.ninja
  2. 生成minibp\bpglob
  3. 通过ninja来编译.minibootstrap/build.ninja 和.bootstrap/build.ninja

首先执行build/blueprint/bootstrap.bash

bootstrap.bash的作用：

  1. 它可以引导独立的blueprint来生成minibp二进制文件，可以直接运行 ./build/blueprint/bootstrap.bash。
  2. 也可以从另一个脚本调用它来引导基于Bleprint的自定义构建系统。
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
func runSoong(ctx Context, config Config) {
        ctx.BeginTrace(metrics.RunSoong, "soong")
        defer ctx.EndTrace()

        func() {
        // ⽣成out/soong/.minibootstrap/build.ninja
                ctx.BeginTrace(metrics.RunSoong, "blueprint bootstrap")
                defer ctx.EndTrace()

                cmd := Command(ctx, config, "blueprint bootstrap", "build/blueprint/bootstrap.bash", "-t")
                cmd.Environment.Set("BLUEPRINTDIR", "./build/blueprint")
                cmd.Environment.Set("BOOTSTRAP", "./build/blueprint/bootstrap.bash")
                cmd.Environment.Set("BUILDDIR", config.SoongOutDir())
                cmd.Environment.Set("GOROOT", "./"+filepath.Join("prebuilts/go", config.HostPrebuiltTag()))
                cmd.Environment.Set("BLUEPRINT_LIST_FILE", filepath.Join(config.FileListDir(), "Android.bp.list"))
                cmd.Environment.Set("NINJA_BUILDDIR", config.OutDir())
                cmd.Environment.Set("SRCDIR", ".")
                cmd.Environment.Set("TOPNAME", "Android.bp")
                cmd.Sandbox = soongSandbox
// build/blueprint/bootstrap.bash⾥的逻辑很简单，就是创建out/soong/.minibootstrap⽬录，
// 把上⾯⼏个变量写⼊到out/soong/.minibootstrap/build.ninja⾥，
// 最后再写⼊ “include ./build/blueprint/bootstrap/build.ninja”
                cmd.RunAndPrintOrFatal()
        }()

        func() {
                ctx.BeginTrace(metrics.RunSoong, "environment check")
                defer ctx.EndTrace()

                envFile := filepath.Join(config.SoongOutDir(), ".soong.environment")
                envTool := filepath.Join(config.SoongOutDir(), ".bootstrap/bin/soong_env")
                if _, err := os.Stat(envFile); err == nil {
                        if _, err := os.Stat(envTool); err == nil {
                                cmd := Command(ctx, config, "soong_env", envTool, envFile)
                                cmd.Sandbox = soongSandbox

                                var buf strings.Builder
                                cmd.Stdout = &buf
                                cmd.Stderr = &buf
                                if err := cmd.Run(); err != nil {
                                        ctx.Verboseln("soong_env failed, forcing manifest regeneration")
                                        os.Remove(envFile)
                                }

                                if buf.Len() > 0 {
                                        ctx.Verboseln(buf.String())
                                }
                        } else {
                                ctx.Verboseln("Missing soong_env tool, forcing manifest regeneration")
                                os.Remove(envFile)
                        }
                } else if !os.IsNotExist(err) {
                        ctx.Fatalf("Failed to stat %f: %v", envFile, err)
                }
        }()

        var cfg microfactory.Config
        cfg.Map("github.com/google/blueprint", "build/blueprint")

        cfg.TrimPath = absPath(ctx, ".")

        func() {
// 编译minibp程序， minibp跟后⾯要讲到的soong_build都是基于Blueprint框架构建的,
// ⽤来解析Android.bp⾥定义的各种构建单元的。
// 它们俩其实就只有⼀个默认参数不同，即minibp在解析Android.bp时若遇到未知的构建类型，
// 不会报错，即它只会从Android.bp⾥解析出blueprint框架⾥预定义的⼏种构建类型，⽣成对应的
// ninja编译规则，例如minibp会解析build/soong/Aosp下所有Android.bp⾥的bootstrap_go_package;
// 而soong_build是⽤来最终编译整个rom的所有编译命令的，所以它不允许未定义的构建类型。
                ctx.BeginTrace(metrics.RunSoong, "minibp")
                defer ctx.EndTrace()

                minibp := filepath.Join(config.SoongOutDir(), ".minibootstrap/minibp")
                if _, err := microfactory.Build(&cfg, minibp, "github.com/google/blueprint/bootstrap/minibp"); err != nil {
                        ctx.Fatalln("Failed to build minibp:", err)
                }
        }()

        func() {
// 编译bpglob程序，⽤于后⾯解析Android.bp⾥定义的 glob(⽂件路径名模式匹配) 规则，查找与之匹配的⽂件
// 例如 miui/frameworks/base/Android.bp⾥定义的`core/java/com/miui/internal/**/*.java`就
// 匹配 miui/frameworks/base/core/java/com/miui/internal⽬录下和其⼦⽬录（⽆限递归！）下所有的java⽂件
                ctx.BeginTrace(metrics.RunSoong, "bpglob")
                defer ctx.EndTrace()

                bpglob := filepath.Join(config.SoongOutDir(), ".minibootstrap/bpglob")
                if _, err := microfactory.Build(&cfg, bpglob, "github.com/google/blueprint/bootstrap/bpglob"); err != nil {
                        ctx.Fatalln("Failed to build bpglob:", err)
                }
        }()

        ninja := func(name, file string) {
                ctx.BeginTrace(metrics.RunSoong, name)
                defer ctx.EndTrace()

                fifo := filepath.Join(config.OutDir(), ".ninja_fifo")
                nr := status.NewNinjaReader(ctx, ctx.Status.StartTool(), fifo)
                defer nr.Close()

                cmd := Command(ctx, config, "soong "+name,
                        config.PrebuiltBuildTool("ninja"),
                        "-d", "keepdepfile",
                        "-w", "dupbuild=err",
                        "-j", strconv.Itoa(config.Parallel()),
                        "--frontend_file", fifo,
                        "-f", filepath.Join(config.SoongOutDir(), file))
                cmd.Environment.Set("SOONG_SANDBOX_SOONG_BUILD", "true")
                cmd.Sandbox = soongSandbox
                //启动ninja程序
                cmd.RunAndStreamOrFatal()
        }
}
// 启动ninja⽣成out/soong/.minbootstrap/build.ninja⾥定义的默认⽬标，实际就是执⾏minibp程序，
// 解析aosp源码⾥所有的Android.bp，从中得到后⾯⽤于处理不同类型构建类型的插件模块的构建⽅式(后⾯会再提到)，
// 并将它们汇总输出到out/soong/.bootstrap/build.ninja
        ninja("minibootstrap", ".minibootstrap/build.ninja")
// 启动ninja⽣成out/soong/.bootstrap/build.ninja⾥定义的默认⽬标，也就是out/soong/build.ninja
        ninja("bootstrap", ".bootstrap/build.ninja")
}

2.4.2.2 Android T版本

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
func runSoong(ctx Context, config Config) {
        ctx.BeginTrace(metrics.RunSoong, "soong")
        defer ctx.EndTrace()

        // We have two environment files: .available is the one with every variable,// .used with the ones that were actually used. The latter is used to// determine whether Soong needs to be re-run since why re-run it if only// unused variables were changed?
        envFile := filepath.Join(config.SoongOutDir(), availableEnvFile)

        buildMode := config.bazelBuildMode()
        integratedBp2Build := buildMode == mixedBuild// This is done unconditionally, but does not take a measurable amount of time
        bootstrapBlueprint(ctx, config)

        soongBuildEnv := config.Environment().Copy()
        soongBuildEnv.Set("TOP", os.Getenv("TOP"))
        // For Bazel mixed builds.
        soongBuildEnv.Set("BAZEL_PATH", "./tools/bazel")
        soongBuildEnv.Set("BAZEL_HOME", filepath.Join(config.BazelOutDir(), "bazelhome"))
        soongBuildEnv.Set("BAZEL_OUTPUT_BASE", filepath.Join(config.BazelOutDir(), "output"))
        soongBuildEnv.Set("BAZEL_WORKSPACE", absPath(ctx, "."))
        soongBuildEnv.Set("BAZEL_METRICS_DIR", config.BazelMetricsDir())
        soongBuildEnv.Set("LOG_DIR", config.LogsDir())

        // For Soong bootstrapping testsif os.Getenv("ALLOW_MISSING_DEPENDENCIES") == "true" {
                soongBuildEnv.Set("ALLOW_MISSING_DEPENDENCIES", "true")
        }

        err := writeEnvironmentFile(ctx, envFile, soongBuildEnv.AsMap())
        if err != nil {
                ctx.Fatalf("failed to write environment file %s: %s", envFile, err)
        }

        func() {
                ctx.BeginTrace(metrics.RunSoong, "environment check")
                defer ctx.EndTrace()

                checkEnvironmentFile(soongBuildEnv, config.UsedEnvFile(soongBuildTag))

                if integratedBp2Build || config.Bp2Build() {
                        checkEnvironmentFile(soongBuildEnv, config.UsedEnvFile(bp2buildTag))
                }

                if config.JsonModuleGraph() {
                        checkEnvironmentFile(soongBuildEnv, config.UsedEnvFile(jsonModuleGraphTag))
                }

                if config.Queryview() {
                        checkEnvironmentFile(soongBuildEnv, config.UsedEnvFile(queryviewTag))
                }

                if config.SoongDocs() {
                        checkEnvironmentFile(soongBuildEnv, config.UsedEnvFile(soongDocsTag))
                }
        }()

        runMicrofactory(ctx, config, "bpglob", "github.com/google/blueprint/bootstrap/bpglob",
                map[string]string{"github.com/google/blueprint": "build/blueprint"})

        ninja := func(name, ninjaFile string, targets ...string) {
                ctx.BeginTrace(metrics.RunSoong, name)
                defer ctx.EndTrace()

                fifo := filepath.Join(config.OutDir(), ".ninja_fifo")
                nr := status.NewNinjaReader(ctx, ctx.Status.StartTool(), fifo)
                defer nr.Close()

                ninjaArgs := []string{
                        "-d", "keepdepfile",
                        "-d", "stats",
                        "-o", "usesphonyoutputs=yes",
                        "-o", "preremoveoutputs=yes",
                        "-w", "dupbuild=err",
                        "-w", "outputdir=warn",
                        "-w", "missingoutfile=warn",
                        "-j", strconv.Itoa(config.Parallel()),
                        "--frontend_file", fifo,
                        "-f", filepath.Join(config.SoongOutDir(), ninjaFile),
                }

                ninjaArgs = append(ninjaArgs, targets...)
                cmd := Command(ctx, config, "soong "+name,
                        config.PrebuiltBuildTool("ninja"), ninjaArgs...)

                var ninjaEnv Environment// This is currently how the command line to invoke soong_build finds the// root of the source tree and the output root
                ninjaEnv.Set("TOP", os.Getenv("TOP"))

                qcEnvVars := []string{
                        "TARGET_BOARD_PLATFORM",
                        "SDCLANG_AE_CONFIG",
                        "SDCLANG_CONFIG",
                        "SDCLANG_SA_ENABLED",
                        "QIIFA_BUILD_CONFIG",
                }
                for _, qcVar := range qcEnvVars {
                        ninjaEnv.Set(qcVar, os.Getenv(qcVar))
                }

                cmd.Environment = &ninjaEnv
                cmd.Sandbox = soongSandbox
                cmd.RunAndStreamOrFatal()
        }

        targets := make([]string, 0, 0)

        if config.JsonModuleGraph() {
                targets = append(targets, config.ModuleGraphFile())
        }

        if config.Bp2Build() {
                targets = append(targets, config.Bp2BuildMarkerFile())
        }

        if config.Queryview() {
                targets = append(targets, config.QueryviewMarkerFile())
        }

        if config.SoongDocs() {
                targets = append(targets, config.SoongDocsHtml())
        }

        if config.SoongBuildInvocationNeeded() {
                // This build generates <builddir>/build.ninja, which is used later by build/soong/ui/build/build.go#Build().
                targets = append(targets, config.SoongNinjaFile())
        }

        ninja("bootstrap", "bootstrap.ninja", targets...)

        var soongBuildMetrics *soong_metrics_proto.SoongBuildMetricsif shouldCollectBuildSoongMetrics(config) {
                soongBuildMetrics := loadSoongBuildMetrics(ctx, config)
                logSoongBuildMetrics(ctx, soongBuildMetrics)
        }

        distGzipFile(ctx, config, config.SoongNinjaFile(), "soong")

        if !config.SkipKati() {
                distGzipFile(ctx, config, config.SoongAndroidMk(), "soong")
                distGzipFile(ctx, config, config.SoongMakeVarsMk(), "soong")
        }

        if shouldCollectBuildSoongMetrics(config) && ctx.Metrics != nil {
                ctx.Metrics.SetSoongBuildMetrics(soongBuildMetrics)
        }
        if config.JsonModuleGraph() {
                distGzipFile(ctx, config, config.ModuleGraphFile(), "soong")
        }
}

2.4.3 runKatiBuild

主要功能：

  1. 加载 build/make/core/main.mk
  2. 所有的Android.mk文件生成ninja文件:out/build-aosp_arm.ninja
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
func runKatiBuild(ctx Context, config Config) {
        ctx.BeginTrace(metrics.RunKati, "kati build")
        defer ctx.EndTrace()

        args := []string{
                // Mark the output directory as writable.
                "--writable", config.OutDir() + "/",
                // Fail when encountering implicit rules. e.g.
                // %.foo: %.bar
                //   cp $< $@
                "--werror_implicit_rules",
                // Entry point for the Kati Ninja file generation.
                "-f", "build/make/core/main.mk", //⼊口mk⽂件，kati会从它开始解析
        }

        if !config.BuildBrokenDupRules() {
                // Fail when redefining / duplicating a target.
                args = append(args, "--werror_overriding_commands")
        }

        args = append(args, config.KatiArgs()...)
//导出下⾯4个环境变量给kati进程，这些变量在build/make/core/main.mk以及被他依赖的mk⽂件会被使⽤到
// 例如SOONG_ANDROID_MK会在build/make/core/main.mk⾥被include，
// build/make/core/main.mk也会⾃动include前⾯FindSources()步骤⾥⽣成的out/.module_paths/Android.
        args = append(args,
                // Location of the Make vars .mk file generated by Soong.
                "SOONG_MAKEVARS_MK="+config.SoongMakeVarsMk(),
                // Location of the Android.mk file generated by Soong. This
                // file contains Soong modules represented as Kati modules,
                // allowing Kati modules to depend on Soong modules.
                "SOONG_ANDROID_MK="+config.SoongAndroidMk(),
                // Directory containing outputs for the target device.
                "TARGET_DEVICE_DIR="+config.TargetDeviceDir(),
                // Directory containing .mk files for packaging purposes, such as
                // the dist.mk file, containing dist-for-goals data.
                "KATI_PACKAGE_MK_DIR="+config.KatiPackageMkDir())

        runKati(ctx, config, katiBuildSuffix, args, func(env *Environment) {})

        // compress and dist the main build ninja file.
        distGzipFile(ctx, config, config.KatiBuildNinjaFile())

        // Cleanup steps.
        cleanCopyHeaders(ctx, config)
        cleanOldInstalledFiles(ctx, config)
}

runKati中会启动Kati程序，开始解析main.mk

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
$(info [1/1] initializing build system ...)
#mk⾥定义的第⼀个⽬标，即默认⽬标
.PHONY: droid
DEFAULT_GOAL := droid
$(DEFAULT_GOAL): droid_targets
.PHONY: droid_targets
droid_targets:
# Set up various standard variables based on configuration
# and host information.
include build/make/core/config.mk
include $(SOONG_MAKEVARS_MK)
...
# SOONG_ANDROID_MK即out/soong/Android-{product}.mk

subdir_makefiles := $(SOONG_ANDROID_MK) $(file
<$(OUT_DIR)/.module_paths/Android.mk.list)
$(SOONG_OUT_DIR)/late-$(TARGET_PRODUCT).mk
# 统计“int $(subdir_makefiles) post finish”⾥`单词`的个数
subdir_makefiles_total := $(words int $(subdir_makefiles) post finish)
# 将out/soong/Android-{product}.mk和out/.module_paths/Android.mk.list⾥所有的Android.mk都包含进来
$(foreach mk,$(subdir_makefiles),$(info [$(call
inc_and_print,subdir_makefiles_inc)/$(subdir_makefiles_total)] including $(mk)
...)$(eval include $(mk)))
...
$(info [$(call inc_and_print,subdir_makefiles_inc)/$(subdir_makefiles_total)]
writing build rules ...)

最终是调用系统准备好的prebuilts/build-tools/linux-x86/bin/ckati参与编译，其中传入的参数有 –ninja\ –regen\–detect_android_echo 等，kati解析完build/make/core/main.mk后，就会⽣成相应的ninja编译规则，runKatiBuild()函数最后⽣成out/build-{product}.ninja⽂件

2.4.4 runKatiPackage

也是运行runKati，只不过携带的参数不同，runKatiPackage函数最后会生成out/build-{product}-package.ninja

2.4.5 createCombinedBuildNinjaFile

会将kati和soong_build两个程序⽣成的ninja⽂件都包含进来，再⽣成⼀个out/combined-${product}.ninja⽂件

2.4.6 runNinja

启动ninja程序解析out/combined-${product}.ninja⽂件，得到我们执⾏的命令⾥指定target-files-package模块的依赖⽂件,最后只要按照拓扑顺序挨个执⾏他们的rule去⽣成他们就可以了