开发者指南
开发者指南
克隆仓库
克隆 Spark Operator 仓库并切换到该目录
git clone git@github.com:kubeflow/spark-operator.git
cd spark-operator
(可选) 配置 Git 预提交钩子
Git 钩子对于在提交代码评审之前识别简单问题非常有用。我们在每次提交时运行钩子，以从 README.md.gotmpl 文件自动生成 helm chart 的 README.md 文件。在运行 git 钩子之前，您需要按如下方式安装 pre-commit 包管理器：
# Using pip
pip install pre-commit
# Using conda
conda install -c conda-forge pre-commit
# Using Homebrew
brew install pre-commit
要设置 pre-commit 钩子，请运行以下命令：
pre-commit install
pre-commit install-hooks
使用 Makefile
我们使用 Makefile 自动化常见任务。例如，要构建 Operator，请运行 build-operator 目标，如下所示，spark-operator 二进制文件将被构建并放置在 bin 目录中：
make build-operator
依赖项会根据需要在本地自动下载到 bin 目录。例如，如果您运行 make manifests 目标，controller-gen 工具将使用 go install 命令自动下载，然后会重命名为 controller-gen-vX.Y.Z 并放置在 bin 目录中。
要查看所有可用目标的完整列表，请运行以下命令：
$ make help
Usage:
make <target>
General
help Display this help.
version Print version information.
Development
manifests Generate CustomResourceDefinition, RBAC and WebhookConfiguration manifests.
generate Generate code containing DeepCopy, DeepCopyInto, and DeepCopyObject method implementations.
update-crd Update CRD files in the Helm chart.
go-clean Clean up caches and output.
go-fmt Run go fmt against code.
go-vet Run go vet against code.
lint Run golangci-lint linter.
lint-fix Run golangci-lint linter and perform fixes.
unit-test Run unit tests.
e2e-test Run the e2e tests against a Kind k8s instance that is spun up.
Build
build-operator Build Spark operator.
build-sparkctl Build sparkctl binary.
install-sparkctl Install sparkctl binary.
clean Clean spark-operator and sparkctl binaries.
build-api-docs Build api documentation.
docker-build Build docker image with the operator.
docker-push Push docker image with the operator.
docker-buildx Build and push docker image for the operator for cross-platform support
Helm
detect-crds-drift Detect CRD drift.
helm-unittest Run Helm chart unittests.
helm-lint Run Helm chart lint test.
helm-docs Generates markdown documentation for helm charts from requirements and values files.
Deployment
kind-create-cluster Create a kind cluster for integration tests.
kind-load-image Load the image into the kind cluster.
kind-delete-custer Delete the created kind cluster.
install-crd Install CRDs into the K8s cluster specified in ~/.kube/config.
uninstall-crd Uninstall CRDs from the K8s cluster specified in ~/.kube/config. Call with ignore-not-found=true to ignore resource not found errors during deletion.
deploy Deploy controller to the K8s cluster specified in ~/.kube/config.
undeploy Undeploy controller from the K8s cluster specified in ~/.kube/config. Call with ignore-not-found=true to ignore resource not found errors during deletion.
Dependencies
kustomize Download kustomize locally if necessary.
controller-gen Download controller-gen locally if necessary.
kind Download kind locally if necessary.
envtest Download setup-envtest locally if necessary.
golangci-lint Download golangci-lint locally if necessary.
gen-crd-api-reference-docs Download gen-crd-api-reference-docs locally if necessary.
helm Download helm locally if necessary.
helm-unittest-plugin Download helm unittest plugin locally if necessary.
helm-docs-plugin Download helm-docs plugin locally if necessary.
使用 Spark Operator 进行开发
构建二进制文件
要构建 Operator，请运行以下命令：
make build-operator
构建 Docker 镜像
如果您想从源代码构建 Operator，例如测试您编写的修复程序或功能，可以按照以下说明进行操作。
构建 Operator 而无需担心其依赖项的最简单方法是使用 Dockerfile 构建镜像。
make docker-build IMAGE_TAG=<image-tag>
Operator 镜像基于一个默认的 Spark 基础镜像 spark:3.5.2 构建。如果您想使用自己的 Spark 镜像（例如，具有不同 Spark 版本或某些自定义依赖项的镜像），请指定参数 SPARK_IMAGE，如下例所示：
docker build --build-arg SPARK_IMAGE=<your Spark image> -t <image-tag> .
更新 API 定义
如果您更新了 API 定义，则还需要更新自动生成的代码。要更新包含 DeepCopy、DeepCopyInto 和 DeepCopyObject 方法实现的自动生成代码，请运行以下命令：
make generate
要更新自动生成的 CustomResourceDefinition (CRD)、RBAC 和 WebhookConfiguration 清单，请运行以下命令：
make manifests
更新 CRD 文件后，运行以下命令将 CRD 文件复制到 helm chart 目录：
make update-crd
此外，API 规范文档 docs/api-docs.md 也需要更新。要更新文档，请运行以下命令：
make build-api-docs
运行单元测试
要运行单元测试，请运行以下命令：
make unit-test
运行端到端测试
要运行端到端测试，请运行以下命令：
# Create a kind cluster
make kind-create-cluster
# Build docker image
make docker-build IMAGE_TAG=local
# Load docker image to kind cluster
make kind-load-image
# Run e2e tests
make e2e-test
# Delete the kind cluster
make kind-delete-cluster
使用 Helm Chart 进行开发
运行 Helm Chart Lint 测试
要运行 Helm chart Lint 测试，请运行以下命令：
$ make helm-lint
Linting charts...
------------------------------------------------------------------------------------------------------------------------
Charts to be processed:
------------------------------------------------------------------------------------------------------------------------
spark-operator => (version: "1.2.4", path: "charts/spark-operator-chart")
------------------------------------------------------------------------------------------------------------------------
Linting chart "spark-operator => (version: \"1.2.4\", path: \"charts/spark-operator-chart\")"
Checking chart "spark-operator => (version: \"1.2.4\", path: \"charts/spark-operator-chart\")" for a version bump...
Old chart version: 1.2.1
New chart version: 1.2.4
Chart version ok.
Validating /Users/user/go/src/github.com/kubeflow/spark-operator/charts/spark-operator-chart/Chart.yaml...
Validation success! 👍
Validating maintainers...
Linting chart with values file "charts/spark-operator-chart/ci/ci-values.yaml"...
==> Linting charts/spark-operator-chart
[INFO] Chart.yaml: icon is recommended
1 chart(s) linted, 0 chart(s) failed
------------------------------------------------------------------------------------------------------------------------
✔︎ spark-operator => (version: "1.2.4", path: "charts/spark-operator-chart")
------------------------------------------------------------------------------------------------------------------------
All charts linted successfully
运行 Helm chart 单元测试
有关如何编写 Helm chart 单元测试的详细信息，请参阅 helm-unittest。要运行 Helm chart 单元测试，请运行以下命令：
$ make helm-unittest
### Chart [ spark-operator ] charts/spark-operator-chart
PASS Test controller deployment charts/spark-operator-chart/tests/controller/deployment_test.yaml
PASS Test controller pod disruption budget charts/spark-operator-chart/tests/controller/poddisruptionbudget_test.yaml
PASS Test controller rbac charts/spark-operator-chart/tests/controller/rbac_test.yaml
PASS Test controller deployment charts/spark-operator-chart/tests/controller/service_test.yaml
PASS Test controller service account charts/spark-operator-chart/tests/controller/serviceaccount_test.yaml
PASS Test prometheus pod monitor charts/spark-operator-chart/tests/prometheus/podmonitor_test.yaml
PASS Test Spark RBAC charts/spark-operator-chart/tests/spark/rbac_test.yaml
PASS Test spark service account charts/spark-operator-chart/tests/spark/serviceaccount_test.yaml
PASS Test webhook deployment charts/spark-operator-chart/tests/webhook/deployment_test.yaml
PASS Test mutating webhook configuration charts/spark-operator-chart/tests/webhook/mutatingwebhookconfiguration_test.yaml
PASS Test webhook pod disruption budget charts/spark-operator-chart/tests/webhook/poddisruptionbudget_test.yaml
PASS Test webhook rbac charts/spark-operator-chart/tests/webhook/rbac_test.yaml
PASS Test webhook service charts/spark-operator-chart/tests/webhook/service_test.yaml
PASS Test validating webhook configuration charts/spark-operator-chart/tests/webhook/validatingwebhookconfiguration_test.yaml
Charts: 1 passed, 1 total
Test Suites: 14 passed, 14 total
Tests: 137 passed, 137 total
Snapshot: 0 passed, 0 total
Time: 477.748ms
构建 Helm 文档
Helm chart 的 README.md 文件是由 helm-docs 工具生成的。如果您想更新 Helm 文档，请记住修改 README.md.gotmpl 而不是 README.md，然后运行 make helm-docs 生成 README.md 文件。
$ make helm-docs
INFO[2024-04-14T07:29:26Z] Found Chart directories [charts/spark-operator-chart]
INFO[2024-04-14T07:29:26Z] Generating README Documentation for chart charts/spark-operator-chart
请注意，如果设置了 git pre-commit 钩子，helm-docs 会在提交任何更改之前自动运行。如果 README.md 文件有任何更改，提交过程将中止。
签署你的提交
在您对代码进行了更改后，请使用 -s 或 --signoff 标志签署您的提交，以便 DCO 检查 CI 通过。
git commit -s -m "Your commit message"
最后修改于 2024年9月28日: 更新 spark-operator 开发者指南 (#3882) (026d8ec)