Development Guide

• 1: Deploy with Customized Images
• 2: Dependency Management
• 3: E2E tests
• 3.1: Azure E2E tests
• 3.2: Kubernetes E2E tests
• 4: Deploy clusters
• 5: Design Docs and KEPs
• 5.1: Azure Private Link Service Integration
• 6: Image building
• 7: Future Plans

1 - Deploy with Customized Images

Switch to the project root directory and run the following command to build both CCM and CNM images:

make image

If you want to build only one of them, try make build-ccm-image or ARCH=amd64 make build-node-image-linux.

To push the images to your own image registry, you can specify the registry and image tag while building:

IMAGE_REGISTRY=<image registry name> IMAGE_TAG=<tag name> make image

After building, you can push them to your image registry by make push.

Please follow here to build multi-arch image

2 - Dependency Management

cloud-provider-azure uses go modules for Go dependency management.

Usage

Run make update-dependencies whenever vendored dependencies change. This takes a minute to complete.

Run make update-mocks whenever implementations for pkg/azureclients change.

Updating dependencies

New dependencies causes golang to recompute the minor version used for each major version of each dependency. And golang automatically removes dependencies that nothing imports any more.

To upgrade to the latest version for all direct and indirect dependencies of the current module:

• run go get -u <package> to use the latest minor or patch releases
• run go get -u=patch <package> to use the latest patch releases
• run go get <package>@VERSION to use the specified version

You can also manually editing go.mod and update the versions in require and replace parts.

Because of staging in Kubernetes, manually go.mod updating is required for Kubernetes and its staging packages. In cloud-provider-azure, their versions are set in replace part, e.g.

replace (
    ...
    k8s.io/kubernetes => k8s.io/kubernetes v0.0.0-20190815230911-4e7fd98763aa
)

To update their versions, you need switch to $GOPATH/src/k8s.io/kubernetes, checkout to the version you want upgrade to, and finally run the following commands to get the go modules expected version:

commit=$(TZ=UTC git --no-pager show --quiet --abbrev=12 --date='format-local:%Y%m%d%H%M%S' --format="%cd-%h")
echo "v0.0.0-$commit"

After this, replace all kubernetes and staging versions (e.g. v0.0.0-20190815230911-4e7fd98763aa in above example) in go.mod.

Always run hack/update-dependencies.sh after changing go.mod by any of these methods (or adding new imports).

See golang’s go.mod, Using Go Modules and Kubernetes Go modules docs for more details.

Updating mocks

mockgen v1.6.0 is used to generate mocks.

mockgen -copyright_file=<copyright file> -source=<azureclient source> -package=<mock package>

3 - E2E tests

3.1 - Azure E2E tests

Overview

Here provides some E2E tests only specific to Azure provider.

Prerequisite

Deploy a Kubernetes cluster with Azure CCM

Refer step 1-3 in e2e-tests for deploying the Kubernetes cluster.

Setup Azure credentials

export AZURE_TENANT_ID=<tenant-id>                    # the tenant ID
export AZURE_SUBSCRIPTION_ID=<subscription-id>        # the subscription ID
export AZURE_CLIENT_ID=<service-principal-id>         # the service principal ID
export AZURE_CLIENT_SECRET=<service-principal-secret> # the service principal secret
export AZURE_ENVIRONMENT=<AzurePublicCloud>           # the cloud environment (optional, default is AzurePublicCloud)
export AZURE_LOCATION=<location>                      # the location
export AZURE_LOADBALANCER_SKU=<loadbalancer-sku>      # the sku of load balancer (optional, default is basic)

Setup KUBECONFIG

• Locate your kubeconfig and set it as env variable export KUBECONFIG=<kubeconfig> or cp <kubeconfig> ~/.kube/config

• Test it via kubectl version

Run tests

• Run default tests

  The following command ensures gingko v2 is installed and then runs default tests.

  make test-ccm-e2e

• Run specific tests

  go test -v ./tests/e2e/ -timeout 0 -ginkgo.focus <focus-keyword> --ginkgo.skip <skip-keyword>

After a long time test, a JUnit report will be generated in a directory named by the cluster name

3.2 - Kubernetes E2E tests

Prerequisite

• An azure service principal

  Please follow this guide for creating an azure service principal The service principal should either have:

  • Contributor permission of a subscription
  • Contributor permission of a resource group. In this case, please create the resource group first

• Docker daemon enabled

How to run Kubernetes e2e tests locally

1. Prepare dependency project

• kubectl

  Kubectl allows you to run command against Kubernetes cluster, which is also used for deploying CSI plugins. You can follow here to install kubectl. e.g. on Linux

  curl -LO https://dl.k8s.io/release/$(curl -sL https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl
  chmod +x kubectl
  sudo mv kubectl /usr/local/bin/
  

2. Build docker images azure-cloud-controller-manager, azure-cloud-node-manager and push them to your image repository.

   git clone https://github.com/kubernetes-sigs/cloud-provider-azure $GOPATH/src/sigs.k8s.io/cloud-provider-azure
   cd $GOPATH/src/sigs.k8s.io/cloud-provider-azure
   export IMAGE_REGISTRY=<your-registry>
   export IMAGE_TAG=<tag>
   make image # build all images of different ARCHs and OSes
   make push # push all images of different ARCHs and OSes to your registry. Or manually `docker push`
   

3. Deploy a Kubernetes cluster with the above azure-cloud-controller-manager and azure-cloud-node-manager images.

   To deploy a cluster, export all the required environmental variables first and then invoke make deploy-cluster. Please notice that cluster-api-provider-azure is used to provision the management and workload clusters. To learn more about this provisioner, you can refer to its quick-start doc.

   export AZURE_SUBSCRIPTION_ID=<subscription-id>
   export AZURE_TENANT_ID=<tenant-id>
   export AZURE_CLIENT_ID=<client-id>
   export AZURE_CLIENT_SECRET=<client-secret>
   export CLUSTER_NAME=<cluster-name>
   export AZURE_RESOURCE_GROUP=<resource-group>
   export AZURE_CLOUD_CONTROLLER_MANAGER_IMG=<cloud-controller-manager-image>
   export AZURE_CLOUD_NODE_MANAGER_IMG=<cloud-node-manager-image>
   
   make deploy-cluster
   

   To connect the cluster:

   export KUBECONFIG=$GOPATH/src/sigs.k8s.io/cloud-provider-azure/$CLUSTER_NAME-kubeconfig
   kubectl cluster-info
   

   To check out more of the deployed cluster , replace kubectl cluster-info with other kubectl commands. To further debug and diagnose cluster problems, use kubectl cluster-info dump

4. Run Kubernetes E2E tests

   make test-e2e-capz
   

4 - Deploy clusters

Cluster API Provider Azure (CAPZ)

A management cluster is needed to deploy workload clusters. Please follow the instructions.

After the management cluster is provisioned, please run the following command in the root directory of the repo.

make deploy-workload-cluster

Customizations are supported by environment variables:

To completely remove the cluster, specify the “${CLUSTER_NAME}” and run:

make delete-workload-cluster

5 - Design Docs and KEPs

This is the staging area of the design docs prior to or under development. Once the feature is done, the corresponding design doc would be moved to Topics.

5.1 - Azure Private Link Service Integration

This feature is now generally available. The page has been moved to (topics).

6 - Image building

multi-arch image

Currently, only Linux multi-arch cloud-node-manager image is supported as a result of customer requests and windows limitations. Supported Linux archs are defined by ALL_ARCH.linux in Makefile, and Windows os versions are by ALL_OSVERSIONS.windows.

Windows multi-arch image limitation

Images nanoserver and servercore are referenced to build a Windows image, but as current officially published servercore images does not support non-amd64 image, and only Windows server 1809 has the support of non-amd64 for nanoserver, amd64 is the only supported arch for a range of Windows OS version so far. This issue is tracked here

hand-on examples

To build and publish the multi-arch image for node manager

IMAGE_REGISTRY=<registry> make build-all-node-images
IMAGE_REGISTRY=<registry> make push-multi-arch-node-manager-image

To build a specific Linux arch image for node manager

IMAGE_REGISTRY=<registry> ARCH=amd64 make build-node-image-linux

To build specific Windows OS and arch image for node manager

IMAGE_REGISTRY=<registry> OUTPUT_TYPE=registry ARCH=amd64 WINDOWS_OSVERSION=1809 build-node-image-windows

The OUTPUT_TYPE registry here means the built image will be published to the registry, this is necessary to build a Windows image from a Linux working environment. An alternative is to export the image tarball to a local destination, like OUTPUT_TYPE=docker,dest=dstdir/azure-cloud-node-manager.tar. For more info about docker buildx output type, please check out here

7 - Future Plans

To be completed.