This page describes how to build and set up CRI-ContainerD with Kubernetes on Windows. This work is tracked for Kubernetes 1.15 as enhancement#1001. For more background on how this will be used and tested, please review the kep.
| Date | Description |
|---|---|
| 2019-05-07 | Split out from README.md |
SaswatB Set up a working environment for testing Kubernetes. It's in the Microsoft SDN repo and is used for the Windows CNI dev/test environments. You can get those scripts here https://github.com/Microsoft/SDN/tree/master/Kubernetes/containerd . I'm aiming to get this better consolidated to clarify how to build and set things up if your setup doesn't match what's prescribed in those scripts.
Getting all the binaries needed will require building from multiple repos. Here's the full list of what's required.
- For the CRI-ContainerD daemon:
- containerd.exe (built from jterry75/cri)
- containerd-shim-runhcs-v1.exe (built from Microsoft/hcsshim)
- Containerd & CRI clients:
- CNI plugin and meta-plugin
- nat.exe - for standalone networking (not Kubernetes) - source:Microsoft/windows-container-networking
- FlannelD.exe & flannel.exe
- SDNBridge.exe - if using
host-gwmode - source:Microsoft/windows-container-networking - SDNOverlay.exe - if using
overlaymode - source:Microsoft/windows-container-networking
- SDNBridge.exe - if using
There's a convenience script you can run from a Linux or Mac with Docker installed. It will create two zip files with everything you need.
Getting the source
go get github.com/Microsoft/hcsshimBuilding it
GOOS=windows go build github.com/Microsoft/hcsshim/cmd/containerd-shim-runhcs-v1There isn't a released version ready for k8s yet, but the master branch of containerd does support CRI for Windows containers. Here's how to build it:
cd $GOPATH
mkdir -p src/github.com/containerd
cd src/github.com/containerd
git clone https://github.com/containerd/containerd.git
cd containerd
GOOS=windows makeThis will produce bin/containerd.exe and ctr.exe
Clone https://github.com/Microsoft/windows-container-networking on your Linux dev/build machine, then run:
make dev
# in the container
make all
exitThat will produce nat.exe, sdnbridge.exe, and sdnoverlay.exe which are needed later.
Don't do this yet. Right now binaries need to be built from jterry75/cri to include CRI support. Skip to the next section
user@machine:/> cd $GOPATH/src/github.com/containerd/containerd
user@machine:/> export GOOS=windows
user@machine:/> make
+ bin/ctr.exe
+ bin/containerd.exe
+ bin/containerd-stress.exe
+ bin/containerd-release.exe
+ bin/containerd-shim-runhcs-v1.exe
+ binaries
This is optional, only if you're testing changes to hcsshim. This won't be needed once
containerd-shim-runhcs-v1.exeis built directly from the hcsshim repo.
user@machine:/> go get -u github.com/lk4d4/vndr
user@machine:/> vndr github.com/Microsoft/hcsshim <new-git-commit>
If you intend to include a vendored change in a PR to containerd, be sure to update vendor.conf too.
TODO - Testing Windows Server 2019 with ContainerD. VM work started here: https://github.com/patricklang/packer-windows/tree/containerd
Configuration Steps
- Register ContainerD as service
- Enable CRI listener for ContainerD.exe
- Getting kubelet configured to use CRI endpoint instead of dockershim
If you don't already have a config file the daemon can generate one for you:
C:\> containerd.exe config default > config.toml
(Common areas of the config to change) root - The root where all daemon data is kept state - The state directory where all plugin data is kept snapshots, images, container bundles, etc. grpc address - The address the containerd daemon will serve. Default is: \.\pipe\containerd-containerd debug level - Set to debug for all daemon debugging
TODO: Missing CNI step
ctr.exe images pull mcr.microsoft.com/windows/nanoserver:1809
ctr.exe run --rm mcr.microsoft.com/windows/nanoserver:1809 argon-test cmd /c "echo Hello World!"
Example Output:
PS C:\containerd> ./ctr.exe images pull mcr.microsoft.com/windows/nanoserver:1809
mcr.microsoft.com/windows/nanoserver:1809: resolved |++++++++++++++++++++++++++++++++++++++|
index-sha256:75bae46687f544f139ec57e1925d184fbb2ed70f6e0e5c993a55bd4f8e8e17a8: exists |++++++++++++++++++++++++++++++++++++++|
manifest-sha256:6603a3e57f2d127fbddbc7b0aa3807b02b3c25163a7c6404da1d107ce33549c4: exists |++++++++++++++++++++++++++++++++++++++|
manifest-sha256:5953d8407d58ddc66e3eb426e320e93786a3cb173957cc5af79d46f731f3301c: exists |++++++++++++++++++++++++++++++++++++++|
config-sha256:3601d6edd492515e2f9b352db93b0d67af0d49f1178561b5a5d50e1232c0276a: done |++++++++++++++++++++++++++++++++++++++|
layer-sha256:1046f7eb9dcd29d2478f707dca8726d2ae066a276196e327bd386d50f6448b2a: done |++++++++++++++++++++++++++++++++++++++|
config-sha256:4702b277b15f4ce1a1a3f26092229e7b79f8f6e11450d9d171bcf7367ab96350: done |++++++++++++++++++++++++++++++++++++++|
elapsed: 0.5 s total: 0.0 B (0.0 B/s)
unpacking windows/amd64 sha256:75bae46687f544f139ec57e1925d184fbb2ed70f6e0e5c993a55bd4f8e8e17a8...
done
PS C:\containerd> ./ctr.exe run --rm mcr.microsoft.com/windows/nanoserver:1809 argon-test cmd /c "echo Hello World!"
Hello World!
TODO: this doesn't work yet, section incomplete. ContainerD fails to start a container without a CNI configured.
First, you need a sandbox/pod configuration. Copy this into a file pod-sandbox-default.json. It will create a process-isolated Windows pod.
{
"metadata": {
"name": "sandbox",
"namespace": "default",
"attempt": 1
}
}./crictl -r npipe:\\\\.\pipe\containerd-containerd pull mcr.microsoft.com/windows/nanoserver:1809
Image is up to date for sha256:4702b277b15f4ce1a1a3f26092229e7b79f8f6e11450d9d171bcf7367ab96350
Create the sandbox with: .\crictl -r npipe:\\\\.\pipe\containerd-containerd runp .\pod-sandbox-default.json
Create a container config, copying this file into container-config-windows-hello-world.json
.\crictl.exe create <POD-ID> .\container-config-windows-hello-world.json .\pod-sandbox-default.json
crictl -r npipe:\\\\.\pipe\containerd-containerd pods
crictl -r npipe:\\\\.\pipe\containerd-containerd inspectp 1c0e277aba1e1