-
Notifications
You must be signed in to change notification settings - Fork 493
Support bootstrapping CNI configs from a template file #2126
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
twz123
wants to merge
9
commits into
cloudnativelabs:master
Choose a base branch
from
twz123:cni-config-template
base: master
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Open
Changes from all commits
Commits
Show all changes
9 commits
Select commit
Hold shift + click to select a range
1c8ce85
chore: Use temp dir facility from Go's test framework in the CNI conf…
twz123 640809b
chore: Replace tabs with spaces in test CNI configs
twz123 bffa4a3
chore: Introduce cniConfContent for CNI config tests
twz123 ecb6790
chore: Rollup the CNI config marshalling roundtrip test
twz123 cb11e1c
chore: Deprecate CNINetworkConfig's IsConfList
twz123 edd8852
chore: Use error strings in CNI config table tests
twz123 b0fe337
chore: Use getPodCIDRsMapFromCNISpec in CNI Config tests
twz123 67fa346
fix: Reject CNI conflists without a bridge plugin
twz123 af56fd3
feat: Support bootstrapping CNI configs from a template file
twz123 File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,217 @@ | ||
| apiVersion: v1 | ||
| kind: ConfigMap | ||
| metadata: | ||
| name: kube-router-cfg | ||
| namespace: kube-system | ||
| labels: | ||
| tier: node | ||
| k8s-app: kube-router | ||
| data: | ||
| cni-conf.json: | | ||
| { | ||
| "cniVersion": "0.3.0", | ||
| "name": "mynet", | ||
| "plugins": [ | ||
| { | ||
| "name": "kubernetes", | ||
| "type": "bridge", | ||
| "bridge": "kube-bridge", | ||
| "isDefaultGateway": true, | ||
| "ipam": { | ||
| "type": "host-local" | ||
| } | ||
| } | ||
| ] | ||
| } | ||
|
|
||
| --- | ||
| apiVersion: apps/v1 | ||
| kind: DaemonSet | ||
| metadata: | ||
| labels: | ||
| k8s-app: kube-router | ||
| tier: node | ||
| name: kube-router | ||
| namespace: kube-system | ||
| spec: | ||
| selector: | ||
| matchLabels: | ||
| k8s-app: kube-router | ||
| tier: node | ||
| template: | ||
| metadata: | ||
| labels: | ||
| k8s-app: kube-router | ||
| tier: node | ||
| spec: | ||
| priorityClassName: system-node-critical | ||
| serviceAccountName: kube-router | ||
| initContainers: | ||
| - name: install-cni | ||
| image: docker.io/cloudnativelabs/kube-router | ||
| imagePullPolicy: Always | ||
| command: | ||
| - /bin/sh | ||
| - -ec | ||
| - | | ||
| [ -f /etc/cni/net.d/10-kuberouter.conflist ] || { | ||
| rm -f /etc/cni/net.d/*.conf | ||
| rm -f /etc/cni/net.d/*.conflist | ||
| } | ||
| /usr/local/bin/cni-install | ||
| volumeMounts: | ||
| - name: cni-conf-dir | ||
| mountPath: /etc/cni/net.d | ||
| - name: host-opt | ||
| mountPath: /opt | ||
| containers: | ||
| - name: kube-router | ||
| image: docker.io/cloudnativelabs/kube-router | ||
| imagePullPolicy: Always | ||
| args: | ||
| - --run-router=true | ||
| - --run-firewall=true | ||
| - --run-service-proxy=false | ||
| - --bgp-graceful-restart=true | ||
| env: | ||
| - name: NODE_NAME | ||
| valueFrom: | ||
| fieldRef: | ||
| fieldPath: spec.nodeName | ||
| - name: POD_NAME | ||
| valueFrom: | ||
| fieldRef: | ||
| fieldPath: metadata.name | ||
| - name: KUBE_ROUTER_CNI_CONF_FILE | ||
| value: /etc/cni/net.d/10-kuberouter.conflist | ||
| - name: KUBE_ROUTER_CNI_CONF_TEMPLATE_FILE | ||
| value: /etc/kube-router/cni-conf.json | ||
| livenessProbe: | ||
| httpGet: | ||
| path: /healthz | ||
| port: 20244 | ||
| initialDelaySeconds: 10 | ||
| periodSeconds: 3 | ||
| resources: | ||
| requests: | ||
| cpu: 250m | ||
| memory: 250Mi | ||
| securityContext: | ||
| privileged: true | ||
| volumeMounts: | ||
| - name: lib-modules | ||
| mountPath: /lib/modules | ||
| readOnly: true | ||
| - name: cni-conf-dir | ||
| mountPath: /etc/cni/net.d | ||
| - name: kube-router-cfg | ||
| mountPath: /etc/kube-router | ||
| readOnly: true | ||
| - name: xtables-lock | ||
| mountPath: /run/xtables.lock | ||
| readOnly: false | ||
| hostNetwork: true | ||
| hostPID: true | ||
| tolerations: | ||
| - effect: NoSchedule | ||
| operator: Exists | ||
| - key: CriticalAddonsOnly | ||
| operator: Exists | ||
| - effect: NoExecute | ||
| operator: Exists | ||
| volumes: | ||
| - name: lib-modules | ||
| hostPath: | ||
| path: /lib/modules | ||
| - name: cni-conf-dir | ||
| hostPath: | ||
| path: /etc/cni/net.d | ||
| - name: kube-router-cfg | ||
| configMap: | ||
| name: kube-router-cfg | ||
| - name: xtables-lock | ||
| hostPath: | ||
| path: /run/xtables.lock | ||
| type: FileOrCreate | ||
| - name: host-opt | ||
| hostPath: | ||
| path: /opt | ||
|
|
||
| --- | ||
| apiVersion: v1 | ||
| kind: ServiceAccount | ||
| metadata: | ||
| name: kube-router | ||
| namespace: kube-system | ||
|
|
||
| --- | ||
| kind: ClusterRole | ||
| apiVersion: rbac.authorization.k8s.io/v1 | ||
| metadata: | ||
| name: kube-router | ||
| namespace: kube-system | ||
| rules: | ||
| - apiGroups: | ||
| - "" | ||
| resources: | ||
| - namespaces | ||
| - pods | ||
| - services | ||
| - nodes | ||
| - endpoints | ||
| verbs: | ||
| - list | ||
| - get | ||
| - watch | ||
| - apiGroups: | ||
| - networking.k8s.io | ||
| resources: | ||
| - networkpolicies | ||
| verbs: | ||
| - list | ||
| - get | ||
| - watch | ||
| - apiGroups: | ||
| - extensions | ||
| resources: | ||
| - networkpolicies | ||
| verbs: | ||
| - get | ||
| - list | ||
| - watch | ||
| - apiGroups: | ||
| - coordination.k8s.io | ||
| resources: | ||
| - leases | ||
| verbs: | ||
| - get | ||
| - create | ||
| - update | ||
| - apiGroups: | ||
| - "" | ||
| resources: | ||
| - services/status | ||
| verbs: | ||
| - update | ||
| - apiGroups: | ||
| - discovery.k8s.io | ||
| resources: | ||
| - endpointslices | ||
| verbs: | ||
| - get | ||
| - list | ||
| - watch | ||
|
|
||
| --- | ||
| kind: ClusterRoleBinding | ||
| apiVersion: rbac.authorization.k8s.io/v1 | ||
| metadata: | ||
| name: kube-router | ||
| roleRef: | ||
| apiGroup: rbac.authorization.k8s.io | ||
| kind: ClusterRole | ||
| name: kube-router | ||
| subjects: | ||
| - kind: ServiceAccount | ||
| name: kube-router | ||
| namespace: kube-system |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -35,7 +35,9 @@ | |
| - [Hairpin Mode Example](#hairpin-mode-example) | ||
| - [SNATing Service Traffic](#snating-service-traffic) | ||
| - [Load balancing Scheduling Algorithms](#load-balancing-scheduling-algorithms) | ||
| - [HostPort support](#hostport-support) | ||
| - [CNI Configuration](#cni-configuration) | ||
| - [CNI Configuration Templates](#cni-configuration-templates) | ||
| - [HostPort support](#hostport-support) | ||
| - [IPVS Graceful termination support](#ipvs-graceful-termination-support) | ||
| - [MTU](#mtu) | ||
| - [BGP configuration](#bgp-configuration) | ||
|
|
@@ -578,46 +580,92 @@ $ kubectl annotate service my-service "kube-router.io/service.schedflags=flag-2" | |
| $ kubectl annotate service my-service "kube-router.io/service.schedflags=flag-1,flag-2" | ||
| ``` | ||
|
|
||
| ## HostPort support | ||
|
|
||
| If you would like to use `HostPort` functionality below changes are required in the manifest. | ||
|
|
||
| - By default kube-router assumes CNI conf file to be `/etc/cni/net.d/10-kuberouter.conf`. Add an environment variable | ||
| `KUBE_ROUTER_CNI_CONF_FILE` to kube-router manifest and set it to `/etc/cni/net.d/10-kuberouter.conflist` | ||
|
|
||
| - Modify `kube-router-cfg` ConfigMap with CNI config that supports `portmap` as additional plug-in | ||
|
|
||
| ```json | ||
| { | ||
| "cniVersion":"0.3.0", | ||
| "name":"mynet", | ||
| "plugins":[ | ||
| { | ||
| "name":"kubernetes", | ||
| "type":"bridge", | ||
| "bridge":"kube-bridge", | ||
| "isDefaultGateway":true, | ||
| "ipam":{ | ||
| "type":"host-local" | ||
| } | ||
| }, | ||
| { | ||
| "type":"portmap", | ||
| "capabilities":{ | ||
| "snat":true, | ||
| "portMappings":true | ||
| } | ||
| } | ||
| ] | ||
| } | ||
| ``` | ||
|
|
||
| - Update init container command to create `/etc/cni/net.d/10-kuberouter.conflist` file | ||
| ## CNI Configuration | ||
|
|
||
| kube-router provides pod networking by means of the standard [`bridge`][cni-bridge] and [`host-local`][cni-host-local] | ||
| [containernetworking plugins], optionally accompanied by others like [`portmap`][cni-portmap]. The kube-router Docker | ||
| image ships these plugins, and the [example daemonsets](../daemonset/) install them onto the node in an init container. | ||
| kube-router's own contribution is the routing of pod traffic between nodes and the management of the plugins' | ||
| configuration file, filling in the parts that are node-specific and only known at runtime. The latter is what | ||
| `--enable-cni` controls, which is enabled by default. Disable it to use kube-router's features alongside another CNI | ||
| provider. On startup, kube-router | ||
|
|
||
| - inserts the node's pod CIDRs into the `host-local` IPAM ranges of the `bridge` plugin, as allocated to the node by | ||
| the kube-controller-manager, or as set via the `kube-router.io/pod-cidrs` annotation | ||
| - and sets the `bridge` plugin's MTU, if `--auto-mtu` is enabled (see [MTU](#mtu)). | ||
|
Collaborator
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Nit: I would drop the starting "and" for this second point and drop the ending period since this is a bulleted list and not meant to be read like complete sentences. |
||
|
|
||
| The completed configuration is written back once the `kube-bridge` interface has been set up. The configuration is read | ||
| from and written to `/etc/cni/net.d/10-kuberouter.conf` by default; the location can be changed via the | ||
| `KUBE_ROUTER_CNI_CONF_FILE` environment variable. A file name ending in `.conflist` is treated as a [configuration | ||
| list][cni-config-format], i.e. a chain of multiple plugins, any other name as a single plugin configuration. | ||
|
|
||
| The configuration file must exist when kube-router starts. The [example daemonsets](../daemonset/) ship it in a | ||
| `kube-router-cfg` ConfigMap and use an init container to copy it into the CNI configuration directory. | ||
|
|
||
| ### CNI Configuration Templates | ||
|
|
||
| A configuration file that's copied into the CNI configuration directory by an init container is already visible to the | ||
| container runtime before kube-router has completed it, so on a freshly booted node, pod sandbox creation may transiently | ||
| fail with errors like "no IP ranges specified" until kube-router has filled in the missing parts. | ||
|
|
||
| To avoid this, the `KUBE_ROUTER_CNI_CONF_TEMPLATE_FILE` environment variable can point kube-router to a potentially | ||
| incomplete CNI configuration outside of the CNI configuration directory, e.g. a ConfigMap mounted at | ||
| `/etc/kube-router/cni-conf.json`. kube-router will then read the configuration from the template instead, and write the | ||
| completed configuration to `KUBE_ROUTER_CNI_CONF_FILE`, which doesn't need to exist beforehand. The template is | ||
| interpreted according to the file name given in `KUBE_ROUTER_CNI_CONF_FILE`, i.e. as a configuration list if that name | ||
| ends in `.conflist`. The template's own file name has no significance. As a result, the container runtime won't pick up | ||
| an incomplete configuration and try to set up pod networking too early. Since the configuration is re-derived from the | ||
| pristine template on every start, the written configuration is wholly owned by kube-router. | ||
|
|
||
| For an example manifest, please look at the [CNI configuration template manifest]. | ||
|
|
||
| ### HostPort support | ||
|
|
||
| If you would like to use `HostPort` functionality, the following changes are required in the manifest: | ||
|
|
||
| - Since the `portmap` plugin needs to be chained after the `bridge` plugin, the CNI configuration has to be a config | ||
| list: set `KUBE_ROUTER_CNI_CONF_FILE` to `/etc/cni/net.d/10-kuberouter.conflist` (see above) | ||
| - Modify the `kube-router-cfg` ConfigMap with a CNI configuration that supports `portmap` as an additional plug-in: | ||
|
|
||
| ```json | ||
| { | ||
| "cniVersion":"0.3.0", | ||
| "name":"mynet", | ||
| "plugins":[ | ||
| { | ||
| "name":"kubernetes", | ||
| "type":"bridge", | ||
| "bridge":"kube-bridge", | ||
| "isDefaultGateway":true, | ||
| "ipam":{ | ||
| "type":"host-local" | ||
| } | ||
| }, | ||
| { | ||
| "type":"portmap", | ||
| "capabilities":{ | ||
| "snat":true, | ||
| "portMappings":true | ||
| } | ||
| } | ||
| ] | ||
| } | ||
| ``` | ||
|
|
||
| - Update the init container command to create `/etc/cni/net.d/10-kuberouter.conflist`, or point | ||
| `KUBE_ROUTER_CNI_CONF_TEMPLATE_FILE` at the ConfigMap mount instead | ||
| - Restart the container runtime | ||
|
|
||
| For an e.g manifest please look at [manifest](../daemonset/kubeadm-kuberouter-all-features-hostport.yaml) with necessary | ||
| changes required for `HostPort` functionality. | ||
|
|
||
| [cni-bridge]: https://www.cni.dev/plugins/v1.1/main/bridge/ | ||
| [cni-host-local]: https://www.cni.dev/plugins/v1.1/ipam/host-local/ | ||
| [containernetworking plugins]: https://github.com/containernetworking/plugins | ||
| [cni-portmap]: https://www.cni.dev/plugins/v1.1/meta/portmap/ | ||
| [cni-config-format]: https://www.cni.dev/docs/spec/#configuration-format | ||
| [CNI configuration template manifest]: ../daemonset/generic-kuberouter-cni-conf-template.yaml | ||
|
|
||
| ## IPVS Graceful termination support | ||
|
|
||
| We support experimental graceful termination of IPVS destinations. When possible, the pod's | ||
|
|
||
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think "On startup..." is a good spot to place a line break for readability.
I think adding a colon at the end helps visually for formatting with the bullet points right below.