Kubernetes联盟演变

s:Irfan Ur Rehman(华为),Paul Morie(RedHat)和Shashidhara T D(华为)

Kubernetes provides great primitives for deploying applications to a cluster: it can be as simple as kubectl create -f app.yaml. Deploy apps across multiple clusters has never been that simple. How should app workloads be distributed? Should the app resources be replicated into all clusters, replicated into selected clusters, or partitioned into clusters? How is access to the clusters managed? What happens if some of the resources that a user wants to distribute pre-exist, in some or all of the clusters, in some form?

在SIG Multicluster中,我们的旅程表明,可以使用多种模型来解决这些问题,并且可能没有单一的最佳拟合,所有方案。 联邦但是,这是Kubernetes最大的单个开源子项目,并且在这个问题空间中社区看到了最大的兴趣和贡献。该项目最初重用了Kubernetes API,以消除现有Kubernetes用户所增加的使用复杂性。由于以下问题,此方法不可行:

  • 在集群级别重新实现Kubernetes API的困难,因为特定于联盟的扩展存储在注释中。
  • 由于Kubernetes API的1:1仿真,在联合类型,放置和对帐方面的灵活性有限。
  • 没有通向通用Analytics(分析)的既定路径,并且对API成熟度普遍感到困惑;例如,在Kubernetes中部署是GA,但在Federation v1中甚至不是Beta。

通过特定于联盟的API架构和社区的努力,这些思想得到了进一步的发展,现在随着Federation v2的发展而不断发展。

概念概述

由于联合会试图解决一系列复杂的问题,因此必须分解这些问题的不同部分。让我们看一下涉及的不同高级领域:

Kubernetes 联邦 v2概念

Kubernetes 联邦 v2概念

联合任意资源

联盟的主要目标之一是能够定义API和API组,这些API和API组包含联合任何给定Kubernetes资源所需的基本原则。由于CustomResourceDefinitions作为使用新API扩展Kubernetes的一种方式,因此这至关重要。

工作组对联邦API和API组达成了共同的定义, “一种将“正常” Kubernetes API资源分配到不同集群中的机制”。最简单形式的分布可以想象为 简单传播 这个的 '普通的Kubernetes API资源' 跨联合集群。除了Kubernetes资源的这种简单传播之外,有思想的读者当然可以辨别更复杂的机制。

在定义联邦API的构建块的过程中,近期目标之一也演变为 '能够创建一个简单的联盟,也就是任何Kubernetes资源或CRD的简单传播,编写几乎为零的代码'. What ensued further was a core API group defining the building blocks as a Template resource, a Placement resource 和 an Override resource per given Kubernetes resource, a TypeConfig to specify sync or no sync for the given resource 和 associated controller(s) to carry out the sync. More details follow 在下一节中。进一步的部分还将讨论能够使用高层联合API来遵循分层行为,从而使用这些核心构建块的行为,以及用户能够使用全部或部分API以及关联的控制器。最后,该架构还允许用户编写其他控制器或用自己的控制器替换可用的参考控制器,以执行所需的行为。

的能力 轻松联合任意Kubernetes资源以及分离后的API(分为构建模块API,更高级别的API和可能的用户预期类型)呈现出来,以使不同的用户可以使用零件并编写控制器来构成特定于他们的解决方案,这对于Federation v2来说是令人信服的。

联合资源:详细信息

从根本上讲,联合身份验证必须配置两种类型的信息:

  • 联盟应处理哪些API类型
  • 哪些集群联合会应以分配这些资源为目标。

对于联盟处理的每种API类型,声明状态的不同部分位于不同的API资源中:

  • A Template type holds the base specification of the resource - for example, a type called FederatedReplicaSet holds the base specification of a ReplicaSet that should be distributed to the targeted clusters
  • A Placement type holds the specification of the clusters the resource should be distributed to - for example, a type called FederatedReplicaSetPlacement holds information about which clusters FederatedReplicaSets should be distributed to
  • An optional Overrides type holds the specification of how the Template resource should be varied in some clusters - for example, a type called FederatedReplicaSetOverrides holds information about how a FederatedReplicaSet should be varied in certain clusters.

These types are all associated by name - meaning that for a particular Template resource with name foo, the Placement 和 Override information for that resource are contained by the Override 和 Placement resources with the name foo 和 in the same namespace as the Template.

高层行为

The architecture of the v2 API allows higher-level APIs to be constructed using the mechanics provided by the core API types (Template, PlacementOverride), 和 associated controllers, for a given resource. In the community we uncovered a few use cases 和 implemented the higher-level APIs 和 associated controllers useful for those cases. Some of these types described in further sections also provide an useful reference to anybody interested in solving more complex use cases, building on top of the mechanics already available with the v2 API.

ReplicaSchedulingPreference

ReplicaSchedulingPreference 提供了一种自动机制,用于将基于Deployment或ReplicaSet的联合工作负载的副本总数分配和维护到联合群集中。这基于用户给出的高级用户首选项。这些首选项包括 加权分布极限 (最小和最大)以分发副本。这些还包括语义,以允许在某些群集中某些副本Pod保持未计划状态(例如,由于该群集中的资源不足)而动态地重新分发副本。 可以在以下网站找到更多详细信息 ReplicaSchedulingPreferences用户指南.

联合服务和跨集群服务发现

Kubernetes Services在构建微服务架构中非常有用。显然希望跨集群,区域,区域和云边界部署服务。跨群集的服务提供了地理分布,支持混合和多云场景,并提高了单群集部署之外的高可用性级别。希望其服务跨越一个或多个(可能是远程)集群的客户,需要能够以一致的方式从集群内部和外部访问它们。

Federated Service, at its core, contains a Template (a definition of a Kubernetes Service), a Placement (which clusters to be deployed into), an Override (optional variation in particular clusters) 和 a ServiceDNSRecord (specifying details on how to discover it).

Note: The federated service has to be of type LoadBalancer in order for it to be discoverable across clusters.

从联合群集中的Pods发现联合服务

By default, Kubernetes clusters come preconfigured with a cluster-local DNS server, as well as an intelligently constructed DNS search path, which together ensure that DNS queries like myservice, myservice.mynamespace, or some-other-service.other-namespace, issued by software running inside Pods, are automatically expanded 和 resolved correctly to the appropriate IP of Services running in the local cluster.

With the introduction of federated services 和 cross-cluster service discovery, this concept is extended to cover Kubernetes Services running in any other cluster across your cluster federation, globally. To take advantage 这个的 extended range, you use a slightly different DNS name (e.g. myservice.mynamespace.myfederation) to resolve federated services. Using a different DNS name also avoids having your existing applications accidentally traversing cross-zone or cross-region networks 和 you incurring perhaps unwanted network charges or latency, without you explicitly opting in to this behavior.

Lets consider an example, using a service named nginx.

A Pod in a cluster in the us-central1-a availability zone needs to contact our nginx service. Rather than use the service’s traditional cluster-local DNS name (nginx.mynamespace, which is automatically expanded to nginx.mynamespace.svc.cluster.local) it can now use the service’s federated DNS name, which is nginx.mynamespace.myfederation. This will be automatically expanded 和 resolved to the closest healthy shard of my nginx service, wherever in the world that may be. If a healthy shard exists in the local cluster, that service’s cluster-local IP address will be returned (by the cluster-local DNS). This is exactly equivalent to non-federated service resolution.

If the Service does not exist in the local cluster (or it exists but has no healthy backend pods), the DNS query is automatically expanded to nginx.mynamespace.myfederation.svc.us-central1-a.us-central1.example.com. Behind the scenes, this finds the external IP of one of the shards closest to my availability zone. This expansion is performed automatically by the cluster-local DNS server, which returns the associated CNAME record. This results in a traversal of the hierarchy of DNS records, 和 ends up at one of the external IP’s of the federated service nearby.

It is also possible to target service shards in availability zones 和 regions other than the ones local to a Pod by specifying the appropriate DNS names explicitly, 和 not relying on automatic DNS expansion. For example, nginx.mynamespace.myfederation.svc.europe-west1.example.comwill resolve to all of the currently healthy service shards in Europe, even if the Pod issuing the lookup is located in the U.S., 和 irrespective of whether or not there are healthy shards of the service in the U.S. This is useful for remote monitoring 和 other similar applications.

从联合群集之外的其他客户端发现联合服务

对于外部客户端,当前无法描述所述的自动DNS扩展。外部客户端需要指定联合服务的标准DNS名称之一,可以是区域名称,区域名称或全局名称。为了方便起见,通常最好在服务中手动配置其他静态CNAME记录,例如:

简称CNAME
eu.nginx.acme.comnginx.mynamespace.myfederation.svc.europe-west1.example.com
us.nginx.acme.comnginx.mynamespace.myfederation.svc.us-central1.example.com
nginx.acme.comnginx.mynamespace.myfederation.svc.example.com

这样,您的客户就可以始终使用左侧的缩写形式,并始终被自动路由到其本国大陆上最接近的健康分片。 Kubernetes集群联合会自动为您处理所有必需的故障转移。

作为进一步的阅读,可以在 带有ExternalDNS的多群集服务DNS指南.

自己尝试

要开始使用联盟v2,请参阅 用户指南。部署可以通过以下方式完成 舵图,并且控制平面可用后, 用户指南的示例 可以用来获得一些使用Federation V2的动手经验。

联盟v2可以部署在两个 集群范围命名空间范围 配置。群集范围的部署将要求主机群集和成员群集都具有群集管理员特权,并且可能非常适合评估未运行关键工作负载的群集上的联合身份验证。命名空间范围的部署仅需要访问主机群集和成员群集上的单个命名空间,因此更适合评估运行工作负载的群集上的联合身份。大多数用户指南都涉及集群范围的部署,其中包括 命名空间联盟 本节记录命名空间部署的用法有何不同。同一集群可以承载多个联盟,并且在使用命名空间联盟时,集群可以成为多个联盟的一部分。

下一步

正如我们在本文开头所指出的,多集群问题空间非常广泛。如果没有具体的软件来围绕这些对话,可能很难确切地知道如何处理如此广泛的问题空间。我们希望在联邦工作组中,联邦v2可以成为框架讨论的具体工具。我们很想知道人们在这个问题空间中的经验,他们对Federation v2的看法以及他们将来有兴趣探索的用例。

请欢迎加入我们 sig-multicluster松弛通道联邦工作组会议 在太平洋标准时间星期三07:30。