血と汗となみだを流す

個人の思ったこと、やったことの吐き出し口です。

Developers.ioの記事を読んでやってみる「Kubernetes Cluster on AWS」

対象

やってみる

前提

  • terraformは今回使わずに、必要なAWSリソースは全てAWSコンソールから作りました

kopsインストール

  • インストール
$ brew update && brew install kops
  • バージョン確認
$ kops version
Version 1.8.1

必要なリソースを作成(手動)

環境変数設定

export NAME=kubernetes.anorlondo448.net
export KOPS_STATE_STORE=s3://kubernetes-anorlondo448-net-state-store

クラスター設定の保存

  • Availability Zoneは1a,1dを指定した
  • --yesを指定しないときは、S3にクラスタ設定の保存のみ
  • --yesを指定すると、クラスターの構築までやってくれる
$ kops create cluster --zones ap-northeast-1a,ap-northeast-1d --name ${NAME}
  • すげーいろいろなリソースの設定が作られる！
~長いし、秘匿情報ありそうなので抜粋~

I0410 06:17:15.182664    2707 create_cluster.go:439] Inferred --cloud=aws from zone "ap-northeast-1a"
I0410 06:17:15.183004    2707 create_cluster.go:971] Using SSH public key: /Users/AnorLondo/.ssh/id_rsa.pub
I0410 06:17:16.433661    2707 subnets.go:184] Assigned CIDR 172.20.32.0/19 to subnet ap-northeast-1a
I0410 06:17:16.433702    2707 subnets.go:184] Assigned CIDR 172.20.64.0/19 to subnet ap-northeast-1d
Previewing changes that will be made:

I0410 06:17:20.281685    2707 executor.go:91] Tasks: 0 done / 75 total; 31 can run
I0410 06:17:21.290746    2707 executor.go:91] Tasks: 31 done / 75 total; 25 can run
I0410 06:17:23.283386    2707 executor.go:91] Tasks: 56 done / 75 total; 17 can run
I0410 06:17:23.446478    2707 executor.go:91] Tasks: 73 done / 75 total; 2 can run
I0410 06:17:23.499685    2707 executor.go:91] Tasks: 75 done / 75 total; 0 can run

Will create resources:
  AutoscalingGroup/master-ap-northeast-1a.masters.kubernetes.anorlondo448.net
  AutoscalingGroup/nodes.kubernetes.anorlondo448.net
  DHCPOptions/kubernetes.anorlondo448.net
  EBSVolume/a.etcd-events.kubernetes.anorlondo448.net
  EBSVolume/a.etcd-main.kubernetes.anorlondo448.net
  IAMInstanceProfile/masters.kubernetes.anorlondo448.net
  IAMInstanceProfile/nodes.kubernetes.anorlondo448.net
  IAMInstanceProfileRole/masters.kubernetes.anorlondo448.net
  IAMInstanceProfileRole/nodes.kubernetes.anorlondo448.net
  IAMRole/masters.kubernetes.anorlondo448.net
  IAMRole/nodes.kubernetes.anorlondo448.net
  IAMRolePolicy/masters.kubernetes.anorlondo448.net
  IAMRolePolicy/nodes.kubernetes.anorlondo448.net
  InternetGateway/kubernetes.anorlondo448.net
  Keypair/apiserver-aggregator
  Keypair/apiserver-aggregator-ca
  Keypair/apiserver-proxy-client
  Keypair/ca
  Keypair/kops
  Keypair/kube-controller-manager
  Keypair/kube-proxy
  Keypair/kube-scheduler
  Keypair/kubecfg
  Keypair/kubelet
  Keypair/kubelet-api
  Keypair/master
  LaunchConfiguration/master-ap-northeast-1a.masters.kubernetes.anorlondo448.net
  LaunchConfiguration/nodes.kubernetes.anorlondo448.net
  ManagedFile/kubernetes.anorlondo448.net-addons-bootstrap
  ManagedFile/kubernetes.anorlondo448.net-addons-core.addons.k8s.io
  ManagedFile/kubernetes.anorlondo448.net-addons-dns-controller.addons.k8s.io-k8s-1.6
  ManagedFile/kubernetes.anorlondo448.net-addons-dns-controller.addons.k8s.io-pre-k8s-1.6
  ManagedFile/kubernetes.anorlondo448.net-addons-kube-dns.addons.k8s.io-k8s-1.6
  ManagedFile/kubernetes.anorlondo448.net-addons-kube-dns.addons.k8s.io-pre-k8s-1.6
  ManagedFile/kubernetes.anorlondo448.net-addons-limit-range.addons.k8s.io
  ManagedFile/kubernetes.anorlondo448.net-addons-rbac.addons.k8s.io-k8s-1.8
  ManagedFile/kubernetes.anorlondo448.net-addons-storage-aws.addons.k8s.io-v1.6.0
  ManagedFile/kubernetes.anorlondo448.net-addons-storage-aws.addons.k8s.io-v1.7.0
  Route/0.0.0.0/0
  RouteTable/kubernetes.anorlondo448.net
  RouteTableAssociation/ap-northeast-1a.kubernetes.anorlondo448.net
  RouteTableAssociation/ap-northeast-1d.kubernetes.anorlondo448.net
  SSHKey/kubernetes.kubernetes.anorlondo448.net-XX:XX:XX:XX:XX:XX:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx
  Secret/admin
  Secret/kube
  Secret/kube-proxy
  Secret/kubelet
  Secret/system:controller_manager
  Secret/system:dns
  Secret/system:logging
  Secret/system:monitoring
  Secret/system:scheduler
  SecurityGroup/masters.kubernetes.anorlondo448.net
  SecurityGroup/nodes.kubernetes.anorlondo448.net
  SecurityGroupRule/all-master-to-master
  SecurityGroupRule/all-master-to-node
  SecurityGroupRule/all-node-to-node
  SecurityGroupRule/https-external-to-master-0.0.0.0/0
  SecurityGroupRule/master-egress
  SecurityGroupRule/node-egress
  SecurityGroupRule/node-to-master-tcp-1-2379
  SecurityGroupRule/node-to-master-tcp-2382-4000
  SecurityGroupRule/node-to-master-tcp-4003-65535
  SecurityGroupRule/node-to-master-udp-1-65535
  SecurityGroupRule/ssh-external-to-master-0.0.0.0/0
  SecurityGroupRule/ssh-external-to-node-0.0.0.0/0
  Subnet/ap-northeast-1a.kubernetes.anorlondo448.net
  Subnet/ap-northeast-1d.kubernetes.anorlondo448.net
  VPC/kubernetes.anorlondo448.net
  VPCDHCPOptionsAssociation/kubernetes.anorlondo448.net

Must specify --yes to apply changes

Cluster configuration has been created.

Suggestions:
 * list clusters with: kops get cluster
 * edit this cluster with: kops edit cluster kubernetes.anorlondo448.net
 * edit your node instance group: kops edit ig --name=kubernetes.anorlondo448.net nodes
 * edit your master instance group: kops edit ig --name=kubernetes.anorlondo448.net master-ap-northeast-1a

Finally configure your cluster with: kops update cluster kubernetes.anorlondo448.net --yes

クラスター構成の変更

  • 以下コマンドで、S3上に保存されているconfigを変更できるっぽい(要update)
$ kops edit cluster ${NAME}
  • viで編集できる
# Please edit the object below. Lines beginning with a '#' will be ignored,
# and an empty file will abort the edit. If an error occurs while saving this file will be
# reopened with the relevant failures.
#
apiVersion: kops/v1alpha2
kind: Cluster
metadata:
  creationTimestamp: 2018-04-09T21:17:17Z
  name: kubernetes.anorlondo448.net
spec:
  api:
    dns: {}
  authorization:
    alwaysAllow: {}
  channel: stable
  cloudProvider: aws
...(略)

クラスター構築

  • 設定作成後にupdateで構築
$ kops update cluster ${NAME} --yes
I0410 06:27:43.004766    2901 executor.go:91] Tasks: 0 done / 75 total; 31 can run
I0410 06:27:43.487336    2901 vfs_castore.go:435] Issuing new certificate: "apiserver-aggregator-ca"
I0410 06:27:43.543962    2901 vfs_castore.go:435] Issuing new certificate: "ca"
I0410 06:27:45.545526    2901 executor.go:91] Tasks: 31 done / 75 total; 25 can run
I0410 06:27:46.344804    2901 vfs_castore.go:435] Issuing new certificate: "master"
I0410 06:27:46.466986    2901 vfs_castore.go:435] Issuing new certificate: "apiserver-aggregator"
I0410 06:27:46.533313    2901 vfs_castore.go:435] Issuing new certificate: "kube-controller-manager"
I0410 06:27:46.674253    2901 vfs_castore.go:435] Issuing new certificate: "apiserver-proxy-client"
I0410 06:27:46.809262    2901 vfs_castore.go:435] Issuing new certificate: "kube-scheduler"
I0410 06:27:46.935837    2901 vfs_castore.go:435] Issuing new certificate: "kubelet"
I0410 06:27:46.998773    2901 vfs_castore.go:435] Issuing new certificate: "kube-proxy"
I0410 06:27:47.134661    2901 vfs_castore.go:435] Issuing new certificate: "kubelet-api"
I0410 06:27:47.333686    2901 vfs_castore.go:435] Issuing new certificate: "kops"
I0410 06:27:47.697080    2901 vfs_castore.go:435] Issuing new certificate: "kubecfg"
I0410 06:27:47.840890    2901 executor.go:91] Tasks: 56 done / 75 total; 17 can run
I0410 06:27:49.015154    2901 launchconfiguration.go:333] waiting for IAM instance profile "masters.kubernetes.anorlondo448.net" to be ready
I0410 06:27:49.159834    2901 launchconfiguration.go:333] waiting for IAM instance profile "nodes.kubernetes.anorlondo448.net" to be ready
I0410 06:28:00.108320    2901 executor.go:91] Tasks: 73 done / 75 total; 2 can run
I0410 06:28:00.824940    2901 executor.go:91] Tasks: 75 done / 75 total; 0 can run
I0410 06:28:00.825059    2901 dns.go:153] Pre-creating DNS records
I0410 06:28:02.910720    2901 update_cluster.go:248] Exporting kubecfg for cluster
kops has set your kubectl context to kubernetes.anorlondo448.net

Cluster is starting.  It should be ready in a few minutes.

Suggestions:
 * validate cluster: kops validate cluster
 * list nodes: kubectl get nodes --show-labels
 * ssh to the master: ssh -i ~/.ssh/id_rsa admin@api.kubernetes.anorlondo448.net
The admin user is specific to Debian. If not using Debian please use the appropriate user based on your OS.
 * read about installing addons: https://github.com/kubernetes/kops/blob/master/docs/addons.md

クラスター確認

$ kops validate cluster
  • master×1,node×2が起動！
Using cluster from kubectl context: kubernetes.anorlondo448.net

Validating cluster kubernetes.anorlondo448.net

INSTANCE GROUPS
NAME                    ROLE    MACHINETYPE     MIN     MAX     SUBNETS
master-ap-northeast-1a  Master  m3.medium       1       1       ap-northeast-1a
nodes                   Node    t2.medium       2       2       ap-northeast-1a,ap-northeast-1d

NODE STATUS
NAME                                                    ROLE    READY
ip-172-20-46-129.ap-northeast-1.compute.internal        master  True

Validation Failed
Ready Master(s) 1 out of 1.
Ready Node(s) 0 out of 2.

your nodes are NOT ready kubernetes.anorlondo448.net

ノードの状態確認

$ kubectl get nodes
NAME                                               STATUS     ROLES     AGE       VERSION
ip-172-20-32-126.ap-northeast-1.compute.internal   NotReady   node      17s       v1.8.7
ip-172-20-46-129.ap-northeast-1.compute.internal   Ready      master    2m        v1.8.7
ip-172-20-70-148.ap-northeast-1.compute.internal   Ready      node      38s       v1.8.7

MasterノードにSSH

  • デフォルトの秘密鍵でログイン
    • これを任意に変えられないものか・・・・？
$ ssh -i ~/.ssh/id_rsa admin@api.kubernetes.anorlondo448.net
...
The programs included with the Debian GNU/Linux system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.

Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
permitted by applicable law.
_____________________________________________________________________
WARNING! Your environment specifies an invalid locale.
 This can affect your user experience significantly, including the
 ability to manage packages. You may install the locales by running:

   sudo apt-get install language-pack-ja
     or
   sudo locale-gen ja_JP.UTF-8

To see all available language packs, run:
   apt-cache search "^language-pack-[a-z][a-z]$"
To disable this message for all users, run:
   sudo touch /var/lib/cloud/instance/locale-check.skip
_____________________________________________________________________

システムコンポーネント確認

$ kubectl -n kube-system get po
NAME                                                                       READY     STATUS    RESTARTS   AGE
dns-controller-54cfdf44c7-j9gd4                                            1/1       Running   0          5m
etcd-server-events-ip-172-20-46-129.ap-northeast-1.compute.internal        1/1       Running   0          5m
etcd-server-ip-172-20-46-129.ap-northeast-1.compute.internal               1/1       Running   0          5m
kube-apiserver-ip-172-20-46-129.ap-northeast-1.compute.internal            1/1       Running   2          5m
kube-controller-manager-ip-172-20-46-129.ap-northeast-1.compute.internal   1/1       Running   0          5m
kube-dns-7f56f9f8c7-2gj2f                                                  3/3       Running   0          3m
kube-dns-7f56f9f8c7-4v6bn                                                  3/3       Running   0          6m
kube-dns-autoscaler-f4c47db64-89s8k                                        1/1       Running   0          6m
kube-proxy-ip-172-20-32-126.ap-northeast-1.compute.internal                1/1       Running   0          3m
kube-proxy-ip-172-20-46-129.ap-northeast-1.compute.internal                1/1       Running   0          5m
kube-proxy-ip-172-20-70-148.ap-northeast-1.compute.internal                1/1       Running   0          3m
kube-scheduler-ip-172-20-46-129.ap-northeast-1.compute.internal            1/1       Running   0          5m

クラスター削除

$ kops delete cluster --name ${NAME} --yes
  • kops createで作成したAWSリソースも全部消える
W0410 06:44:32.446868    3296 aws.go:2036] (new) cluster tag not found on route-table:rtb-e81cf98e
W0410 06:44:35.537119    3296 aws.go:2036] (new) cluster tag not found on route-table:rtb-e81cf98e
TYPE                    NAME                                                                                    ID
autoscaling-config      master-ap-northeast-1a.masters.kubernetes.anorlondo448.net-20180409212748               master-ap-northeast-1a.masters.kubernetes.anorlondo448.net-20180409212748
autoscaling-config      nodes.kubernetes.anorlondo448.net-20180409212748                                        nodes.kubernetes.anorlondo448.net-20180409212748
autoscaling-group       master-ap-northeast-1a.masters.kubernetes.anorlondo448.net                              master-ap-northeast-1a.masters.kubernetes.anorlondo448.net
autoscaling-group       nodes.kubernetes.anorlondo448.net                                                       nodes.kubernetes.anorlondo448.net
dhcp-options            kubernetes.anorlondo448.net                                                             dopt-ff269698
iam-instance-profile    masters.kubernetes.anorlondo448.net                                                     masters.kubernetes.anorlondo448.net
iam-instance-profile    nodes.kubernetes.anorlondo448.net                                                       nodes.kubernetes.anorlondo448.net
iam-role                masters.kubernetes.anorlondo448.net                                                     masters.kubernetes.anorlondo448.net
iam-role                nodes.kubernetes.anorlondo448.net                                                       nodes.kubernetes.anorlondo448.net
instance                master-ap-northeast-1a.masters.kubernetes.anorlondo448.net                              i-00fc02ce018386dff
instance                nodes.kubernetes.anorlondo448.net                                                       i-067e4c59f1fad0284
instance                nodes.kubernetes.anorlondo448.net                                                       i-09b799b6079c9a70d
internet-gateway        kubernetes.anorlondo448.net                                                             igw-088ef66c
keypair                 kubernetes.kubernetes.anorlondo448.net-xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx  kubernetes.kubernetes.anorlondo448.net-xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx
route-table             kubernetes.anorlondo448.net                                                             rtb-e81cf98e
route53-record          api.internal.kubernetes.anorlondo448.net.                                               Z3PCHTDVQ5REWS/api.internal.kubernetes.anorlondo448.net.
route53-record          api.kubernetes.anorlondo448.net.                                                        Z3PCHTDVQ5REWS/api.kubernetes.anorlondo448.net.
route53-record          etcd-a.internal.kubernetes.anorlondo448.net.                                            Z3PCHTDVQ5REWS/etcd-a.internal.kubernetes.anorlondo448.net.
route53-record          etcd-events-a.internal.kubernetes.anorlondo448.net.                                     Z3PCHTDVQ5REWS/etcd-events-a.internal.kubernetes.anorlondo448.net.
security-group          masters.kubernetes.anorlondo448.net                                                     sg-7bf8e502
security-group          nodes.kubernetes.anorlondo448.net                                                       sg-82fce1fb
subnet                  ap-northeast-1a.kubernetes.anorlondo448.net                                             subnet-a75579ee
subnet                  ap-northeast-1d.kubernetes.anorlondo448.net                                             subnet-5feed077
volume                  a.etcd-events.kubernetes.anorlondo448.net                                               vol-00be8f4de4ef6adb2
volume                  a.etcd-main.kubernetes.anorlondo448.net                                                 vol-00c4a7b8c0a6c4b04
vpc                     kubernetes.anorlondo448.net                                                             vpc-f42abb93

route53-record:Z3PCHTDVQ5REWS/api.kubernetes.anorlondo448.net.  ok
autoscaling-group:nodes.kubernetes.anorlondo448.net     ok
keypair:kubernetes.kubernetes.anorlondo448.net-xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx  ok
internet-gateway:igw-088ef66c   still has dependencies, will retry
autoscaling-group:master-ap-northeast-1a.masters.kubernetes.anorlondo448.net    ok
instance:i-067e4c59f1fad0284    ok
instance:i-00fc02ce018386dff    ok
instance:i-09b799b6079c9a70d    ok
iam-instance-profile:nodes.kubernetes.anorlondo448.net  ok
iam-instance-profile:masters.kubernetes.anorlondo448.net        ok
iam-role:masters.kubernetes.anorlondo448.net    ok
iam-role:nodes.kubernetes.anorlondo448.net      ok
subnet:subnet-5feed077  still has dependencies, will retry
autoscaling-config:nodes.kubernetes.anorlondo448.net-20180409212748     ok
autoscaling-config:master-ap-northeast-1a.masters.kubernetes.anorlondo448.net-20180409212748    ok
volume:vol-00c4a7b8c0a6c4b04    still has dependencies, will retry
subnet:subnet-a75579ee  still has dependencies, will retry
volume:vol-00be8f4de4ef6adb2    still has dependencies, will retry
security-group:sg-7bf8e502      still has dependencies, will retry
security-group:sg-82fce1fb      still has dependencies, will retry
Not all resources deleted; waiting before reattempting deletion
        dhcp-options:dopt-ff269698
        volume:vol-00be8f4de4ef6adb2
        security-group:sg-82fce1fb
        internet-gateway:igw-088ef66c
        route-table:rtb-e81cf98e
        volume:vol-00c4a7b8c0a6c4b04
        subnet:subnet-5feed077
        security-group:sg-7bf8e502
        vpc:vpc-f42abb93
        subnet:subnet-a75579ee
subnet:subnet-5feed077  still has dependencies, will retry
subnet:subnet-a75579ee  still has dependencies, will retry
volume:vol-00be8f4de4ef6adb2    still has dependencies, will retry
internet-gateway:igw-088ef66c   still has dependencies, will retry
security-group:sg-7bf8e502      still has dependencies, will retry
security-group:sg-82fce1fb      still has dependencies, will retry
volume:vol-00c4a7b8c0a6c4b04    still has dependencies, will retry
Not all resources deleted; waiting before reattempting deletion
        security-group:sg-7bf8e502
        subnet:subnet-5feed077
        vpc:vpc-f42abb93
        subnet:subnet-a75579ee
        security-group:sg-82fce1fb
        internet-gateway:igw-088ef66c
        dhcp-options:dopt-ff269698
        volume:vol-00be8f4de4ef6adb2
        volume:vol-00c4a7b8c0a6c4b04
        route-table:rtb-e81cf98e
subnet:subnet-a75579ee  still has dependencies, will retry
subnet:subnet-5feed077  still has dependencies, will retry
internet-gateway:igw-088ef66c   still has dependencies, will retry
volume:vol-00be8f4de4ef6adb2    still has dependencies, will retry
volume:vol-00c4a7b8c0a6c4b04    still has dependencies, will retry
security-group:sg-82fce1fb      still has dependencies, will retry
security-group:sg-7bf8e502      still has dependencies, will retry
Not all resources deleted; waiting before reattempting deletion
        internet-gateway:igw-088ef66c
        dhcp-options:dopt-ff269698
        volume:vol-00be8f4de4ef6adb2
        security-group:sg-82fce1fb
        volume:vol-00c4a7b8c0a6c4b04
        route-table:rtb-e81cf98e
        subnet:subnet-5feed077
        security-group:sg-7bf8e502
        vpc:vpc-f42abb93
        subnet:subnet-a75579ee
subnet:subnet-5feed077  still has dependencies, will retry
subnet:subnet-a75579ee  still has dependencies, will retry
internet-gateway:igw-088ef66c   still has dependencies, will retry
volume:vol-00be8f4de4ef6adb2    still has dependencies, will retry
volume:vol-00c4a7b8c0a6c4b04    still has dependencies, will retry
security-group:sg-7bf8e502      still has dependencies, will retry
security-group:sg-82fce1fb      still has dependencies, will retry
Not all resources deleted; waiting before reattempting deletion
        subnet:subnet-5feed077
        security-group:sg-7bf8e502
        vpc:vpc-f42abb93
        subnet:subnet-a75579ee
        dhcp-options:dopt-ff269698
        volume:vol-00be8f4de4ef6adb2
        security-group:sg-82fce1fb
        internet-gateway:igw-088ef66c
        route-table:rtb-e81cf98e
        volume:vol-00c4a7b8c0a6c4b04
volume:vol-00c4a7b8c0a6c4b04    ok
volume:vol-00be8f4de4ef6adb2    ok
subnet:subnet-5feed077  ok
security-group:sg-7bf8e502      ok
internet-gateway:igw-088ef66c   ok
subnet:subnet-a75579ee  ok
security-group:sg-82fce1fb      ok
route-table:rtb-e81cf98e        ok
vpc:vpc-f42abb93        ok
dhcp-options:dopt-ff269698      ok
Deleted kubectl config for kubernetes.anorlondo448.net

Deleted cluster: "kubernetes.anorlondo448.net"

次

  • 一通り流したので、出力情報などを細かく見ていくなど
プライバシーポリシー