MOSIP cluster - NGINX VMs, DNS Requirements, Istio ...using RKE/Rancher

rcsampang · July 29, 2022, 6:13am

Good day!
I am able to install Rancher on RKE cluster and now in the process of installing the MOSIP cluster.
I want to clarify, the VM for NGINX LB for MOSIP cluster is different from the VM for Rancher RKE cluster? Right?
Thus the internal IP for this VM is the one needed in the DNS Requirements’ internal IP?
One more thing, in installing NGINX in this VM, the install.sh script is configured to get IP from dummy-ips-nginx.service. Will this work on prem?
I tried it but the installation hangs at getting IP. What did I miss?
Also, any step by step procedure in installing Istio? Where to put the values for $INTERNAL and $Public or what to edit and what information to put.
Thank you for your kind assistance.

ckm007 · August 1, 2022, 8:35am

Good Day

Yes the VM’s for Rancher cluster nginx and Mosip cluster nginx are different.

Yes the internal IP is the one needed for the internal communication and external ip is the one needed for the external interaction with publically accessible modules.

Actually our install.sh script prompts there to input the internal and external ips seperately along with other parameters and then sets the same and install the nginx there.

github.com

mosip/k8s-infra/blob/main/mosip/on-prem/nginx/install.sh

#!/usr/bin/env bash
# Script to configure and install Nginx with public and private interfaces
# Usage: ./install.sh.


  if [ -z "$cluster_nginx_internal_ip" ]; then
    echo -en "=====>\nThe following internal ip will have to be DNS-mapped to all internal domains from you global_configmap.yaml. Ex: api-internal.sandbox.xyz.net, iam.sandbox.xyz.net, etc.\n"
    echo -en "Give the internal interface ip of this node here. Run \`ip a\` to get all the interface addresses (without any whitespaces) : "
    read cluster_nginx_internal_ip
  fi &&
  if [ -z "$cluster_nginx_public_ip" ]; then
    echo -en "=====>\nThis nginx's public ip will have to be DNS-mapped to all public domains from you global_configmap.yaml. Ex: api.sandbox.xyz.net, prereg.sandbox.xyz.net, etc.\nThe above mentioned nginx's public ip might be different from this nginx machine's public interface ip, if you have provisioned public ip seperately that might be forwarding traffic to this interface ip.\n"
    echo -en "Give the public interface ip of this node here. Run \`ip a\` to get all the interfaces, In case not exposing api's to public give private ip only. : "
    read cluster_nginx_public_ip
  fi &&
  if [ -z "$cluster_public_domains" ]; then
    echo -en "=====>\nGive list of (comma seperated) publicly exposing domain names (without any whitespaces). Ex: api.sandbox.xyx.net, prereg.sandbox.xyz.net, resident.sandbox.xyz.net, etc : "
    read cluster_public_domains
  fi &&
  if [ -z "$cluster_nginx_certs" ]; then

This file has been truncated. show original

Can you again brief where are we facing issues. also we have now stopped using dummy-ips-nginx.service.

github.com

mosip/k8s-infra/blob/main/mosip/on-prem/nginx/README.md

# NGINX Reverse Proxy Setup

## Introduction
* Nginx is used as a reverse proxy to direct traffic into the cluster via two channels - public and internal.
* The internal channel is front-ended by Wireguard. 
* The traffic is directed to NodePort of respective Ingress gateways (Istio). 
* The Nginx runs on a separate node that has access to public Internet and connects to services via nodeport.

## Prerequisites
* [Ansible](https://docs.ansible.com/ansible/latest/installation_guide/intro_installation.html).
* Provision one VM for Nginx, or multiple VMs for high avaiability like Nginx Plus.
* OS: Debian based. Recommended Ubuntu Server.
* [SSL certificates](../../../docs/wildcard-ssl-certs-letsencrypt.md).
* Make sure this Nginx node has two network interfaces:
    *  Public: Facing public Internet. _(Only required when accessing APIs over Intenet)_.
    *  Private: Must be on the same subnet as cluster nodes/machines.  Wireguard connects to this interface. 
* Command-line utilities:
  * `bash`
  * `sed`
* Nginx machine details are updated in `../hosts.ini`.

This file has been truncated. show original

rcsampang · August 1, 2022, 3:58pm

Right away I saw where my mistake is -

I have been following this MOSIP-infra github - mosip-infra/deployment/v3 at master · mosip/mosip-infra · GitHub

Thank you!

I have to read through this and try again. GitHub - mosip/k8s-infra: Kubernetes infrastructure to deploy MOSIP modules.

After going through this briefly, it only takes care of setting up the Kubernetes cluster.
I guess it means going back to MOSIP-infra github v3 and then follow the instructions in installing the MOSIP external components and core modules:

github.com

mosip/mosip-infra/blob/master/deployment/v3/external/README.md

# MOSIP External Components

This folder contains installation procedures for all components that are not part of core mosip offering - they need to be installed separately and integrated.  MOSIP has provided reference/mock implementations of some of these components that may be used for development and testing.  Some components may run inside the cluster itself, or outside depending on the choice of deployment.

## External components
* [Postgres](postgres/README.md)
* [IAM (Keycloak)](iam/README.md)
* [HSM](hsm/README.md)
* [Object Storage](object_store/README.md)
* [Anti-virus (Clamav)](antivirus/clamav/README.md)
* [ActiveMQ](activemq/aws/README.md)
* [Kafka](kafka/README.md)
* [BioSDK](biosdk/README.md)
* [ABIS](abis/README.md)
* [Message Gateways](msg_gateway/README.md)

github.com

mosip/mosip-infra/blob/master/deployment/v3/mosip/README.md

# MOSIP Core

Install in the following order:
* [Docker secrets](docker_secrets/README.md)
* [Prereg captcha](captcha/README.md)
* [Config server](config_server/README.md)
* [Artifactory](artifactory/README.md)
* [Keymanager](keymanager/README.md)
* [Kernel](kernel/README.md)
* [Packetmanager](packetmanager/README.md)
* [Datashare](datashare/README.md)
* [Pre-Reg](prereg/README.md)
* [Websub](websub/README.md)
* [IDRepo](idrepo/README.md)
* [PMS](pms/README.md)
* [RegProc](regproc/README.md)
* [Admin](admin/README.md)
* [Mock ABIS](mock-abis/README.md)
* [IDA](ida/README.md)
* [Registration Client](regclient/README.md)

This file has been truncated. show original

Is that right?

ckm007 · August 9, 2022, 10:01am

Yes… this is the correct sequence…
Basically start from mosip-infra. It will point you to cluster related scripts and docs in k8s-infra, once cluster is ready start with external and then MOSIP modules from mosip-infra.

rcsampang · August 11, 2022, 7:45pm

@ckm007 Thank you.

Everything seems to be installing fine, now that I am working on 1.2.0.1 branch.

I have additional questions though. Do I install Monitoring (Prometheus) before installing Istio?
I encountered this error when installing Istio, although afterwards everything looks to be fine.

:~/k8s-infra/mosip/on-prem/istio$ ./install.sh
Operator init
Installing operator controller in namespace: istio-operator using image: docker.io/istio/operator:1.14.3
Operator controller will watch namespaces: istio-system
Istio operator installed
Installation complete
Create ingress gateways, load balancers and istio monitoring
istiooperator.install.istio.io/istio-operators-mosip created
resource mapping not found for name: “envoy-stats-monitor” namespace: “istio-system” from “istio-monitoring/PodMonitor.yaml”: no matches for kind “PodMonitor” in version “monitoring.coreos.com/v1”
ensure CRDs are installed first
resource mapping not found for name: “istio-component-monitor” namespace: “istio-system” from “istio-monitoring/ServiceMonitor.yaml”: no matches for kind “ServiceMonitor” in version “monitoring.coreos.com/v1”
ensure CRDs are installed first
Wait for all resources to come up
Error from server (NotFound): deployments.apps “istiod” not found
Error from server (NotFound): deployments.apps “istio-ingressgateway” not found
Error from server (NotFound): deployments.apps “istio-ingressgateway-internal” not found
Installing gateways, proxy protocol, authpolicies
Public domain: api.dcsmosip.science.upd.edu.ph
Internal dome: api-internal.dcsmosip.science.upd.edu.ph
NAME: istio-addons
LAST DEPLOYED: Thu Aug 11 18:52:45 2022
NAMESPACE: istio-system
STATUS: deployed
REVISION: 1
TEST SUITE: None
------ IMPORTANT ---------
If you already have pods running with envoy sidecars, restart all of them NOW. Check if all of them appear with command istioctl proxy-status
--------------------------

:~/k8s-infra/mosip/on-prem/istio$ istioctl proxy-status
NAME CLUSTER CDS LDS EDS RDS ECDS ISTIOD VERSION
istio-ingressgateway-55cc548579-ljvfs.istio-system Kubernetes SYNCED SYNCED SYNCED SYNCED NOT SENT istiod-5947f988df-zd7nd 1.14.3
istio-ingressgateway-internal-78bd7888f-mt7h7.istio-system Kubernetes SYNCED SYNCED SYNCED SYNCED NOT SENT istiod-5947f988df-zd7nd 1.14.3

:~/k8s-infra/mosip/on-prem/istio$ kubectl get svc -n istio-system
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
istio-ingressgateway NodePort 10.43.214.92 15021:30521/TCP,80:30080/TCP 3m17s
istio-ingressgateway-internal NodePort 10.43.69.173 15021:31521/TCP,80:31080/TCP,61616:31616/TCP,5432:31432/TCP 3m17s
istiod ClusterIP 10.43.65.178 15010/TCP,15012/TCP,443/TCP,15014/TCP 3m27s

:~/k8s- NAMESPACE cattle-fleet-system cattle-system cattle-system cattle-system istio-operator istio-system istio-system istio-system kube-system kube-system kube-system kube-system kube-system kube-system kube-system kube-system kube-system kube-system kube-system kube-system kube-system longhorn-system longhorn-system longhorn-system longhorn-system longhorn-system longhorn-system longhorn-system longhorn-system longhorn-system longhorn-system longhorn-system longhorn-system longhorn-system longhorn-system longhorn-system longhorn-system longhorn-system longhorn-system longhorn-system longhorn-system longhorn-system longhorn-system longhorn-system longhorn-system longhorn-system longhorn-system longhorn-system longhorn-system longhorn-system longhorn-system longhorn-system longhorn-system longhorn-system longhorn-system longhorn-system longhorn-system longhorn-system longhorn-system longhorn-system longhorn-system longhorn-system longhorn-system longhorn-system longhorn-system longhorn-system longhorn-system longhorn-system longhorn-system longhorn-system longhorn-system longhorn-system longhorn-system longhorn-system infra/mosip/on-prem/istio$ kubectl get pods -A
NAME READY STATUS RESTARTS AGE
fleet-agent-96f6f455c-b68c9 1/1 Running 0 75m
cattle-cluster-agent-6569ffb898-stmms 1/1 Running 3 77m
cattle-cluster-agent-6569ffb898-t6bkf 1/1 Running 0 76m
helm-operation-hv42d 0/2 Completed 0 54m
istio-operator-84d9f8d555-fqbhs 1/1 Running 0 5m58s
istio-ingressgateway-55cc548579-ljvfs 1/1 Running 0 5m23s
istio-ingressgateway-internal-78bd7888f-mt7h7 1/1 Running 0 5m23s
istiod-5947f988df-zd7nd 1/1 Running 0 5m33s
calico-kube-controllers-85f985646c-2xr9c 1/1 Running 2 108m
canal-5hngh 2/2 Running 0 108m
canal-c22pl 2/2 Running 0 108m
canal-dlhkt 2/2 Running 0 108m
canal-jbzrn 2/2 Running 0 108m
canal-tn5qr 2/2 Running 0 108m
coredns-685d6d555d-56dh5 1/1 Running 0 108m
coredns-685d6d555d-96dtt 1/1 Running 0 107m
coredns-autoscaler-96789f8f7-gs4v2 1/1 Running 0 108m
metrics-server-d67876c44-j2pjg 0/1 Running 0 108m
rke-coredns-addon-deploy-job-zph4f 0/1 Completed 0 108m
rke-metrics-addon-deploy-job-46w49 0/1 Completed 0 108m
rke-network-plugin-deploy-job-cjmtl 0/1 Completed 0 108m
csi-attacher-5ddf9c48cf-4sqv5 1/1 Running 0 54m
csi-attacher-5ddf9c48cf-74tt9 1/1 Running 0 54m
csi-attacher-5ddf9c48cf-lq5kg 1/1 Running 0 54m
csi-provisioner-59b7b8b7b8-bbsc2 1/1 Running 0 54m
csi-provisioner-59b7b8b7b8-fcnt6 1/1 Running 0 54m
csi-provisioner-59b7b8b7b8-mj98b 1/1 Running 0 54m
csi-resizer-68ccff94-75kbs 1/1 Running 0 54m
csi-resizer-68ccff94-vjqfz 1/1 Running 0 54m
csi-resizer-68ccff94-zc2pc 1/1 Running 0 54m
csi-snapshotter-6d7d679c98-8gprg 1/1 Running 0 54m
csi-snapshotter-6d7d679c98-n2fzr 1/1 Running 0 54m
csi-snapshotter-6d7d679c98-q24vk 1/1 Running 0 54m
engine-image-ei-dae99989-24l44 1/1 Running 0 54m
engine-image-ei-dae99989-bvnpm 1/1 Running 0 54m
engine-image-ei-dae99989-n4rql 1/1 Running 0 54m
engine-image-ei-dae99989-tnwfr 1/1 Running 0 54m
engine-image-ei-dae99989-z5zr7 1/1 Running 0 54m
instance-manager-e-21c0775e 1/1 Running 0 31m
instance-manager-e-2b38f706 1/1 Running 0 31m
instance-manager-e-52b8c8a9 1/1 Running 0 31m
instance-manager-e-819e3e0e 1/1 Running 0 31m
instance-manager-e-864a1812 1/1 Running 0 31m
instance-manager-r-1bfeb938 1/1 Running 0 31m
instance-manager-r-25790442 1/1 Running 0 31m
instance-manager-r-3356dc3a 1/1 Running 0 31m
instance-manager-r-8a0e0d51 1/1 Running 0 31m
instance-manager-r-bfdb03da 1/1 Running 0 31m
longhorn-admission-webhook-57ddddf8f8-8xtnn 1/1 Running 0 54m
longhorn-admission-webhook-57ddddf8f8-r4m55 1/1 Running 0 54m
longhorn-conversion-webhook-7d7cf6f877-qgnwh 1/1 Running 0 54m
longhorn-conversion-webhook-7d7cf6f877-wcjc7 1/1 Running 0 54m
longhorn-csi-plugin-5ztkr 2/2 Running 0 54m
longhorn-csi-plugin-dsdjv 2/2 Running 0 54m
longhorn-csi-plugin-g9ppp 2/2 Running 0 54m
longhorn-csi-plugin-h874p 2/2 Running 0 54m
longhorn-csi-plugin-r9ngt 2/2 Running 0 54m
longhorn-driver-deployer-69fcc75f85-j9xb2 1/1 Running 0 54m
longhorn-iscsi-installation-6lfpz 1/1 Running 0 65m
longhorn-iscsi-installation-bvrwd 1/1 Running 0 65m
longhorn-iscsi-installation-l78c7 1/1 Running 0 65m
longhorn-iscsi-installation-pcg7z 1/1 Running 0 65m
longhorn-iscsi-installation-xwjq9 1/1 Running 0 65m
longhorn-manager-cn5n7 1/1 Running 1 54m
longhorn-manager-ftjfw 1/1 Running 0 54m
longhorn-manager-lldfj 1/1 Running 0 54m
longhorn-manager-lrg59 1/1 Running 1 54m
longhorn-manager-pr5dm 1/1 Running 0 54m
longhorn-nfs-installation-cnsjf 1/1 Running 0 65m
longhorn-nfs-installation-jh49h 1/1 Running 0 65m
longhorn-nfs-installation-lm49n 1/1 Running 0 65m
longhorn-nfs-installation-rc4l4 1/1 Running 0 65m
longhorn-nfs-installation-vxwh6 1/1 Running 0 65m
longhorn-ui-6f45bd5c65-zrnxt 1/1 Running 0 54m

Also, what needs to be done in this part?

Istio injection

To enable Istio injection in a namespace:

*kubectl label ns <namespace> istio-injection=enabled --overwrite*

What namespaces do I need to configure with istio-injection ?

ckm007 · August 16, 2022, 5:30am

Noted @rcsampang.

Basically as per documents we should install istion first as part of cluster setup. Since the same was not followed it is throwing error for already deployed monitoring services.
So you can delete the monitoring services there and re install the same. It will fix the existing issue. Also note the error:
ensure CRDs are installed first*
resource mapping not found for name: “istio-component-monitor” namespace: “istio-system” from “istio-monitoring/ServiceMonitor.yaml”: no matches for kind
Also for Istio injectio part:
we have already added this as part of all the install.sh scripts to enable or disable istio injection as and when required. So no need to make any seperate changes for this.
example: mosip-infra/install.sh at 1.2.0.1 · mosip/mosip-infra · GitHub

rcsampang · August 17, 2022, 4:45am

@ckm007 Thank you for the clarifications.

I deleted and recreated my MOSIP clusters and followed the instructions as per 1.2.0.1 branch. Happy to say that installation up to external modules went fine without any errors. Did not encounter any more errors in Istio and was able to install Prometheus successfully.

The only issue I am having right now is that I have run out of resources halfway through the installation of MOSIP services. The culprit is that the resources I requested was based on the Master branch which has lower specifications than what is needed as per 1.2.0.1 branch. Thus, it wasn’t enough to run MOSIP services. The good news is that it looks like the services are installing fine although it has run out of resources to continue the installation of other services.

I have already requested for upgrades and once approved I will do the installation again. Once it is running I would gladly report it.

Thank you for all the kind understanding, assistance, and continuous support.

Topic		Replies	Views
V3 - install Rancher using helm - TLS external Platform	12	528	June 3, 2022
Osip 1.2.0 deployment issues, problems with not using WireGuard, and errors during Istio installation Developer	7	485	July 28, 2023
Mosip VMs for observation cluster and mosip cluster Developer	2	217	November 20, 2023
Failed to install MOSIP Support	21	757	October 30, 2023
Import mosip cluster freeze at Pending state Support	7	264	August 23, 2023

MOSIP cluster - NGINX VMs, DNS Requirements, Istio ...using RKE/Rancher

Istio injection

Related topics