Installing Registration Processor (version 3 branch 1.2.0.1)

Support
1

Hi! I encountered issues in installing Registration Processor
While following the instruction :
cd topic
./create_topics.sh
there were dependency issues:

rke@mosipv3-vm2:~/mosip-infra/deployment/v3/mosip/regproc/topic$ ./create_topics.sh
Enter IAM username: admin
Creating topics
Traceback (most recent call last):
File “create_topics.py”, line 7, in
import pandas as pd
ModuleNotFoundError: No module named ‘pandas’

i installed pandas sudo apt install python3-pandas then tried again
another dependency issue
ModuleNotFoundError: No module named ‘sqlalchemy’

i installed pyhton 3 sqlalchemy since pandas is using python 3
sudo apt install python3-pip
pip3 install SQLAlchemy

too many errors so I installed python2 version of sqlalchemy and removed python3 version
sudo apt-get install sqlalchemy

another dependency error among others again
ModuleNotFoundError: No module named ‘psycopg2’

I installed pip install psycopg2-binary

still too may errors:
The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File “create_topics.py”, line 36, in
main()
File “create_topics.py”, line 33, in main
upload_xlsx(args.xls, args.user, args.db_user, args.db_pwd, args.db_host, args.db_port)
File “create_topics.py”, line 17, in upload_xlsx
df.to_sql(‘topic’, engine, index=False, if_exists=‘append’)
File “/usr/lib/python3/dist-packages/pandas/core/generic.py”, line 2702, in to_sql
sql.to_sql(
File “/usr/lib/python3/dist-packages/pandas/io/sql.py”, line 509, in to_sql
pandas_sql.to_sql(
File “/usr/lib/python3/dist-packages/pandas/io/sql.py”, line 1319, in to_sql
table.create()
File “/usr/lib/python3/dist-packages/pandas/io/sql.py”, line 641, in create
if self.exists():
File “/usr/lib/python3/dist-packages/pandas/io/sql.py”, line 628, in exists
return self.pd_sql.has_table(self.name, self.schema)
File “/usr/lib/python3/dist-packages/pandas/io/sql.py”, line 1343, in has_table
return self.connectable.run_callable(
File “”, line 2, in run_callable
File “/home/rke/.local/lib/python3.8/site-packages/sqlalchemy/util/deprecations.py”, line 402, in warned
return fn(*args, **kwargs)
File “/home/rke/.local/lib/python3.8/site-packages/sqlalchemy/engine/base.py”, line 3223, in run_callable
with self.connect() as conn:
File “/home/rke/.local/lib/python3.8/site-packages/sqlalchemy/engine/base.py”, line 3315, in connect
return self._connection_cls(self, close_with_result=close_with_result)
File “/home/rke/.local/lib/python3.8/site-packages/sqlalchemy/engine/base.py”, line 96, in init
else engine.raw_connection()
File “/home/rke/.local/lib/python3.8/site-packages/sqlalchemy/engine/base.py”, line 3394, in raw_connection
return self._wrap_pool_connect(self.pool.connect, _connection)
File “/home/rke/.local/lib/python3.8/site-packages/sqlalchemy/engine/base.py”, line 3364, in _wrap_pool_connect
Connection.handle_dbapi_exception_noconnection(
File “/home/rke/.local/lib/python3.8/site-packages/sqlalchemy/engine/base.py”, line 2198, in handle_dbapi_exception_noconnection
util.raise(
File “/home/rke/.local/lib/python3.8/site-packages/sqlalchemy/util/compat.py”, line 208, in raise
raise exception
File “/home/rke/.local/lib/python3.8/site-packages/sqlalchemy/engine/base.py”, line 3361, in _wrap_pool_connect
return fn()
File “/home/rke/.local/lib/python3.8/site-packages/sqlalchemy/pool/base.py”, line 320, in connect
return _ConnectionFairy._checkout(self)
File “/home/rke/.local/lib/python3.8/site-packages/sqlalchemy/pool/base.py”, line 884, in _checkout
fairy = _ConnectionRecord.checkout(pool)
File “/home/rke/.local/lib/python3.8/site-packages/sqlalchemy/pool/base.py”, line 486, in checkout
rec = pool._do_get()
File “/home/rke/.local/lib/python3.8/site-packages/sqlalchemy/pool/impl.py”, line 146, in do_get
self.dec_overflow()
File “/home/rke/.local/lib/python3.8/site-packages/sqlalchemy/util/langhelpers.py”, line 70, in exit
compat.raise(
File “/home/rke/.local/lib/python3.8/site-packages/sqlalchemy/util/compat.py”, line 208, in raise
raise exception
File “/home/rke/.local/lib/python3.8/site-packages/sqlalchemy/pool/impl.py”, line 143, in _do_get
return self._create_connection()
File “/home/rke/.local/lib/python3.8/site-packages/sqlalchemy/pool/base.py”, line 266, in _create_connection
return _ConnectionRecord(self)
File “/home/rke/.local/lib/python3.8/site-packages/sqlalchemy/pool/base.py”, line 381, in init
self.__connect()
File “/home/rke/.local/lib/python3.8/site-packages/sqlalchemy/pool/base.py”, line 678, in connect
pool.logger.debug(“Error on connect(): %s”, e)
File “/home/rke/.local/lib/python3.8/site-packages/sqlalchemy/util/langhelpers.py”, line 70, in exit
compat.raise(
File “/home/rke/.local/lib/python3.8/site-packages/sqlalchemy/util/compat.py”, line 208, in raise
raise exception
File “/home/rke/.local/lib/python3.8/site-packages/sqlalchemy/pool/base.py”, line 673, in __connect
self.dbapi_connection = connection = pool._invoke_creator(self)
File “/home/rke/.local/lib/python3.8/site-packages/sqlalchemy/engine/create.py”, line 578, in connect
return dialect.connect(*cargs, **cparams)
File “/home/rke/.local/lib/python3.8/site-packages/sqlalchemy/engine/default.py”, line 598, in connect
return self.dbapi.connect(*cargs, **cparams)
File “/home/rke/.local/lib/python3.8/site-packages/psycopg2/init.py”, line 122, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
sqlalchemy.exc.OperationalError: (psycopg2.OperationalError) could not translate host name “api-internal.x.x.x” to address: System error

In short to avoid these errors, how do I install the dependencies for the create_topic.sh ?

4

Even though I was able to install the dependencies, there is still an error = (sqlalchemy.exc.OperationalError: (psycopg2.OperationalError) could not translate host name
Also i am now doubting my initial assumption that being unable to run the create_topics.sh script is the cause for the errors in regproc_group1 and regproc group6.

Looking at the logs:
regproc_group1
Oct 03, 2022 4:57:11 AM brave.Tracing$LoggingReporter report
INFO: {“traceId”:“5b110d0786dff97a”,“id”:“5b110d0786dff97a”,“kind”:“SERVER”,“name”:“get /registrationprocessor/v1/packetreceiver/actuator/health”,“timestamp”:1664773030009067,“duration”:1367884,“localEndpoint”:{“serviceName”:“unknown”,“ipv4”:“10.42.2.35”},“remoteEndpoint”:{“ipv4”:“127.0.0.6”,“port”:59145},“tags”:{“error”:“500”,“http.method”:“GET”,“http.path”:“/registrationprocessor/v1/packetreceiver/actuator/health”,"http.status_code

regproc_group6
Oct 03, 2022 4:56:46 AM brave.Tracing$LoggingReporter report
INFO: {“traceId”:“cca551a6d6870f8d”,“id”:“cca551a6d6870f8d”,“kind”:“SERVER”,“name”:“get /registrationprocessor/v1/uploader/actuator/health”,“timestamp”:1664773006654688,“duration”:27070,“localEndpoint”:{“serviceName”:“unknown”,“ipv4”:“10.42.2.33”},“remoteEndpoint”:{“ipv4”:“127.0.0.1”,“port”:41566},“tags”:{“error”:“500”,“http.method”:“GET”,“http.path”:“/registrationprocessor/v1/uploader/actuator/health”,“http.status_code”:“500”}}

Can anyone help me fix these errors? I am stuck with these errors despite of doing a delete and fresh install of the MOSIP cluster.

5

Finally had these fixed

I had to delete and re-install Kafka and ClamAV. I missed configuration changes to it before but after editing and correcting the configuration files, I also deleted and reinstalled regproc.

I updated these files from github branch 1.2.0.1
mosip-infra/deployment/v3/external/antivirus/clamav/install.sh

mosip-infra/deployment/v3/external/kafka/install.sh
*take note that changes should be made in this file on your server because there are some typos, otherwise there will be errors-
helm -n $NS install kafka-ui kafka-ui/kafka-ui -f ui-values.yaml –VERSION $CHART_VERSION
change to
helm -n $NS install kafka-ui kafka-ui/kafka-ui -f ui-values.yaml –version $CHART_VERSION
(use --version, --VERSION will not be recognized as option - Error: unknown flag: --VERSION)
also change this line
Echo Installed kafka and kafka-ui services
to this
echo Installed kafka and kafka-ui services
(Echo is not a recognized command - ./install.sh: 35: Echo: not found)

Consider this issue closed. Thank you very much.

2 Likes
6

thanks @rcsampang we have corrected the kafka ui install script:

also with latest changes in websub-consolidator we dont need create_topics script to be executed anymore, the same is handled as part of websub consolidator installation hence removing the script.

2 Likes