From 494a639c24cde7d04b65b33b27945e34d5e0ad07 Mon Sep 17 00:00:00 2001 From: Aditya Anil Poduval Date: Wed, 1 Dec 2021 14:06:20 +0530 Subject: [PATCH 1/6] Updated README. --- fluentd/README.md | 102 ++++++++++++++++++++++++++++++++++++++++++++++ fluentd/ReadMe | 1 - 2 files changed, 102 insertions(+), 1 deletion(-) create mode 100644 fluentd/README.md delete mode 100644 fluentd/ReadMe diff --git a/fluentd/README.md b/fluentd/README.md new file mode 100644 index 0000000..bdef176 --- /dev/null +++ b/fluentd/README.md @@ -0,0 +1,102 @@ + +# Fluentd Conainter deployment for TCI Hybrid Applications + +For the TCI Hybrid applications the Fluentd container works as a sidecar +for the On-prem BWCE Applications to write the Process Monitoring data +into the Database. + + +## Configuring Fluentd connection to PostgreSQL Database + +To establish the connection between the Fluentd and PostgreSQL Database, update the [fluentd.conf](./PostgreSQL/config/fluentd.conf) file located inside the PostgreSQL/config folder. + +In this file the following lines need to be updated with the PostgreSQL Database details:- +- **host** :- Update the Line #12 in the file with the host address of the Machine with the Database +- **port** :- Update the Line #13 in the file with the Database port (For PostgreSQL the default port is 5432) +- **database** :- Update the Line #14 in the file with the Database name. +- **username** :- Update the Line #15 in the file with the 'username', with which the connection can be established. +- **password** :- Update the Line #16 in the file with the 'password' for the 'username' provided in Line #10. + +There are some additional configuring that the user can use:- +- **port** :- The Line #3 in the fluentd.conf file contains the port number for establishing connection with the Fluentd.(By default th port '24224' will be used.) + + +## Configuring Fluentd connection to MySQL Database + +To establish the connection between the Fluentd and MySQL Database, update the [fluentd.conf](./MySQL/config/fluentd.conf) file located inside the MySQL/config folder. + +In this file the following lines need to be updated with the MySQL Database details:- +- **host** :- Update the Line #13 in the file with the host address of the Machine with the Database +- **port** :- Update the Line #14 in the file with the Database port (For MySQL the default port is 3306) +- **database** :- Update the Line #15 in the file with the Database name. +- **username** :- Update the Line #16 in the file with the 'username', with which the connection can be established. +- **password** :- Update the Line #17 in the file with the 'password' for the 'username' provided in Line #10. + +There are some additional configuring that the user can use:- +- **port** :- The Line #3 in the fluentd.conf file contains the port number for establishing connection with the Fluentd.(By default th port '24224' will be used.) + + +## Configuring Fluentd connection to Microsoft SQL Server Database + +To establish the connection between the Fluentd and Microsoft SQL Server Database, update the [fluentd.conf](./MSSQL/config/fluentd.conf) file located inside the MSSQL/config folder. + +In this file the following lines need to be updated with the MSSQL Database details:- +- **host** :- Update the Line #13 in the file with the host address of the Machine with the Database +- **port** :- Update the Line #14 in the file with the Database port (For MSSQL the default port is 1433) +- **database** :- Update the Line #15 in the file with the Database name. +- **username** :- Update the Line #16 in the file with the 'username', with which the connection can be established. +- **password** :- Update the Line #17 in the file with the 'password' for the 'username' provided in Line #10. + +There are some additional configuring that the user can use:- +- **port** :- The Line #3 in the fluentd.conf file contains the port number for establishing connection with the Fluentd.(By default th port '24224' will be used.) + + +## Configuring Fluentd connection to Oracle Database + +To establish the connection between the Fluentd and Oracle Database, update the [fluentd.conf](./Oracle/config/fluentd.conf) file located inside the Oracle/config folder. + +In this file the following lines need to be updated with the MSSQL Database details:- +- **host** :- Update the Line #21 in the file with the host address of the Machine with the Database +- **port** :- Update the Line #22 in the file with the Database port (For Oracle DB the default port is 1521) +- **database** :- Update the Line #23 in the file with the "Service Name/SID". +- **username** :- Update the Line #24 in the file with the 'username', with which the connection can be established. +- **password** :- Update the Line #25 in the file with the 'password' for the 'username' provided in Line #10. + +There are some additional configuring that the user can use:- +- **port** :- The Line #3 in the fluentd.conf file contains the port number for establishing connection with the Fluentd.(By default th port '24224' will be used.) + +**Note**:- As there is a [issue](https://github.com/rsim/oracle-enhanced/issues/2211) related to data insertion in Oracle DB using fluentd sql plugin for CLOB datatype. We are using varchar2 data type with a size limit of 4000 bytes. This size can be extended to 32767 bytes by performing the steps mentioned in the following [document](https://docs.oracle.com/en/database/oracle/oracle-database/18/spmsu/enabling-the-new-extended-data-type-capability.html#GUID-88FB7FFD-4392-49C6-843A-45B49F8A1821). **The steps mentioned is irreversible.** + - If the steps mentioned above are being used then the user can update the create table script for ActivityLoggingStats table with the enhanced size. + - The user will have to comment Line #9 and #10 and un-comment Line #11 and #12. +## Steps to Build Fluentd Image + +Steps to build the Fluentd Docker image:- +- Step #1:- Configure the fluentd.conf file with the steps mentioned above, depending on the type of database in use. +- Step #2:- This Step is only for Oracle database. + - Depending on the version of Oracle DB in use download the following files from the [Oracle Website](https://www.oracle.com/sg/database/technologies/instant-client/linux-x86-64-downloads.html) + - instantclient-basic-linux.x64 + - instantclient-sdk-linux.x64 + - instantclient-sqlplus-linux.x64 + - Place these files in the Oracle/instantclient folder. + - Update the Line #16 with the file name for instantclient-basic-linux.x64 zip file. + - Update the Line #24 with the file name for instantclient-sqlplus-linux.x64 zip file. + - Update the Line #32 with the file name for instantclient-sdk-linux.x64 zip file. +- To build the fluentd docker image use the following docker command. +```bash + docker build -t fluentd . +``` + + + +## Steps to run Fluentd Image + +To run fluentd container the user can use the following command. + +``` + docker run -p 24224:24224 -e FLUENTD_CONF=fluentd.conf fluent +``` + +The port number present in the Line #3 of the fluentd.conf file must be published during execution and the same port must also be provided to the BWCE Application. By default the port number 24224 is used. + +### Environment Variables + - **FLUENTD_CONF**:- Name of the fluentd config file present as per the database in use. \ No newline at end of file diff --git a/fluentd/ReadMe b/fluentd/ReadMe deleted file mode 100644 index d0b28a0..0000000 --- a/fluentd/ReadMe +++ /dev/null @@ -1 +0,0 @@ -Repository to provide fluentd config scripts for monitoring Hybrid Apps in TCI-HybridMon From 63f34f95a79ef3614ea75da20d8e491579541c41 Mon Sep 17 00:00:00 2001 From: Aditya Anil Poduval Date: Wed, 1 Dec 2021 14:10:03 +0530 Subject: [PATCH 2/6] Updated the file names for Create table SQL files. --- fluentd/MSSQL/{mssql_pm_create.sql => mssql_eh_create.sql} | 0 fluentd/MySQL/{mysql_pm_create.sql => mysql_eh_create.sql} | 0 fluentd/Oracle/{oracle_pm_create.sql => oracle_eh_create.sql} | 0 .../{postgresql_pm_create.sql => postgresql_eh_create.sql} | 0 4 files changed, 0 insertions(+), 0 deletions(-) rename fluentd/MSSQL/{mssql_pm_create.sql => mssql_eh_create.sql} (100%) rename fluentd/MySQL/{mysql_pm_create.sql => mysql_eh_create.sql} (100%) rename fluentd/Oracle/{oracle_pm_create.sql => oracle_eh_create.sql} (100%) rename fluentd/PostgreSQL/{postgresql_pm_create.sql => postgresql_eh_create.sql} (100%) diff --git a/fluentd/MSSQL/mssql_pm_create.sql b/fluentd/MSSQL/mssql_eh_create.sql similarity index 100% rename from fluentd/MSSQL/mssql_pm_create.sql rename to fluentd/MSSQL/mssql_eh_create.sql diff --git a/fluentd/MySQL/mysql_pm_create.sql b/fluentd/MySQL/mysql_eh_create.sql similarity index 100% rename from fluentd/MySQL/mysql_pm_create.sql rename to fluentd/MySQL/mysql_eh_create.sql diff --git a/fluentd/Oracle/oracle_pm_create.sql b/fluentd/Oracle/oracle_eh_create.sql similarity index 100% rename from fluentd/Oracle/oracle_pm_create.sql rename to fluentd/Oracle/oracle_eh_create.sql diff --git a/fluentd/PostgreSQL/postgresql_pm_create.sql b/fluentd/PostgreSQL/postgresql_eh_create.sql similarity index 100% rename from fluentd/PostgreSQL/postgresql_pm_create.sql rename to fluentd/PostgreSQL/postgresql_eh_create.sql From 04e512d3bb006390d8df8267c845cb290cac01f7 Mon Sep 17 00:00:00 2001 From: Aditya Anil Poduval Date: Tue, 4 Oct 2022 14:11:08 +0530 Subject: [PATCH 3/6] Updated the MySQL Dockerfile for Fluentd, added version specific libraries to avoide issues. --- fluentd/MySQL/Dockerfile | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/fluentd/MySQL/Dockerfile b/fluentd/MySQL/Dockerfile index 2aa272c..1e9ad6a 100644 --- a/fluentd/MySQL/Dockerfile +++ b/fluentd/MySQL/Dockerfile @@ -11,8 +11,8 @@ RUN apt-get install -y make RUN apt-get install -y libpq-dev RUN gem -v RUN ruby -v -RUN ["gem", "install", "fluent-plugin-sql", "--no-document"] +RUN ["gem", "install", "fluent-plugin-sql", "--no-document", "-v", "2.3.0"] RUN apt-get install -y default-libmysqlclient-dev RUN apt-get install -y libmariadb-dev -RUN ["gem", "install", "mysql2"] +RUN ["gem", "install", "mysql2", "-v", "0.5.4"] COPY config /fluentd/etc From 871a1b2c3ea0ffc932bccba877a8beec9913a338 Mon Sep 17 00:00:00 2001 From: Aditya Anil Poduval Date: Tue, 4 Oct 2022 14:22:06 +0530 Subject: [PATCH 4/6] Updated the PostgreSQL Dockerfile for Fluentd, added version specific libraries to avoide issues. --- fluentd/PostgreSQL/Dockerfile | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/fluentd/PostgreSQL/Dockerfile b/fluentd/PostgreSQL/Dockerfile index 0c975b4..8295aa6 100644 --- a/fluentd/PostgreSQL/Dockerfile +++ b/fluentd/PostgreSQL/Dockerfile @@ -11,7 +11,7 @@ RUN apt-get install -y make RUN apt-get install -y libpq-dev RUN gem -v RUN ruby -v -RUN ["gem", "install", "fluent-plugin-sql", "--no-document"] +RUN ["gem", "install", "fluent-plugin-sql", "--no-document", "-v", "2.3.0"] RUN ["gem", "install", "pg", "-v", "1.1"] -RUN ["gem", "install", "fluent-plugin-postgres"] +RUN ["gem", "install", "fluent-plugin-postgres", "-v", "0.1.0"] COPY config /fluentd/etc From 986ea696a7ca9cb7d2eded7dd62e50b3c20e5d2b Mon Sep 17 00:00:00 2001 From: Aditya Anil Poduval Date: Tue, 4 Oct 2022 15:28:41 +0530 Subject: [PATCH 5/6] Updated the MSSQL Dockerfile for Fluentd, added version specific libraries to avoide issues. --- fluentd/MSSQL/Dockerfile | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/fluentd/MSSQL/Dockerfile b/fluentd/MSSQL/Dockerfile index ebe7787..cd720f9 100644 --- a/fluentd/MSSQL/Dockerfile +++ b/fluentd/MSSQL/Dockerfile @@ -12,7 +12,7 @@ RUN apt-get install -y libpq-dev RUN apt-get install -y freetds-dev RUN gem -v RUN ruby -v -RUN ["gem", "install", "fluent-plugin-sql", "--no-document"] -RUN ["gem", "install", "tiny_tds"] -RUN ["gem", "install", "activerecord-sqlserver-adapter"] +RUN ["gem", "install", "fluent-plugin-sql", "--no-document", "-v", "2.3.0"] +RUN ["gem", "install", "tiny_tds", "-v", "2.1.5"] +RUN ["gem", "install", "activerecord-sqlserver-adapter", "-v", "6.1.2.1"] COPY config /fluentd/etc From 43fa10320a54a50cdccf0774e807bdf24eb94706 Mon Sep 17 00:00:00 2001 From: Aditya Anil Poduval Date: Tue, 4 Oct 2022 16:51:02 +0530 Subject: [PATCH 6/6] Updated the Oracle Dockerfile for Fluentd, added version specific libraries to avoide issues. --- fluentd/Oracle/Dockerfile | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/fluentd/Oracle/Dockerfile b/fluentd/Oracle/Dockerfile index 0536785..520830d 100644 --- a/fluentd/Oracle/Dockerfile +++ b/fluentd/Oracle/Dockerfile @@ -39,7 +39,7 @@ RUN LIBS="*/*" && \ ENV LD_LIBRARY_PATH=/usr/lib RUN export LD_BIND_NOW=1 -RUN ["gem", "install", "ruby-oci8"] -RUN ["gem", "install", "activerecord-oracle_enhanced-adapter"] -RUN ["gem", "install", "fluent-plugin-sql", "--no-document"] +RUN ["gem", "install", "ruby-oci8", "-v", "2.2.11"] +RUN ["gem", "install", "activerecord-oracle_enhanced-adapter","-v","6.1.5"] +RUN ["gem", "install", "fluent-plugin-sql", "--no-document", "-v", "2.3.0"] COPY config /fluentd/etc