Build your own SCH on premise in a few steps (dev)

  • 25 November 2021
  • 0 replies
  • 151 views

Userlevel 4
Badge

Set up the environment:

 

  1. Launch MySql docker container

  docker run -d -p 3306:3306 -e MYSQL_ROOT_PASSWORD=root mysql:5.7

  2. Launch Influxdb docker container

docker run -d -p 8086:8086 -v $PWD:/var/lib/influxdb influxdb

  3 Configure Influxdb docker container. 

     Unlike MySql, Influxdb needs to be set manually after the container is started.

docker exec -it container_id bin/bash
$> influx
CREATE DATABASE dpm
CREATE DATABASE dpm_app
CREATE USER root WITH PASSWORD 'root'
grant ALL on dpm to root
grant ALL on dpm_app to root
grant read on _internal to root

show databases;
_internal
dpm
dpm_app

 

  • Set environment variables

              export DPM_CONF=/absolute_path/dpm_home/etc

              export DPM_DIST=/absolute_path/dpm_home

  • Download MySQL JDBC driver:

               Place the JDBC driver for MySQL in DPM_DIST/extra-lib directory

  • Update files in the DPM_CONF directory

                cd $DPM_CONF and change the following files:

                1. timesseries-app.properties file

# Time Series Data Store Details
19 # Example influx db url: http://influxhost:8500
20 db.url=http://localhost:8500
21 db.name=dpm
22 db.user=root
23 db.password=root
24 db.retentionPolicy=autogen
25
26 # Example influx app db url: http://influxhost:8500
27 dpm.app.db.url=http://localhost:8500
28 dpm.app.db.name=dpm_app
29 dpm.app.db.user=root
30 dpm.app.db.password=root
31 dpm.app.db.retentionPolicy=autogen

                2. common-to-all-apps.properties.

6 # URL used for login
7 # If proxy is used, should point to outside of proxy
8 dpm.base.url=http://localhost:18631

 

Now we will need to change all app.properties files with the Database parameters, we can use the following script to modify all at once:

#!/bin/bash
#This script has to be placed under DPM_DIST/etc directory!

BASEDIR="$PWD"
for file in $BASEDIR/*-app.properties
do
sed -i.backup "s/db.openjpa.ConnectionURL=/db.openjpa.ConnectionURL=jdbc:mysql:\/\/localhost:3306\/$(basename "${file%-app*}")/g" "$file"
sed -i.backup "s/db.openjpa.ConnectionUserName=/db.openjpa.ConnectionUserName=root/g" "$file"
sed -i.backup "s/db.openjpa.ConnectionPassword=/db.openjpa.ConnectionPassword=root/g" "$file"
done
  • Run the 99-fullmysqlinit.sh

           #cd $DPM_DIST/dev

           #./99mysqlinit.sh 

  • Once it is finished run dpm:

           #./$DPM_DIST/bin/streamsets dpm

 

Building the SCH:

 

1. Clone the DPM project from StreamSets repository in Github. 

git clone http://github.com/streamsets/domainserver 
cd domainserver

2. Compile

./gradlew installDist -PfilterFile=dev-mysql

3. Set up an environment variable for convenience.

pushd build/install/streamsets-dpm* && export DPM_DIST=$(pwd) && popd

4. Execute a convenience script to run all the necessary steps for MySQL.

$DPM_DIST/dev/99-fullmysqlinit.sh

 


This topic has been closed for comments