Reporting System Installation

Installation Steps

The high-level steps for installing and configuring N2ACD reporting node is:

1. Determine the server(s) that will supply the processing node for reporting, bearing in mind the supported operating systems and minimum server requirements.
2. Install the NiFi and Superset packages.
3. Perform required post-installation steps to configure the reporting components.
4. Install and configure MiNiFi packages on service nodes. See here.

OS-Specific Setup

Refer to the specific Red Hat or Debian instructions for any pre-requisites as required. Note that the EPEL repository is required for the reporting server and so the EPEL release package must be installed.

Apache NiFi Installation

The ACD reporting data pipeline is built on Apache NiFi and consists of three core data processing functions:

• The export of the ACD primary database into timestamped point-in-time snapshots of data in the reporting database.
• The copy of EDR files from service nodes to the reporting node for future processing.
• The stream processing of ACD EDRs to extract freephone call activities, storing aggregate information in the reporting database.

Apache NiFi is distributed by the Apache Foundation as a .tar.gz file. N-Squared provides an RPM wrapper to install this which is also preconfigured with a N2ACD-appropriate configuration.

Install Apache NiFi from the N-Squared repository. Execute the instructions specific to your operating system:

Warning: The package installation will shut down Apache NiFi if it is running. It will require manually restarting.

The Apache NiFi installation using the wrapper package will:

1. Install Apache NiFi in a version-specific subdirectory of /opt/nifi.
2. Create and link Apache NiFi to /opt/nifi/latest, installing the base configuraiton.
3. Configure Apache NiFi for execution via systemd as the service nifi if it is not already.
4. Configure the Apache HTTP server to proxy access to the Apache NiFi frontend GUI.
5. Configure a default EDR file management location /opt/nifi/edr.

Apache Nifi Configuration

Apache NiFi’s configuration must be set up with the correct DNS in the nifi.properties file. This is for MiNiFi to correctly connect to NiFi.

The base configuration distributed with NiFi is designed to set up NiFi with insecure (HTTP) connectivity. An insecure setup is significantly less complex to achieve and is a valuable step to perform prior to the secure setup discussed below.

An initial installation of NiFi will set up the service with most configuration correctly defined for an N2ACD NiFi deployment. However due to the need to use correct DNS information, several configuration options must be set for the environment installed.

Edit /opt/nifi/latest/conf/nifi.properties, and ensure the following section is configured consistently:

# Site to Site properties
nifi.remote.input.host=n2-p-acd-rep-01.nsquared.co.nz
nifi.remote.input.secure=false
nifi.remote.input.socket.port=
nifi.remote.input.http.enabled=true
nifi.remote.input.http.transaction.ttl=30 sec
nifi.remote.contents.cache.expiration=30 secs

# web properties #
#############################################

# For security, NiFi will present the UI on 127.0.0.1 and only be accessible through this loopback interface.
# Be aware that changing these properties may affect how your instance can be accessed without any restriction.
# We recommend configuring HTTPS instead. The administrators guide provides instructions on how to do this.

nifi.web.http.host=n2-reporting-01.nsquared.co.nz
nifi.web.http.port=8080
nifi.web.http.network.interface.default=

#############################################
# nifi can't listen on https and http
# if you don't run as https, then you don't get generated keystore/truststore
# which makes installation harder, so we listen over https and proxy through to it
nifi.web.https.host=
nifi.web.https.port=
nifi.web.https.network.interface.default=

Note that HTTP is used for NiFi in this initial configuration. To ensure NiFi starts up without HTTPS certificates, the following are commented out:

#nifi.security.keystore=/opt/nifi/data/conf/keystore.p12
#nifi.security.keystoreType=PKCS12
#nifi.security.keystorePasswd=
#nifi.security.keyPasswd=
#nifi.security.truststore=/opt/nifi/data/conf/truststore.p12
#nifi.security.truststoreType=PKCS12
#nifi.security.truststorePasswd=

The following configuration options should be reconfigured as appropriate:

All other configuration options can be left as the shipped defaults.

Firewall Access

For MiNiFi access, and for the reverse proxy access, NiFi’s HTTP(S) port must be allowed through the firewall. With the default configuration, port 8080 must be allowed:

firewall-cmd --add-port=8080/tcp --permanent
firewall-cmd --reload

If secure access is enabled (see the later sections of this document), this port may be changed - or kept the same but with HTTPS configured.

Enabling Apache Nifi

Enable the nifi service, and start it for the first time:

systemctl enable nifi
systemctl start nifi

First Post-Startup Actions

After first startup, check that NiFi has completed setup correctly.

1. NiFi secure passwords are generated and stored back into the nifi.properties file by NiFi after first startup. The parameter nifi.sensitive.props.key is updated.

   Copy the password stored in this field and ensure the password is saved securely.

   Note that if this parameter are not updated with a password, then the likely problem is that the file is not owned by n2in (the user that runs NiFi)

2. Capture the NiFi default username and password. This can be retrieved by running the command grep -i 'Generated ' /opt/nifi/latest/logs/nifi-app*. The username and password for the admin user can be reset using the command:

   /opt/nifi/latest/bin/nifi.sh set-single-user-credentials <username> <password>

The NiFi configuration can be backed up by storing the flow.json.gz and flow.xml.gz files that are stored in the /opt/nifi/data/conf directory. Note that NiFi itself backs up files to the /opt/nifi/data/conf/archive directory.

Apache Superset

Apache Superset is a data visualisation platform for creating dashboards and generating and downloading tabular data. With Superset, N2ACD reporting data can be accessed.

Note that reports available to N2ACD users via the N2ACD GUI are built from reporting data directly from the database, and not from the Apache Superset.

Apache Superset is distributes as a Python package and requires pip for installation, which in turn requires a general internet connection. N-Squared provides a prepackaged distribution of Apache Superset so that a general internet connection is not required.

Install Apache Superset from the N-Squared repository. Execute the instructions specific to your operating system:

Warning: The package installation will shut down Apache Superset if it is running. It will require manually restarting.

The Apache Superset installation using the wrapper package will:

1. Install a pre-build Python virtual environment in /opt/superset/ that includes Apache Superset and all necessary python dependencies.
2. Configure Apache Superset for execution via systemd as the service superset.
3. Configure the Apache HTTP server to proxy access to Apache Superset via port 80.

After the first installation of Superset it is important to create the administration account and initialise the Superset database. On the installed server, execute the following commands, starting as the root user:

su - n2in
cd /opt/superset && . venv/bin/activate
export FLASK_APP=superset
superset fab create-admin
superset init

Once this is done the first time, the normal systemctl service can be used to manage the running superset instance.

Post-Installation Steps

HTTP(S) Reverse Proxy Configuration

The HTTP(S) reverse proxy configuration for Apache NiFi must be configured for each node installed. As NiFi must be configured to run on the reporting node’s correct host IP for connectivity from NiFi, the installed default reverse proxy must be updated.

Edit the proxy configuration:

vi /etc/httpd/conf.d/nifi.conf

The configuration:

<LocationMatch "^/nifi(.*)$">
    ProxyPassMatch "http://127.0.0.1:8080/nifi$1"
</LocationMatch>

Must be changed to reference the correct hostname configured in the nifi.properties file for the nifi.web.http.host (or if secured, the nifi.web.https.host) configuration options.

Then restart Apache httpd:

systemctl restart httpd

Starting Systems

After verifying the configuration of Apache NiFi, start the service as root:

systemctl start nifi

After verifying the configuration of Apache Superset, start the service as root:

systemctl start superset

GUI Access

Apache NiFi is configured through it’s GUI design tool, a web-based stream processing GUI with a visual design tool used for both building and monitoring the stream processing engine.

Access is proxied via HTTP in the default configuration, and HTTPS once secured. Ensure that the firewall ports are opened. Firstly for HTTP and HTTPS:

firewall-cmd --add-port=80/tcp --permanent
firewall-cmd --add-port=443/tcp --permanent
firewall-cmd --reload

Note that once HTTPS is configured, port 80 can be removed and firewalled off using:

firewall-cmd --remove-port=80/tcp --permanent
firewall-cmd --reload

To access the GUI services, access the host with the following URLs:

• Access to Apache NiFi is at http://n2-reporting-01.nsquared.co.nz/nifi/
• Access to Apache Superset is at http://n2-reporting-01.nsquared.co.nz/

Where n2-reporting-01.nsquared.co.nz is the hostname or IP of the installed reporting server.

EDR Streaming

ACD EDRs are expected to be streamed from service nodes to the reporting platform. N2ACD provides a pre-built MiNiFi configuration for EDR streaming, however EDRs can be loaded on to the platform in any practical manner.

EDRs for processing can be placed in any directory, however the default configured directory is /opt/nifi/edr/received.

EDRs are moved from this directory to opt/nifi/edr/input by a daemon process called moveAndCopyOnWriteComplete which monitors for new EDR files and moves them between the received and input directory. The moveAndCopyOnWriteComplete daemon is installed as a systemd service by the n2nifi-wrapper package installation. The service script /usr/lib/systemd/system/moveAndCopyOnWriteComplete.service itself determines the source and destination directories, and file suffix to monitor.

To change the configuration if this is not using the right EDR directories, edit the installed file and then reload the systemd configuration:

systemctl edit moveAndCopyOnWriteComplete.service --full

Only files ending in .edr are moved between these directories. Note that the default configuration:

/opt/nifi/install/bin/moveAndCopyOnWriteComplete.sh '^[^.].*\.edr' /opt/nifi/edr/received /opt/nifi/edr/input /opt/nifi/edr/backup 

Excludes files starting with a dot. This is crucial to avoid in-progress written files which are being written by the NiFi process that is receiving files from service nodes.

It is also important, if you want to back up files received and not have them processed without backups (because NiFi reads then deletes files), to configure moveAndCopyOnWriteComplete to use the backup directory. The command should be changed to:

/opt/nifi/install/bin/moveAndCopyOnWriteComplete.sh '^[^.].*\.edr' /opt/nifi/edr/received /opt/nifi/edr/input /opt/nifi/edr/backup 

This script must also be enabled. Ensure it is run on startup:

systemctl enable moveAndCopyOnWriteComplete.service
systemctl start moveAndCopyOnWriteComplete.service

File Monitoring

ACD EDRs are moved and processed through directories in /opt/nifi/edr automatically by Apache NiFi and moveAndCopyOnWriteComplete. This process will be automated and extremely performant in general.

A daemon process called monitorAndAuditFileChanges is installed as part of the n2nifi-wrapper package. This daemon process monitors changes to files in directories under /opt/nifi/edr and writes a log of all file changes to /var/log/nifi/monitorAndAuditFileChanges.log

For viewing how files are being processed by the reporting system, tail this file:

tail -f /var/log/nifi/monitorAndAuditFileChanges.log

This log file is further managed by the Linux logrotate daemon. To configure this script to monitor all the correct directories, edit and change if required:

systemctl edit monitorAndAuditFileChanges.service --full

This script must also be enabled. Ensure it is run on startup:

systemctl enable monitorAndAuditFileChanges.service
systemctl start monitorAndAuditFileChanges.service

Securing NiFi

Securing NiFi requires firstly the generation of an (intermediate) CA certificate for NiFi to use to self-generate server certificates and to verify client certificates. The following guide is only one approach to this configuration, however other configurations (e.g. using a separate CA and providing an intermediate CA for NiFi to use) are out of scope of this guide.

This section of the ACD installation guide documents how to:

• Generate a self signed CA for NiFi to use for communication between NiFi and MiNiFi instances.
• Generate client certificates for MiNiFi to use when communicating with NiFi
• Configure NiFi to accept MiNiFi connections without passphrases where a client certificate is available.
• Configure NiFi for user access via the installed Apache HTTPD proxy over HTTPS with (a) an organisation-provided certificate (i.e. not self-signed) with username/password security.

For additional information on NiFI configuration, particularly security configuration, see the NiFi documentation.

TLS Certificate Framework Generation

NiFi provide a TLS certificate generation tool called tls-toolkit.sh. Download the NiFi toolkit and unzip the downloaded package into a Windows or Linux based environment with Java.

Run the tls-toolkit.sh (or .bat) command. For the linux .sh script:

cd nifi-toolkit-1.22.0/bin
export NIFIHOST=n2-reporting-01.nsquared.co.nz
mkdir $NIFIHOST
./tls-toolkit.sh standalone -d 7300 -n "$NIFIHOST" -C "CN=minifi, OU=NIFI" -o $NIFIHOST

Where:

• -d 7300 is 20 years. Set this as required based on certificate cycling requirements of the organisation, and expected certificate renewal support processes.
• -n $NIFIHOST is a comma-separated set of hostnames that NiFi will be reachable on.
• -C "CN=minifi, OU=NIFI" is the distinguished name (DN) to use in the client certificate for MiNiFi. If multiple MiNiFi clients are being configured, and each requires a unique client certificate, the -C option can be specified multiple times. It is possible to configure a client certificate for the Apache http reverse host as well using this method (giving it a DN different to the MiNiFi DN).
• -o $NIFIHOST is the output directory to generate files to. This can be any directory name.

Note that the toolkit version will likely be different to the example above. Ensure that the NIFIHOST environment is correctly defined for the hostname of NiFi for the environment being installed.

The TSL files will be generated into the directory $NIFIHOST. These will be installed into /opt/nifi/data/ssl on the NiFi processing server.

Note that tls-toolkit.sh has many options which can be used in more complex scenarios - such as multiple client certificates, multiple hostnames (requiring subject alternate name configuration). Additionally, underlying tools can be used to generate the files directly.

Using tls-toolkit.sh, the following files will be created:

Note that NiFi does not generate an explicit server certificate for NiFi. This is done internally by NiFi as required based on the CA.

Extract and Install Files into MiNiFi

Note that if MiNiFi is not yet installed on service nodes, follow the MiNiFi installation guide first, and then return to this section for completing the secure configuration.

Secure NiFi – MiNiFi connectivity requires a client certificate for MiNiFi that has been generated based on the NiFi CA. This is done when the tls-toolkit.sh script is used as described above.

Assuming tls-toolkit.sh has been run as above, generate the files for MiNiFi:

openssl pkcs12 -in $HOSTNAME/CN=minifi_OU=NIFI.p12 -out $HOSTNAME/nifi-rest.key -nocerts -nodes
openssl pkcs12 -in $HOSTNAME/CN=minifi_OU=NIFI.p12 -out $HOSTNAME/nifi-rest.crt -clcerts -nokeys

When extracting the

This generates the two files for MiNiFi - nifi-rest.key and nifi-rest.crt. A third file (generated by tls-toolkit.sh) - nifi-cert.pem is also required.

Once the client certificates are generated, copy these files to the service nodes, and place in a secure directory. E.g:

scp -p nifi-rest.key nifi-rest.crt nifi-cert.pem root@minifi-host:/opt/minifi/conf/ssl

Assuming /opt/minifi/conf/ssl exist. If it does not, create it on each service node and make sure it is only readable by root:

mkdir -p /opt/minifi/conf/ssl
chmod 700 /opt/minifi/conf/ssl

For each MiNiFi instance, edit the /opt/minifi/conf/minifi.properties file and configure the following configuration options:

nifi.remote.input.secure=true
nifi.security.need.ClientAuth=true
nifi.security.client.certificate=/opt/minifi/conf/ssl/nifi-rest.crt
nifi.security.client.private.key=/opt/minifi/conf/ssl/nifi-rest.key
nifi.security.client.pass.phrase=
nifi.security.client.ca.certificate=/opt/minifi/conf/ssl/nifi-cert.pem

Note that the passphrase is empty as the passphrase is stripped when using openssl to extract the private key.

When ready to do so (i.e. once NiFi is configured - see below), restart MiNiFi using systemctl restart minifi.

Install Files to NiFi

Install the generated TLS files into the NiFi server. On the reporting system, create the holding directory:

mkdir -p /opt/nifi/data/ssl/

Copy the generated files from tls-toolkit.sh to this directory. On the system that generated the files:

scp -p -r $HOSTNAME/* root@nifi-host:/opt/nifi/data/ssl/

Then back on the Nifi server, secure the files copied in:

chown -R n2in:daemon /opt/nifi/data/ssl/

Once the TLS files are available, configure NiFi to use them. Editing nifi.properties, update the configuration options as the following diff shows:

133c133
< nifi.remote.input.secure=false
---
> nifi.remote.input.secure=true
146,147c146,147
< nifi.web.http.host=n2-reporting-01.nsquared.co.nz
< nifi.web.http.port=8080
---
> nifi.web.http.host=
> nifi.web.http.port=
152,153c152,153
< nifi.web.https.host=
< nifi.web.https.port=
---
> nifi.web.https.host=n2-reporting-01.nsquared.co.nz
> nifi.web.https.port=8443
201a202,208
> nifi.security.keystore=/opt/nifi/data/ssl/n2-reporting-01.nsquared.co.nz/keystore.jks
> nifi.security.keystoreType=jks
> nifi.security.keystorePasswd=1P3VEBITy/ha2gkf1a7xIxqL/RDaw61Mp/Qsyl1B6p8
> nifi.security.keyPasswd=1P3VEBITy/ha2gkf1a7xIxqL/RDaw61Mp/Qsyl1B6p8
> nifi.security.truststore=/opt/nifi/data/ssl/n2-reporting-01.nsquared.co.nz/truststore.jks
> nifi.security.truststoreType=jks
> nifi.security.truststorePasswd=4q4HxdI9nhiKdHX5I1g8/jPuibNnnC812ohW2GySNbA
271c278
< nifi.cluster.protocol.is.secure=false
---
> nifi.cluster.protocol.is.secure=true
390a398,406
> #
>
> # For secure access by MiNiFi using client certificates
> nifi.rest.host=n2-reporting-01.nsquared.co.nz
> nifi.rest.keystorePath=/opt/nifi/data/ssl/CN=minifi_OU=NIFI.p12
> nifi.rest.keystorePassword=/opt/nifi/data/ssl/CN=minifi_OU=NIFI.password
> nifi.rest.keystoreType=PKCS12
> #nifi.rest.api.user.name=
> #nifi.rest.api.password=

The following configuration fields in /opt/nifi/latest/conf/nifi.properties must be updated:

Note that editing config.yaml may also be required to update the url, proxy host and proxy port fields to match the values in the MiNiFi File Push process group (which will now be using HTTPS).

Configure NiFi Authorizer Support

To allow MiNiFi to connect without a username/password, the /opt/nifi/latest/conf/authorizers.xml needs to be configured to support user logins with role definitions defined by the associated authorizations.xml file. This file defines one or more security methods to be used when secure NiFi is configured (these are ignored when using HTTP access).

Be default the following are configured:

• file-user-group-provider which defines users in a users.xml file.
• file-access-policy-provider which defines granular roles in an authorisations.xml file to grant users.
• single-user-authorizer which defines the default admin user (which can log into the GUI) and whose password is configured using /opt/nifi/latest/bin/nifi.sh set-single-user-credentials.

By default users.xml and authorisations.xml are not themselves created or configured.

To automatically generate them, edit authorizers.xml:

Locate the block:

    <!--
    <authorizer>
        <identifier>file-provider</identifier>
        <class>org.apache.nifi.authorization.FileAuthorizer</class>
        <property name="Authorizations File">./conf/authorizations.xml</property>
        <property name="Users File">./conf/users.xml</property>
        <property name="Initial Admin Identity"></property>
        <property name="Legacy Authorized Users File"></property>

        <property name="Node Identity 1"></property>
    </authorizer>
    -->

This section should be uncommented, and Initial Admin Identity configured to identify the CN and OU configured e.g.:

    <authorizer>
        <identifier>file-provider</identifier>
        <class>org.apache.nifi.authorization.FileAuthorizer</class>
        <property name="Authorizations File">./conf/authorizations.xml</property>
        <property name="Users File">./conf/users.xml</property>
        <property name="Initial Admin Identity">CN=minifi, OU=NIFI</property>
        <property name="Legacy Authorized Users File"></property>
        <property name="Node Identity 1"></property>
    </authorizer>

Restart NiFi, and the resulting filesusers.xml and authorizations.xml will be automatically created. The authorizer block can now be commented out again.

The users.xml file contains the following content e.g.:

<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<tenants>
    <groups/>
    <users>
        <user identifier="8e3c6b0e-c415-3bf3-87ff-e98c2de5ea5b" identity="CN=minifi, OU=NIFI"/>
    </users>
</tenants>

The authorizations.xml file contains the following content e.g.:

<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<authorizations>
    <policies>
        <policy identifier="1f6ae57a-08bc-11eb-9242-bf69163fde10" resource="/site-to-site" action="R">
        <!-- copy the user identifier from users.xml -->
        <user identifier="8e3c6b0e-c415-3bf3-87ff-e98c2de5ea5b"/>
        </policy>
        <policy identifier="282818e0-08bc-11eb-8508-2b51c9d70d42" resource="/site-to-site" action="W">
        <user identifier="8e3c6b0e-c415-3bf3-87ff-e98c2de5ea5b"/>
        </policy>

        <!-- copy the port identifier from Remote Process Groups/Input Ports/id in the minifi config.yml -->
        <policy identifier="f512f796-7afb-4c9f-ab68-b5eaf6d5d0cf" resource="/data-transfer/input-ports/c171f9da-689f-41e2-98c4-9d785c59c306" action="R">
        <user identifier="8e3c6b0e-c415-3bf3-87ff-e98c2de5ea5b"/>
        </policy>
        <policy identifier="b4e836ee-d526-4e16-8bf3-ee1d8fa3d5e6" resource="/data-transfer/input-ports/c171f9da-689f-41e2-98c4-9d785c59c306" action="W">
        <user identifier="8e3c6b0e-c415-3bf3-87ff-e98c2de5ea5b"/>
        </policy>
    </policies>
</authorizations>

Note the user identifier 8e3... in this example matches the users.xml file, and the c171f9da-689f-41e2-98c4-9d785c59c306 in the input-ports url matches the UUID of the input port in your NiFi’s installation.

Restart NiFi

Once reconfigured with the TLS configuration, restart NiFi:

systemctl restart nifi

Remember: update all MiNiFi configuration as previously described and restart all MiNiFi instances as well.

Test MiNiFi Connectivity

On the service nodes, verify that a request to NiFi succeeds using the client certificates previously generated and installed:

curl -k --key /opt/minifi/conf/ssl/nifi-rest.key --cert /opt/minifi/conf/ssl/nifi-rest.crt https://n2-reporting-01.nsquared.co.nz:8443/nifi-api/site-to-site

Alter the hostname as required. A request with the certificate should respond with a JSON file that describes the site-to-site configuration for the NiFi instance.

Further verify that without the client certificate, the request fails:

curl https://n2-reporting-01.nsquared.co.nz:8443/nifi-api/site-to-site

A request without a certificate should response with Unauthorized.

Reconfigure the Apache HTTPD Reverse Proxy

The default installation of the HTTP reverse proxy for NiFi is to connect to NiFi on port 8080 with HTTPS. For secure communications, this changes to HTTPS. As NiFi does not allow both HTTPS and HTTP configured at the same time, the reverse proxy must use HTTPS when communicating between Apache httpd and NiFi.

To correctly enable HTTPS, reconfigure the reverse proxy file /etc/httpd/conf.d/nifi.conf to enable the HTTPS configuration. Using the default config as a guide, the following configuration would be required:

RequestHeader set X-Forwarded-Proto "https"
RequestHeader set X-Forwarded-Port "443"
RequestHeader set X-ProxyHost "n2-reporting-01.nsquared.co.nz"
RequestHeader set X-ProxyPort "443"
RequestHeader set X-ProxyScheme "https"
SSLProxyCheckPeerCN off
SSLProxyCheckPeerName off
SSLProxyEngine on
SSLProxyVerify none
<LocationMatch "^/nifi(.*)$">
    ProxyPassMatch "https://127.0.0.1:8080/nifi$1"
</LocationMatch>

Note that the X-ProxyHost and ProxyPassMatch both must use the hostname configured in nifi.properties as the nifi.web.https.host configuration value. The ProxyPassMatch must also be configured to match nifi.web.https.port.

Once reconfigured, restart apache using service httpd restart.

Note that this configuration does not hold a client certificate for NiFi, and so the following error will occur in the apache /var/log/httpd/ssl_error_log file when the GUI is being used by a user:

AH02268: Proxy client certificate callback: (<host>:443) downstream server wanted client certificate but none are configured

This error can be removed by creating and configuring a client certificate specifically for the HTTP reverse proxy. The general steps to do this are:

1. Create a client certificate using the CA used by NiFi.
2. Define a different DN to the one used by MiNiFi
3. Load the client certificate into the client certificate p12 file used by NiFi
4. Configure the apache directives SSLProxyCACertificateFile and SSLProxyMachineCertificateFile

However detailed steps for this configuration are out of scope of this installation guide.

RAW vs HTTP Data Transfer

NiFi offers two different options for data transfer between MiNiFi and NiFi:

• RAW transfer uses a non-HTTP based protocol which may be TLS encrypted. When using RAW data transfer, the ACD NiFi installation requires three ports open:

  • A RAW data transfer port (e.g. 1026) for transfer of data only.
  • A HTTP(S) port for the web interface for NiFi. This web interface is used by MiNiFi and also for web-access by users (via the reverse proxy). If NiFi is to be installed only in a HTTP (insecure) mode, then this port can be fully proxied via apache HTTP. However HTTPS cannot be proxied for MiNiFi, and the NiFi HTTPS port must be directly accessible to MiNiFi.
  • The HTTP(S) reverse proxy port (i.e. 80 or 443) to proxy web requests through to Superset and NiFi for user access via web browsers.

• HTTP transfer uses HTTP for both data transfer and HTTP requests. When using HTTP data transfer, the ACD NiFi installation requires only two ports open:

  • A HTTP(S) port for the web interface for NiFi. This interface is used by MiNiFi and also for web-access by users. If NiFi is to be installed only in a HTTP (insecure) mode, then this port can be proxied via the reverse proxy. However HTTPS cannot be proxied for MiNiFi, and the NiFi HTTPS port must be directly accessible to MiNiFi.
  • The HTTP(S) reverse proxy port (i.e. 80 or 443) to proxy web requests through to Superset and NiFi for user access via web browsers.

It is possible, if HTTP is being used, to use only one port for all data traffic, with NiFi completely proxied behind Apache HTTPD. However in this configuration all transfer is insecure over HTTP and not HTTPS.

If using only insecure HTTP the “HTTP proxy” configuration of the remote process groups NiFi can be configured to support all access via the reverse proxy, and NiFi listens on an insecure HTTP port - e.g. port 8080.

Note that the default installation of the n2nifi-wrapper package is configured to use HTTP in this manner. It is recommended once this configuration is completely configured as working for HTTPS to be enabled as described in the previous section.

To enable RAW data transfer regardless of NiFi’s HTTPS secure configuration, follow these steps:

1. Set the configuration option nifi.remote.input.socket.port in the nifi.properties file to the desired value (e.g. 1026).
2. Add the port to the firewall rules to allow traffic from MiNiFi instances on the service nodes to this port.
3. Restart NiFi.
4. Swap the Remote Process Groups in the NiFi GUI configuration to use RAW rather than HTTP transport.