NEM Guide

NEM stands for Nativ Environment Manager.

This section introduces the technologies and processes Nativ recommend for delivering the optimal service levels from a Flex Enterprise system, while retaining the agility required to support development and testing of new functionality and configuration. Nativ uses a combination of standard Linux and bespoke tools to manage the deployment and configuration of Flex.

Components

Maintenance of systems managed with the Nativ Environment Manager (NEM) is principally done through a revision control system shared between all parties. This provides the following:

• A centralized, definitive definition: This reduces ambiguity over which is considered the latest version, or which one is desired to be live

• An audit trail: As all changes are forced to go into a central repository, and all changes are required to have change log entries, problems can be easily isolated. NEM and other required tools are installed via Linux (Ubuntu and Red Hat) packages detailed in the requirements section below.

• The Subversion revision control system is used due to its current popularity and granular access control. Most work is done in the trunk but experimental work, such as testing new web applications, branches of the NEM definition may be created.

Building out the definition held in Subversion onto the target file system is done by Apache Ant. This was chosen as it supports simple templates, and it has a reasonable developer mind share.

Upgrading from version 5.0.X to 5.1.X

Flex version 5.1.0 introduces a lot of new functionality such as comments on Metadata fields, interdependent fields, LDAP integration, Data Source timed actions, Flex Objects, major changes to the Scripting API and a lot more. Depending on the Flex functionality that you're currently using in the 5.0.x series this can be slightly more complex than upgrades between minor versions.

In order to guarantee a quick and successful upgrade, please get in touch with Nativ so that we can work with you to identify necessary changes to your configuration and come up with a migration strategy tailored to how you use Flex.

NEM Implementation

The NEM start state for a NEM system is to have no services are running and the ‘/nem/’ directory empty. The following list describes the transitions that are possible with NEM. The possible states and transitions are also shown by a diagram in the appendix section.

Download assets

• The required assets are copied locally, using bespoke NEM code calling libraries.

• Subversion, HTTP and local files are supported.

• A failure during this transition will typically be due to networking issues.

Build out

Apache Ant moves assets into place.

This transition fails if Ant exits with non-zero status.

This is the only stage of the deployment where configuration is applied.

Subsequent states do, however, read from the configuration written at this stage.

Enable

• Recursively finds service dependencies of the ‘/nem/nem.conf’ meta-service and enables them accordingly.

• Typically this is starting processes but can also enable NFS mounts, check database status, add users.

Note: As part of the Flex deployment, NEM will create a 'mio' user with a UID of 25001. This user will be the owner of the Flex process. When using network shares you'll need to ensure they are readable and writeable by this user.

Disable

• All dependencies of ‘/nem/nem.conf’ meta-service are disabled. For example: Processes are killed, and mounts, users, and Apache hosts are removed.

Destroy

• Performed as part of an upgrade, to clean ‘/nem/’ and return the system to the start state.

There has been a bug in MySQL since a separate TRIGGER privilege was introduced in version 5.1 (http://bugs.mysql.com/bug.php?id=39489) which will result in an error message when trying to create a trigger on a MySQL instance with binary logging enabled (for example when replicating to a slave). This can be worked around by adding the following line to your MySQL configuration file.

NEM Installation

Red Hat

Start by adding the adding the Nativ repository:

[root@localhost ~]# tee /etc/yum.repos.d/ns.repo <<EOF

[ns]

name=Red Hat Enterprise Linux 6Server - $basearch - Source

baseurl=http://packages.nativ-systems.com/redhat/ns/6Server/$basearch/

enabled=1

gpgcheck=0

EOF

Oracle Java 7 is a prerequisite, you can install it from Nativ’s repository:

[root@localhost ~]# tee /etc/yum.repos.d/java7.repo <<EOF

[java7]

name=Red Hat Enterprise Linux $releasever - $basearch - Source

baseurl=http://packages.nativ-systems.com/redhat/java7/$releasever/$basearch/

enabled=1

gpgcheck=0

EOF

Note: It is assumed that the EPEL repository is already available, so NEM and Oracle Java can now be installed

[root@localhost ~]# yum install nativsystems-environment-manager-agent jdk

Loaded plugins: fastestmirror

Loading mirror speeds from cached hostfile

* base: mirrors.ukfast.co.uk

* epel: epel.check-update.co.uk

* extras: mirror.econdc.com

* updates: centos.mirroring.pulsant.co.uk

Setting up Install Process

Resolving Dependencies

--> Running transaction check

---> Package jdk.x86_64 2000: 1.7.0_40 will be installed

---> Package nativsystems-environment-manager-agent.i686 0:0.1-97.el6 will be installed

Transaction Summary

================================================================================

Install 69 Package(s)

Upgrade 1 Package(s)

Total download size: 164 M

Is this ok [y/N]: y

Downloading Packages:

(1/70): alsa-lib-1.0.22-3.el6.x86_64.rpm | 370 kB 00:00

…

Complete!

[root@localhost ~]#

Ubuntu

root@ip-10-9-5-22:~# wget -qO - http://packages.nativ-systems.com/keys | sudo apt-key add -

root@ip-10-9-5-22:~# echo deb http://packages.nativ-systems.com/ stable ns | sudo tee -a /etc/apt/sources.list

deb http://packages.nativ-systems.com/ stable ns

root@ip-10-9-5-22:~# sudo apt-get update

Ign http://us-west-2.ec2.archive.ubuntu.com trusty InRelease

Ign http://us-west-2.ec2.archive.ubuntu.com trusty-updates InRelease

Hit http://us-west-2.ec2.archive.ubuntu.com trusty Release.gpg

Hit http://us-west-2.ec2.archive.ubuntu.com trusty-updates Release.gpg

(...)

Ign http://packages.nativ-systems.com stable InRelease

Ign http://security.ubuntu.com trusty-security InRelease

Hit http://packages.nativ-systems.com stable Release.gpg

Hit http://security.ubuntu.com trusty-security Release.gpg

Hit http://packages.nativ-systems.com stable Release

Hit http://security.ubuntu.com trusty-security Release

Hit http://packages.nativ-systems.com stable/ns amd64 Packages

(...)

Fetched 15.9 MB in 9s (1,623 kB/s)

Reading package lists... Done

root@ip-10-9-5-22:~# apt-get install nativsystems-environment-manager-agent

Reading package lists... Done

Building dependency tree

Reading state information... Done

The following extra packages will be installed:

(...)

The following NEW packages will be installed:

ant ant-optional apache2 apache2-bin apache2-data ca-certificates-java cifs-utils

(...)

nativsystems-environment-manager-agent nfs-common odbcinst odbcinst1debian2 openjdk-7-jre-headless oracle-java7-bin oracle-java7-jdk oracle-java7-jre

(...)

1 upgraded, 89 newly installed, 0 to remove and 66 not upgraded.

Need to get 187 MB of archives.

After this operation, 404 MB of additional disk space will be used.

Do you want to continue? [Y/n] Y

Get:1 http://us-west-2.ec2.archive.ubuntu.com/ubuntu/ trusty/main libapr1 amd64 1.5.0-1 [85.1 kB]

(...)

done.

Note: Flex 5.1.0 requires NEM version 0.1.4 (or later).

NEM Configuration

$ sudo vim /etc/nem-upgrade.conf

Add the configuration and definition URLs:

[nem]

definition=svn https://nem.mio:XXXXXXX@repository.nativ-systems.com/svn/nem/projects/mio/platform/branches/MIO-Prod

configuration=file:///etc/mio.conf

Save and exit.

Environment Configuration

One configuration file should be made per environment. These configurations should be as minimal as possible. Normally a huge bundle of configuration files are associated with a project but the actual changes between environments are quite small. For example: The database host will differ between environments so this should be put into the configuration file, but Tomcat ports should be consistent through all environments, so these should be hard coded. Moving parameters into the configuration should be done as lazily as possible as this reduces the number of differences between environments. The configuration file is provided to Apache Ant during the build out stage.

Index

# Flex node type: master, job, index, playout or resource

nodetype=index

# ID to use for clustering.

clusterid=uatmio

# Defines the name of the logs directory

# /var/nem-logs/<clientname>/<domainname>/jboss/<hostname>/now/

clientname=company

# Defines master account URL. All account names will be templated according to this, ie. will be in the *.uat.company.com format

domainname=master.uat.company.com

# log4j log level

log.level=WARN

# Enable JVM debugging

jvm.debug=false

# Database configuration

# This is the same database as the master node

databasehost=10.10.10.10

databasename=miomaster

databaseuser=miomaster_user

databasepassword=miomaster_password

# Shared storage

# filestore is for indexes, cluster state and various temporary work files

# You can force nem to recursively chown the mounted storage to the user specified - recommended to set to 'mio' when using NFS.

filestore.owner=

filestore.mount=nfs 10.10.20.20:/export/storage/filestore

# Java executable path

java.executable=/usr/lib/jvm/java-7-oracle/bin/java

# Holds IP addresses of masters (i.e. where HornetQ is deployed)

master.ip.addresses=10.10.11.11,10.10.11.12

#####################

# Optional settings:

# set initial and maximum amount of allocated heapspace:

# heapspace=2g

# send log files to a log4j server listening on port 4560 (for example Logstash with a log4j input):

# logstash.server=10.10.10.9

# add a JMX username/password to allow for monitoring over JMX. please note that the JMX port is

# listening on localhost:9999 only and you will need to use JBoss EAP remoting-jmx libraries to connect.

# the connection string is service:jmx:remoting-jmx://localhost:9999

# you can find detailed information about JMX in JBoss EAP at https://planet.jboss.org/post/jmx_connections_to_jboss_as

# jmx.username=user

# jmx.password=password

Master

# Flex node type: master, job, index, playout or resource

nodetype=master

# ID to use for clustering.

clusterid=uatmio

# Defines the name of the logs directory

# /var/nem-logs/<clientname>/<domainname>/jboss/<hostname>/now/

clientname=company

# Defines master account URL. All account names will be templated according to this, ie. for the example below it would be in the "account.uat.company.com" format

domainname=master.uat.company.com

# log4j log level

log.level=WARN

# Enable JVM debugging

jvm.debug=false

# Database configuration

databasehost=10.10.10.10

databasename=miomaster

databaseuser=miomaster_user

databasepassword=miomaster_password

# Shared storage

# filestore is for indexes, cluster state and various temporary work files

# media is for your media files

# You can force nem to recursively chown the mounted storage to the user specified - recommended to set to 'mio' when using NFS.

filestore.owner=

# You can use the 'local' option ONLY if you're running in a non-clustered, single-master environment

# filestore.mount=local

filestore.mount=nfs 10.10.20.20:/export/storage/filestore

# This is your media storage share

# - you can pass options to mount.cifs as the third argument, like in the example below

# - the uid and gid options are mandatory

# - when using newer Linux kernels you might need to use sec=ntlm to mount successfully (for example with netApp)

media.mount=cifs //10.10.20.10/media username=mio,password=pass,uid=mio,gid=mio,sec=ntlm

media.dirs=RECURSIVEWILDCARD

# Keep this set to local.

public.mount=local

public.owner=

# Subdirectory of /nem/apache/holdingpages containing a page displayed by Apache when Flex is down

holdingpages=default

# CARP configuration

# Use the following if you are not using CARP:

sharedip.vhid = 1

sharedip.password = password

sharedip.addresses = 127.0.0.1

sharedip.ip = 127.0.0.3

# If using CARP, you'll need a unique vhid!

# sharedip.vhid = 13

# sharedip.password = yourp4ssw0rd

# sharedip.addresses = 10.10.11.11,10.10.11.12 # comma-separated list of IP addresses of Flex Master nodes

# sharedip.ip = 10.10.11.10 # the virtual CARP IP

# You can declare extra packages that NEM will check the operating system for before allowing you to start Flex

supplementalpackages=cifs-utils,clamav

# Intermediate SSL certificate for Apache. This is only required if specified by your certification authority.

# If not, ensure this file is a commented string

# SSLCertificateChainFile = SSLCertificateChainFile /etc/apache2/mio_intermediate.cer

SSLCertificateChainFile= # not applicable

# Java executable path

java.executable=/usr/lib/jvm/java-7-oracle/bin/java

# Holds IP addresses for master nodes (i.e. where HornetQ is deployed)

master.ip.addresses=10.10.11.11,10.10.11.12

# Holds IP addresses for job nodes (so that we can union with master nodes to form JGroups/Hazelcast clusters)

# If this is not defined, NEM will buildout with jobs executing on masters

job.ip.addresses=10.10.11.81,10.10.11.82

# The 'testuser' password if different from default

testuser.password=testuser

# Hibernate search

# indexing topology - 'standalone' means local indexing, 'slave' means relying on an indexing node

# If this is set to standalone or not set, NEM will buildout with indexing executing on masters

hibernate.search.index.topology=slave

# Flex Playout API settings

tv.nativ.mio.playout.api.endpoint=https://playout.uat.company.com

tv.nativ.mio.playout.api.username=testuser

tv.nativ.mio.playout.api.password=testuser

#####################

# Optional settings:

# set initial and maximum amount of allocated heapspace:

# heapspace=2g

# send log files to a log4j server listening on port 4560 (for example Logstash with a log4j input):

# logstash.server=10.10.10.9

# comma-separated list of hosts Apache should be load balancing over. If not specified, the default is all hosts in the cluster (nem.cluster.ip.addresses)

# ajp13workers=10.10.11.11,10.10.11.12

# add a JMX username/password to allow for monitoring over JMX. please note that the JMX port is

# listening on localhost:9999 only and you will need to use JBoss EAP remoting-jmx libraries to connect.

# the connection string is service:jmx:remoting-jmx://localhost:9999

# you can find detailed information about JMX in JBoss EAP at https://planet.jboss.org/post/jmx_connections_to_jboss_as

# jmx.username=user

# jmx.password=password

# LDAP integration

# ldap.url=ldap://ldap.server.company.com:389/

# ldap.domain=company.com

# ldap.searchbase=OU=employees,DC=company,DC=com

Job

# Flex node type: master, job, index, playout or resource

nodetype=job

# ID to use for clustering.

clusterid=uatmio

# Defines the name of the logs directory

# /var/nem-logs/<clientname>/<domainname>/jboss/<hostname>/now/

clientname=company

# Defines master account URL. All account names will be templated according to this, ie. for the example below it would be in the "account.uat.company.com" format

domainname=master.uat.company.com

# log4j log level

log.level=WARN

# Enable JVM debugging

jvm.debug=false

# Database configuration

databasehost=10.10.10.10

databasename=miomaster

databaseuser=miomaster_user

databasepassword=miomaster_password

# Shared storage

# filestore is for indexes, cluster state and various temporary work files

# media is for your media files

# You can force nem to recursively chown the mounted storage to the user specified - recommended to set to 'mio' when using NFS.

filestore.owner=

# You can use the 'local' option ONLY if you're running in a non-clustered, single-master environment

# filestore.mount=local

filestore.mount=nfs 10.10.20.20:/export/storage/filestore

# This is your media storage share

# - you can pass options to mount.cifs as the third argument, like in the example below

# - the uid and gid options are mandatory

# - when using newer Linux kernels you might need to use sec=ntlm to mount successfully (for example with netApp)

media.mount=cifs //10.10.20.10/media username=mio,password=pass,uid=mio,gid=mio,sec=ntlm

media.dirs=RECURSIVEWILDCARD

# Keep this set to local.

public.mount=local

public.owner=

# CARP configuration

# Use the following if you are not using CARP:

sharedip.vhid = 1

sharedip.password = password

sharedip.addresses = 127.0.0.1

sharedip.ip = 127.0.0.3

# If using CARP, you'll need a unique vhid!

# sharedip.vhid = 13

# sharedip.password = yourp4ssw0rd

# sharedip.addresses = 10.10.11.11,10.10.11.12 # comma-separated list of IP addresses of Flex Master nodes

# sharedip.ip = 10.10.11.10 # the virtual CARP IP

# You can declare extra packages that NEM will check the operating system for before allowing you to start Flex

supplementalpackages=cifs-utils,clamav

# Java executable path

java.executable=/usr/lib/jvm/java-7-oracle/bin/java

# Holds IP addresses for master nodes (i.e. where HornetQ is deployed)

master.ip.addresses=10.10.11.11,10.10.11.12

# Holds IP addresses for job nodes (so that we can union with master nodes to form JGroups/Hazelcast clusters)

# If this is not defined, NEM will buildout with jobs executing on masters

job.ip.addresses=10.10.11.81,10.10.11.82

# Hibernate search

# indexing topology - 'standalone' means local indexing, 'slave' means relying on an indexing node

# If this is set to standalone or not set, NEM will buildout with indexing executing on masters

hibernate.search.index.topology=slave

# Flex Playout API settings

tv.nativ.mio.playout.api.endpoint=https://playout.uat.company.com

tv.nativ.mio.playout.api.username=testuser

tv.nativ.mio.playout.api.password=testuser

#####################

# Optional settings:

# set initial and maximum amount of allocated heapspace:

# heapspace=2g

# send log files to a log4j server listening on port 4560 (for example Logstash with a log4j input):

# logstash.server=10.10.10.9

# add a JMX username/password to allow for monitoring over JMX. please note that the JMX port is

# listening on localhost:9999 only and you will need to use JBoss EAP remoting-jmx libraries to connect.

# the connection string is service:jmx:remoting-jmx://localhost:9999

# you can find detailed information about JMX in JBoss EAP at https://planet.jboss.org/post/jmx_connections_to_jboss_as

# jmx.username=user

# jmx.password=password

Playout

# Flex node type: master, job, index, playout or resource

nodetype=playout

# ID to use for clustering.

clusterid=uatmioplayout

# Defines the name of the logs directory

# /var/nem-logs/<clientname>/<domainname>/jboss/<hostname>/now/

clientname=nativ

# Defines the playout domain name

domainname=playout.stage.mioeverywhere.tv

# log4j log level

log.level=WARN

# Enable JVM debugging

jvm.debug=false

# database config

databasehost=10.10.10.10

databasename=mioplayout

databaseuser=mioplayout_user

databasepassword=mioplayout_password

# Shared storage

# This is where the lucene playout search index is stored - needs to be accessible by all playout nodes.

lucene.mount=nfs 10.10.20.20/export/storage/playout

lucene.owner=

public.mount=local

public.owner=mio

# Web service base URL to be used by Playout Wrapper.

# This is deprecated, set it as below.

tv.nativ.mio.playout.webservices.baseurl=http://127.0.0.1:30044/webservices/

# CARP configuration

# Use the following if you are not using CARP:

sharedip.vhid = 1

sharedip.password = password

sharedip.addresses = 127.0.0.1

sharedip.ip = 127.0.0.3

# If using CARP, you'll need a unique vhid!

# sharedip.vhid = 26

# sharedip.password = yourp4ssw0rd

# sharedip.addresses = 10.10.12.11,10.10.12.12 # comma-separated list of IP addresses of Flex Playout nodes

# sharedip.ip = 10.10.12.10 # the virtual CARP IP

# Intermediate SSL certificate for Apache. This is only required if specified by your certification authority.

# If not, ensure this file is a commented string

# SSLCertificateChainFile = SSLCertificateChainFile /etc/apache2/mio_intermediate.cer

SSLCertificateChainFile= # not applicable

# Java executable path

java.executable=/usr/lib/jvm/java-7-oracle/bin/java

# Holds comma separated IP addresses of all playout nodes.

playout.ip.addresses=10.10.12.11,10.10.12.12

# Total number of search results returned by lucene

tv.nativ.mio.playout.lucene.hits.per.page=5000

#####################

# Optional settings:

# set initial and maximum amount of allocated heapspace:

# heapspace=2g

# send log files to a log4j server listening on port 4560 (for example Logstash with a log4j input):

# logstash.server=10.10.10.9

# comma-separated list of hosts Apache should be load balancing over. If not specified, the default is all hosts in the cluster (nem.cluster.ip.addresses)

# ajp13workers=10.10.11.11,10.10.11.12

# add a JMX username/password to allow for monitoring over JMX. please note that the JMX port is

# listening on localhost:9999 only and you will need to use JBoss EAP remoting-jmx libraries to connect.

# the connection string is service:jmx:remoting-jmx://localhost:9999

# you can find detailed information about JMX in JBoss EAP at https://planet.jboss.org/post/jmx_connections_to_jboss_as

# jmx.username=user

# jmx.password=password

Resource

# Flex node type: master, job, index, playout or resource

nodetype=resource

# Defines the name of the logs directory

# /var/nem-logs/<clientname>/<domainname>/jboss/<hostname>/now/

clientname=nativ

# Master account URL

domainname=master.uat.company.com

# log4j log level

log.level=WARN

# Enable JVM debugging

jvm.debug=true

# For FTP resources set these to local

filestore.mount=local

filestore.owner=mio

incoming.mount=local

incoming.owner=mio

public.mount=local

public.owner=mio

# CARP configuration

# Use the following if you are not using CARP:

sharedip.vhid = 1

sharedip.password = password

sharedip.addresses = 127.0.0.1

sharedip.ip = 127.0.0.3

# If using CARP, you'll need a unique vhid!

# sharedip.vhid = 13

# sharedip.password = yourp4ssw0rd

# sharedip.addresses = 10.10.11.11,10.10.11.12 # comma-separated list of IP addresses of Flex Master nodes

# sharedip.ip = 10.10.11.10 # the virtual CARP IP

# You can declare extra packages that NEM will check the operating system for before allowing you to start Flex

supplementalpackages=

# Intermediate SSL certificate for Apache. This is only required if specified by your certification authority.

# If not, ensure this file is a commented string

#SSLCertificateChainFile = SSLCertificateChainFile /etc/apache2/mio.intermediate.crt

SSLCertificateChainFile = # not applicable

# Database configuration

http_databasehost=10.10.10.10

http_databasename=mio_web_upload

http_databaseuser=mio_web_upload

http_databasepassword=mio_web_upload_password

# Comma-separated list of all master nodes

master.ip.addresses=10.5.13.13,10.5.13.14

# Java executable path

java.executable=/usr/lib/jvm/java-7-oracle/bin/java

#####################

# Optional settings:

# set initial and maximum amount of allocated heapspace:

# heapspace=2g

# send log files to a log4j server listening on port 4560 (for example Logstash with a log4j input):

# logstash.server=10.10.10.9

# comma-separated list of hosts Apache should be load balancing over. If not specified, the default is all hosts in the cluster (nem.cluster.ip.addresses)

# ajp13workers=10.10.11.11,10.10.11.12

# add a JMX username/password to allow for monitoring over JMX. please note that the JMX port is

# listening on localhost:9999 only and you will need to use JBoss EAP remoting-jmx libraries to connect.

# the connection string is service:jmx:remoting-jmx://localhost:9999

# you can find detailed information about JMX in JBoss EAP at https://planet.jboss.org/post/jmx_connections_to_jboss_as

# jmx.username=user

# jmx.password=password

Deployment Walkthrough

The following example shows a change to the Flex Enterprise release version and its deployment.

Before deploying Flex for the first time, ensure that:

• Apache will be able to create outgoing connections (SELinux on RedHand CentOS blocks this by default)

• iptables is disabled (RedHat and CentOS have a restrictive set of firewall rules by default)

Log into the Flex server and perform the upgrade

$ sudo nem upgrade

INFO:root:Destroying existing environment...

INFO:root:Disabling [ucarp_01] /nem/carp/nem.conf (running)...

INFO:root:Disabled [ucarp_01] /nem/carp/nem.conf (not running)

INFO:root:Disabling [daemon_01] /nem/mio/nem.conf (running)...

WARNING:root:Still trying to kill: [daemon_01] /nem/mio/nem.conf (running)

INFO:root:Disabled [daemon_01] /nem/mio/nem.conf (not running)

INFO:root:Disabling [folder_01] /nem/public/nem.conf (running)...

INFO:root:Disabled [folder_01] /nem/public/nem.conf (not running)

...

INFO:root:Disabled [sql_01] /nem/db/nem.conf (running)

INFO:root:Destroying existing environment done

INFO:root:Building out new environment...

INFO:root:Checking out assets...

It presents you the following question:

"Deploying svn https://****:****@repository.nativ-systems.com/svn/path/to/sbn/branch, revision 25. Is that okay?"

Responding ‘yes’ to this question will deploy the latest revision of the chosen branch in Subversion. Responding ‘no’ will prompt the user for an older revision to deploy. Once complete, the upgrade process will print a summary of what’s been deployed:

INFO:root:Checking out assets done

INFO:root:Running ant buildout...

INFO:root:Ant buildout complete

INFO:root:Tidying up checkout assets...

INFO:root:Tidying up checkout assets done

INFO:root:Building out new environment done

INFO:root:Starting new environment...

INFO:root:Enabling [meta_01] /nem/nem.conf (not running)...

INFO:root:Already enabled: [sql_01] /nem/db/nem.conf (running)

INFO:root:Enabling [users_01] /nem/users/nem.conf (not running)...

...

INFO:root:Enabled [meta_01] /nem/nem.conf (running)

INFO:root:Starting new environment done

INFO:root:

Deployment seems to have gone okay of:

Definition: svn https://****:****@repository.nativsystems.com/svn/nem/projects/mio/platform/branches/MIO-Maint (revision 5152)

Configuration: file:///etc/mio.conf

When deploying Flex for the first time, you will also need to perform two other steps:

Create necessary directory structures on the filestore and media network shares. You can do this by running nem run in the /nem/filestore and /nem/media directories - you will be shown a series of prompts asking if you'd like to create missing directories: answer yes to all of those questions.

Deploy the SQL schema on your previously created database - see Database Maintenance for detailed instructions below.

Database Maintenance

Please note that the Flex Master is using SQL triggers in the database. You will need to ensure that the TRIGGER privilege is granted to the database user as it is not granted by default as part of GRANT ALL.

GRANT TRIGGER ON miomaster.* to 'miomaster'@'hostname' IDENTIFIED BY 'password';

Note: There has been a bug in MySQL since a separate TRIGGER privilege was introduced in version 5.1 (http://bugs.mysql.com/bug.php?id=39489) which will result in an error message when trying to create a trigger on a MySQL instance with binary logging enabled (for example when replicating to a slave). This can be worked around by adding the following line to your MySQL configuration file.

log_bin_trust_function_creators = 1

Updating the database schema

Since version 5.1.0, NEM uses Flyway (http://flywaydb.org) to track schema changes (also known as database migrations). When trying to start an environment which hasn't had the changes applied yet, the following will happen:

INFO:root:Starting new environment...

INFO:root:Enabling [meta_01] /nem/nem.conf (not running)...

INFO:root:Enabling [flyway_01] /nem/db/nem.conf (not running)...

Command:

/usr/bin/sudo -u nem /usr/lib/jvm/java-7-oracle/bin/java -cp /usr/share/nativsystems-environment-manager-agent/flyway/lib/*:/usr/share/nativsystems-environment-manager-agent/flyway/drivers/* org.flywaydb.commandline.Main -url=jdbc:mysql://10.5.1.12:3306/miotest?autoReconnect=true&useUnicode=true&characterEncoding=UTF-8&characterSetResults=UTF-8&jdbcCompliantTruncation=false&zeroDateTimeBehavior=convertToNull -user=miotest -password=miotest -sqlMigrationPrefix=V -sqlMigrationSeparator=__ -sqlMigrationSuffix= -placeholderPrefix=asdf -locations=filesystem:/nem/db/migrations validate

Exit status:

Stdout:

Flyway 3.2.1 by Boxfuse

Database: jdbc:mysql://10.10.10.10:3306/miotest (MySQL 5.6)

Validated 7 migrations (execution time 00:00.014s)

ERROR: Validate failed. Detected resolved migration not applied to database: 001

Stderr:

Apply the patch by doing the following:

~$ sudo nem -f /nem/db/nem.conf run

NEM then presents a menu showing multiple Flyway options. Choose Migrate and press the Enter key to apply all remaining changes. At any time you can also use the Info option to see which changes have already been applied and which are still pending.

Maintenance

The ‘nem’ command accepts the following commands/arguments:

Enable

• For example this will start up processes or mount folders via NFS.

• For some destructive changes such as applying SQL, services will refuse to be enabled (throwing an exception, halting deployment) until the change has been applied.

• This recursively starts the service dependencies. If this is not required or desired then supply the ‘--no-dependencies’ command line argument.

Disable

• Stop process, ‘unmount’ NFS, etc.

• Does nothing for some services such as SQL changes.

Restart

• Calls disable and enable commands in sequence.

Status

• Describes the status (running or not running) of the module and its dependencies.run:

• Provides an interactive session for modules such as SQL and folders or runs a process in the foreground for debugging.

• For destructive changes are only made following interaction with the user.

Upgrade

Reads /etc/nem-upgrade.conf and performs an upgrade based on the configuration and definition URLs.

• The system is taken from the built out or fully enabled states, described by the diagram in the appendix to this document, via the start state back to the fully enabled state.

• All running services are disabled, the new code is dropped into place, and then the services are enabled. The order of enabling and disabling is calculated based on the dependencies.

History

• Lists the dates, times, revision numbers and the URLs of previous deployments.

Tree

Prints a tree of dependencies of the selected module. Most of these commands need to find a ‘nem.conf’ file in the current working directory or can be given the path to a ‘nem.conf’ file with the -f flag. For example:

$ cd /nem/apache/

/nem/apache$ sudo nem status

INFO:root:[apache_01] /nem/apache/nem.conf (running)

/nem/apache$

As an alternative, the following command is equivalent to the ones above:

$ sudo nem -f /nem/apache/nem.conf status

INFO:root:[apache_01] /nem/apache/nem.conf (running)

SSL Certificate

The NEM deployment process will expect a SSL certificate (and key) present in the system in a predefined location. If your certificate authority requires the use of an intermediate certificate, this can be done using the SSLCertificateChainFile option in your Environment Configuration. This is required even for environments in which the web console is accessed via a load balancer performing the SSL encryption.

You will need to place your certificate in /etc/apache2/mio.crt and your key in /etc/apache2/mio.key.

If you do not currently have a certificate (for example if you're just setting up a test environment or simply are still awaiting for your certificate order to be fulfilled), you can create and use a self-signed certificate as follows:

1.) Generate a private key:

root@ip-10-9-5-22:~# openssl genrsa -des3 -out server.key 1024

Generating RSA private key, 1024 bit long modulus

.......++++++

....++++++

e is 65537 (0x10001)

Enter pass phrase for server.key

Verifying - Enter pass phrase for server.key:

2.) Generate a Certificate Signing Request:

root@ip-10-9-5-22:~# openssl req -new -key server.key -out server.csr

Enter pass phrase for server.key:

You are about to be asked to enter information that will be incorporated

into your certificate request.

What you are about to enter is what is called a Distinguished Name or a DN.

There are quite a few fields but you can leave some blank

For some fields there will be a default value,

If you enter '.', the field will be left blank.

-----

Country Name (2 letter code) [AU]:

State or Province Name (full name) [Some-State]:

Locality Name (eg, city) []:

Organization Name (eg, company) [Internet Widgits Pty Ltd]:

Organizational Unit Name (eg, section) []:

Common Name (e.g. server FQDN or YOUR name) []:

Email Address []:

Please enter the following 'extra' attributes

to be sent with your certificate request

A challenge password []:

An optional company name []:

3.) Remove passphrase from key:

root@ip-10-9-5-22:~# cp server.key server.key.org

root@ip-10-9-5-22:~# openssl rsa -in server.key.org -out server.key

Enter pass phrase for server.key.org:

writing RSA key

4.) Generate a self-signed certificate:

root@ip-10-9-5-22:~# openssl x509 -req -days 365 -in server.csr -signkey server.key -out server.crt

Signature ok

subject=/C=AU/ST=Some-State/O=Internet Widgits Pty Ltd

Getting Private key

5.) Install the key and certificate files:

root@ip-10-9-5-22:~# cp server.key /etc/apache2/mio.key

root@ip-10-9-5-22:~# cp server.crt /etc/apache2/mio.crt

Environment Configuration

Index

# Flex node type: master, job, index, playout or resource

nodetype=index

# ID to use for clustering.

clusterid=uatmio

# Defines the name of the logs directory

# /var/nem-logs/<clientname>/<domainname>/jboss/<hostname>/now/

clientname=company

# Defines master account URL. All account names will be templated according to this, ie. will be in the *.uat.company.com format

domainname=master.uat.company.com

# log4j log level

log.level=WARN

# Enable JVM debugging

jvm.debug=false

# Database configuration

# This is the same database as the master node

databasehost=10.10.10.10

databasename=miomaster

databaseuser=miomaster_user

databasepassword=miomaster_password

# Shared storage

# filestore is for indexes, cluster state and various temporary work files

# You can force nem to recursively chown the mounted storage to the user specified - recommended to set to 'mio' when using NFS.

filestore.owner=

filestore.mount=nfs 10.10.20.20:/export/storage/filestore

# Java executable path

java.executable=/usr/lib/jvm/java-7-oracle/bin/java

# Holds IP addresses of masters (i.e. where HornetQ is deployed)

master.ip.addresses=10.10.11.11,10.10.11.12

#####################

# Optional settings:

# set initial and maximum amount of allocated heapspace:

# heapspace=2g

# send log files to a log4j server listening on port 4560 (for example Logstash with a log4j input):

# logstash.server=10.10.10.9

# add a JMX username/password to allow for monitoring over JMX. please note that the JMX port is

# listening on localhost:9999 only and you will need to use JBoss EAP remoting-jmx libraries to connect.

# the connection string is service:jmx:remoting-jmx://localhost:9999

# you can find detailed information about JMX in JBoss EAP at https://planet.jboss.org/post/jmx_connections_to_jboss_as

# jmx.username=user

# jmx.password=password

Master

# Flex node type: master, job, index, playout or resource

nodetype=master

# ID to use for clustering.

clusterid=uatmio

# Defines the name of the logs directory

# /var/nem-logs/<clientname>/<domainname>/jboss/<hostname>/now/

clientname=company

# Defines master account URL. All account names will be templated according to this, ie. for the example below it would be in the "account.uat.company.com" format

domainname=master.uat.company.com

# log4j log level

log.level=WARN

# Enable JVM debugging

jvm.debug=false

# Database configuration

databasehost=10.10.10.10

databasename=miomaster

databaseuser=miomaster_user

databasepassword=miomaster_password

# Shared storage

# filestore is for indexes, cluster state and various temporary work files

# media is for your media files

# You can force nem to recursively chown the mounted storage to the user specified - recommended to set to 'mio' when using NFS.

filestore.owner=

# You can use the 'local' option ONLY if you're running in a non-clustered, single-master environment

# filestore.mount=local

filestore.mount=nfs 10.10.20.20:/export/storage/filestore

# This is your media storage share

# - you can pass options to mount.cifs as the third argument, like in the example below

# - the uid and gid options are mandatory

# - when using newer Linux kernels you might need to use sec=ntlm to mount successfully (for example with netApp)

media.mount=cifs //10.10.20.10/media username=mio,password=pass,uid=mio,gid=mio,sec=ntlm

media.dirs=RECURSIVEWILDCARD

# Keep this set to local.

public.mount=local

public.owner=

# Subdirectory of /nem/apache/holdingpages containing a page displayed by Apache when Flex is down

holdingpages=default

# CARP configuration

# Use the following if you are not using CARP:

sharedip.vhid = 1

sharedip.password = password

sharedip.addresses = 127.0.0.1

sharedip.ip = 127.0.0.3

# If using CARP, you'll need a unique vhid!

# sharedip.vhid = 13

# sharedip.password = yourp4ssw0rd

# sharedip.addresses = 10.10.11.11,10.10.11.12 # comma-separated list of IP addresses of Flex Master nodes

# sharedip.ip = 10.10.11.10 # the virtual CARP IP

# You can declare extra packages that NEM will check the operating system for before allowing you to start Flex

supplementalpackages=cifs-utils,clamav

# Intermediate SSL certificate for Apache. This is only required if specified by your certification authority.

# If not, ensure this file is a commented string

# SSLCertificateChainFile = SSLCertificateChainFile /etc/apache2/mio_intermediate.cer

SSLCertificateChainFile= # not applicable

# Java executable path

java.executable=/usr/lib/jvm/java-7-oracle/bin/java

# Holds IP addresses for master nodes (i.e. where HornetQ is deployed)

master.ip.addresses=10.10.11.11,10.10.11.12

# Holds IP addresses for job nodes (so that we can union with master nodes to form JGroups/Hazelcast clusters)

# If this is not defined, NEM will buildout with jobs executing on masters

job.ip.addresses=10.10.11.81,10.10.11.82

# Hibernate search

# indexing topology - 'standalone' means local indexing, 'slave' means relying on an indexing node

# If this is set to standalone or not set, NEM will buildout with indexing executing on masters

hibernate.search.index.topology=slave

# Flex Playout API settings

tv.nativ.mio.playout.api.endpoint=https://playout.uat.company.com

tv.nativ.mio.playout.api.username=testuser

tv.nativ.mio.playout.api.password=testuser

#####################

# Optional settings:

# set initial and maximum amount of allocated heapspace:

# heapspace=2g

# send log files to a log4j server listening on port 4560 (for example Logstash with a log4j input):

# logstash.server=10.10.10.9

# comma-separated list of hosts Apache should be load balancing over. If not specified, the default is all hosts in the cluster (nem.cluster.ip.addresses)

# ajp13workers=10.10.11.11,10.10.11.12

# add a JMX username/password to allow for monitoring over JMX. please note that the JMX port is

# listening on localhost:9999 only and you will need to use JBoss EAP remoting-jmx libraries to connect.

# the connection string is service:jmx:remoting-jmx://localhost:9999

# you can find detailed information about JMX in JBoss EAP at https://planet.jboss.org/post/jmx_connections_to_jboss_as

# jmx.username=user

# jmx.password=password

Master

# Flex node type: master, job, index, playout or resource

nodetype=master

# ID to use for clustering.

clusterid=uatmio

# Defines the name of the logs directory

# /var/nem-logs/<clientname>/<domainname>/jboss/<hostname>/now/

clientname=company

# Defines master account URL. All account names will be templated according to this, ie. for the example below it would be in the "account.uat.company.com" format

domainname=master.uat.company.com

# log4j log level

log.level=WARN

# Enable JVM debugging

jvm.debug=false

# Database configuration

databasehost=10.10.10.10

databasename=miomaster

databaseuser=miomaster_user

databasepassword=miomaster_password

# Shared storage

# filestore is for indexes, cluster state and various temporary work files

# media is for your media files

# You can force nem to recursively chown the mounted storage to the user specified - recommended to set to 'mio' when using NFS.

filestore.owner=

# You can use the 'local' option ONLY if you're running in a non-clustered, single-master environment

# filestore.mount=local

filestore.mount=nfs 10.10.20.20:/export/storage/filestore

# This is your media storage share

# - you can pass options to mount.cifs as the third argument, like in the example below

# - the uid and gid options are mandatory

# - when using newer Linux kernels you might need to use sec=ntlm to mount successfully (for example with netApp)

media.mount=cifs //10.10.20.10/media username=mio,password=pass,uid=mio,gid=mio,sec=ntlm

media.dirs=RECURSIVEWILDCARD

# Keep this set to local.

public.mount=local

public.owner=

# Subdirectory of /nem/apache/holdingpages containing a page displayed by Apache when Flex is down

holdingpages=default

# CARP configuration

# Use the following if you are not using CARP:

sharedip.vhid = 1

sharedip.password = password

sharedip.addresses = 127.0.0.1

sharedip.ip = 127.0.0.3

# If using CARP, you'll need a unique vhid!

# sharedip.vhid = 13

# sharedip.password = yourp4ssw0rd

# sharedip.addresses = 10.10.11.11,10.10.11.12 # comma-separated list of IP addresses of Flex Master nodes

# sharedip.ip = 10.10.11.10 # the virtual CARP IP

# You can declare extra packages that NEM will check the operating system for before allowing you to start Flex

supplementalpackages=cifs-utils,clamav

# Intermediate SSL certificate for Apache. This is only required if specified by your certification authority.

# If not, ensure this file is a commented string

# SSLCertificateChainFile = SSLCertificateChainFile /etc/apache2/mio_intermediate.cer

SSLCertificateChainFile= # not applicable

# Java executable path

java.executable=/usr/lib/jvm/java-7-oracle/bin/java

# Holds IP addresses for master nodes (i.e. where HornetQ is deployed)

master.ip.addresses=10.10.11.11,10.10.11.12

# Holds IP addresses for job nodes (so that we can union with master nodes to form JGroups/Hazelcast clusters)

# If this is not defined, NEM will buildout with jobs executing on masters

job.ip.addresses=10.10.11.81,10.10.11.82

# Hibernate search

# indexing topology - 'standalone' means local indexing, 'slave' means relying on an indexing node

# If this is set to standalone or not set, NEM will buildout with indexing executing on masters

hibernate.search.index.topology=slave

# Flex Playout API settings

tv.nativ.mio.playout.api.endpoint=https://playout.uat.company.com

tv.nativ.mio.playout.api.username=testuser

tv.nativ.mio.playout.api.password=testuser

#####################

# Optional settings:

# set initial and maximum amount of allocated heapspace:

# heapspace=2g

# send log files to a log4j server listening on port 4560 (for example Logstash with a log4j input):

# logstash.server=10.10.10.9

# comma-separated list of hosts Apache should be load balancing over. If not specified, the default is all hosts in the cluster (nem.cluster.ip.addresses)

# ajp13workers=10.10.11.11,10.10.11.12

# add a JMX username/password to allow for monitoring over JMX. please note that the JMX port is

# listening on localhost:9999 only and you will need to use JBoss EAP remoting-jmx libraries to connect.

# the connection string is service:jmx:remoting-jmx://localhost:9999

# you can find detailed information about JMX in JBoss EAP at https://planet.jboss.org/post/jmx_connections_to_jboss_as

# jmx.username=user

# jmx.password=password

Job

# Flex node type: master, job, index, playout or resource

nodetype=job

# ID to use for clustering.

clusterid=uatmio

# Defines the name of the logs directory

# /var/nem-logs/<clientname>/<domainname>/jboss/<hostname>/now/

clientname=company

# Defines master account URL. All account names will be templated according to this, ie. for the example below it would be in the "account.uat.company.com" format

domainname=master.uat.company.com

# log4j log level

log.level=WARN

# Enable JVM debugging

jvm.debug=false

# Database configuration

databasehost=10.10.10.10

databasename=miomaster

databaseuser=miomaster_user

databasepassword=miomaster_password

# Shared storage

# filestore is for indexes, cluster state and various temporary work files

# media is for your media files

# You can force nem to recursively chown the mounted storage to the user specified - recommended to set to 'mio' when using NFS.

filestore.owner=

# You can use the 'local' option ONLY if you're running in a non-clustered, single-master environment

# filestore.mount=local

filestore.mount=nfs 10.10.20.20:/export/storage/filestore

# This is your media storage share

# - you can pass options to mount.cifs as the third argument, like in the example below

# - the uid and gid options are mandatory

# - when using newer Linux kernels you might need to use sec=ntlm to mount successfully (for example with netApp)

media.mount=cifs //10.10.20.10/media username=mio,password=pass,uid=mio,gid=mio,sec=ntlm

media.dirs=RECURSIVEWILDCARD

# Keep this set to local.

public.mount=local

public.owner=

# CARP configuration

# Use the following if you are not using CARP:

sharedip.vhid = 1

sharedip.password = password

sharedip.addresses = 127.0.0.1

sharedip.ip = 127.0.0.3

# If using CARP, you'll need a unique vhid!

# sharedip.vhid = 13

# sharedip.password = yourp4ssw0rd

# sharedip.addresses = 10.10.11.11,10.10.11.12 # comma-separated list of IP addresses of Flex Master nodes

# sharedip.ip = 10.10.11.10 # the virtual CARP IP

# You can declare extra packages that NEM will check the operating system for before allowing you to start Flex

supplementalpackages=cifs-utils,clamav

# Java executable path

java.executable=/usr/lib/jvm/java-7-oracle/bin/java

# Holds IP addresses for master nodes (i.e. where HornetQ is deployed)

master.ip.addresses=10.10.11.11,10.10.11.12

# Holds IP addresses for job nodes (so that we can union with master nodes to form JGroups/Hazelcast clusters)

# If this is not defined, NEM will buildout with jobs executing on masters

job.ip.addresses=10.10.11.81,10.10.11.82

# Hibernate search

# indexing topology - 'standalone' means local indexing, 'slave' means relying on an indexing node

# If this is set to standalone or not set, NEM will buildout with indexing executing on masters

hibernate.search.index.topology=slave

# Flex Playout API settings

tv.nativ.mio.playout.api.endpoint=https://playout.uat.company.com

tv.nativ.mio.playout.api.username=testuser

tv.nativ.mio.playout.api.password=testuser

#####################

# Optional settings:

# set initial and maximum amount of allocated heapspace:

# heapspace=2g

# send log files to a log4j server listening on port 4560 (for example Logstash with a log4j input):

# logstash.server=10.10.10.9

# add a JMX username/password to allow for monitoring over JMX. please note that the JMX port is

# listening on localhost:9999 only and you will need to use JBoss EAP remoting-jmx libraries to connect.

# the connection string is service:jmx:remoting-jmx://localhost:9999

# you can find detailed information about JMX in JBoss EAP at https://planet.jboss.org/post/jmx_connections_to_jboss_as

# jmx.username=user

# jmx.password=password

Playout

# Flex node type: master, job, index, playout or resource

nodetype=playout

# ID to use for clustering.

clusterid=uatmioplayout

# Defines the name of the logs directory

# /var/nem-logs/<clientname>/<domainname>/jboss/<hostname>/now/

clientname=nativ

# Defines the playout domain name

domainname=playout.stage.mioeverywhere.tv

# log4j log level

log.level=WARN

# Enable JVM debugging

jvm.debug=false

# database config

databasehost=10.10.10.10

databasename=mioplayout

databaseuser=mioplayout_user

databasepassword=mioplayout_password

# Shared storage

# This is where the lucene playout search index is stored - needs to be accessible by all playout nodes.

lucene.mount=nfs 10.10.20.20/export/storage/playout

lucene.owner=

# Web service base URL to be used by Playout Wrapper.

# This is deprecated, set it as below.

tv.nativ.mio.playout.webservices.baseurl=http://127.0.0.1:30044/webservices/

# CARP configuration

# Use the following if you are not using CARP:

sharedip.vhid = 1

sharedip.password = password

sharedip.addresses = 127.0.0.1

sharedip.ip = 127.0.0.3

# If using CARP, you'll need a unique vhid!

# sharedip.vhid = 26

# sharedip.password = yourp4ssw0rd

# sharedip.addresses = 10.10.12.11,10.10.12.12 # comma-separated list of IP addresses of Flex Playout nodes

# sharedip.ip = 10.10.12.10 # the virtual CARP IP

# Intermediate SSL certificate for Apache. This is only required if specified by your certification authority.

# If not, ensure this file is a commented string

# SSLCertificateChainFile = SSLCertificateChainFile /etc/apache2/mio_intermediate.cer

SSLCertificateChainFile= # not applicable

# Java executable path

java.executable=/usr/lib/jvm/java-7-oracle/bin/java

# Holds comma separated IP addresses of all playout nodes.

playout.ip.addresses=10.10.12.11,10.10.12.12

# Total number of search results returned by lucene

tv.nativ.mio.playout.lucene.hits.per.page=5000

#####################

# Optional settings:

# set initial and maximum amount of allocated heapspace:

# heapspace=2g

# send log files to a log4j server listening on port 4560 (for example Logstash with a log4j input):

# logstash.server=10.10.10.9

# comma-separated list of hosts Apache should be load balancing over. If not specified, the default is all hosts in the cluster (nem.cluster.ip.addresses)

# ajp13workers=10.10.11.11,10.10.11.12

# add a JMX username/password to allow for monitoring over JMX. please note that the JMX port is

# listening on localhost:9999 only and you will need to use JBoss EAP remoting-jmx libraries to connect.

# the connection string is service:jmx:remoting-jmx://localhost:9999

# you can find detailed information about JMX in JBoss EAP at https://planet.jboss.org/post/jmx_connections_to_jboss_as

# jmx.username=user

# jmx.password=password

Resource

# Flex node type: master, job, index, playout or resource

nodetype=resource

# Defines the name of the logs directory

# /var/nem-logs/<clientname>/<domainname>/jboss/<hostname>/now/

clientname=nativ

# Master account URL

domainname=master.uat.company.com

# log4j log level

log.level=WARN

# Enable JVM debugging

jvm.debug=true

# For FTP resources set these to local

filestore.mount=local

filestore.owner=mio

incoming.mount=local

incoming.owner=mio

# CARP configuration

# Use the following if you are not using CARP:

sharedip.vhid = 1

sharedip.password = password

sharedip.addresses = 127.0.0.1

sharedip.ip = 127.0.0.3

# If using CARP, you'll need a unique vhid!

# sharedip.vhid = 13

# sharedip.password = yourp4ssw0rd

# sharedip.addresses = 10.10.11.11,10.10.11.12 # comma-separated list of IP addresses of Flex Master nodes

# sharedip.ip = 10.10.11.10 # the virtual CARP IP

# You can declare extra packages that NEM will check the operating system for before allowing you to start Flex

supplementalpackages=

# Intermediate SSL certificate for Apache. This is only required if specified by your certification authority.

# If not, ensure this file is a commented string

#SSLCertificateChainFile = SSLCertificateChainFile /etc/apache2/mio.intermediate.crt

SSLCertificateChainFile = # not applicable

# Database configuration

databasehost=10.10.10.10

databasename=mioresource

databaseuser=mioresource_user

databasepassword=mioresource_password

# Database configuration for the HTML5 web uploader

http_databasehost=10.10.10.10

http_databasename=mio_web_upload

http_databaseuser=mio_web_upload

http_databasepassword=mio_web_upload_password

# Comma-separated list of all master nodes

master.ip.addresses=10.5.13.13,10.5.13.14

# Java executable path

java.executable=/usr/lib/jvm/java-7-oracle/bin/java

#####################

# Optional settings:

# set initial and maximum amount of allocated heapspace:

# heapspace=2g

# send log files to a log4j server listening on port 4560 (for example Logstash with a log4j input):

# logstash.server=10.10.10.9

# comma-separated list of hosts Apache should be load balancing over. If not specified, the default is all hosts in the cluster (nem.cluster.ip.addresses)

# ajp13workers=10.10.11.11,10.10.11.12

# add a JMX username/password to allow for monitoring over JMX. please note that the JMX port is

# listening on localhost:9999 only and you will need to use JBoss EAP remoting-jmx libraries to connect.

# the connection string is service:jmx:remoting-jmx://localhost:9999

# you can find detailed information about JMX in JBoss EAP at https://planet.jboss.org/post/jmx_connections_to_jboss_as

# jmx.username=user

# jmx.password=password

Database Maintenance

GRANT TRIGGER ON miomaster.* to 'miomaster'@'hostname' IDENTIFIED BY 'password';

log_bin_trust_function_creators = 1

Updating Metadata Schema

As stated earlier, database changes are tracked with Liquibase. If a database change has not been applied then an upgrade will fail until it has been applied. The following will happen:

INFO:root:Starting new environment...

INFO:root:Enabling [meta_01] /nem/nem.conf (not running)...

INFO:root:Enabling [sql_01] /nem/db/nem.conf (not running)...

Could not start

Command:

/usr/bin/sudo -u nem /usr/java/default/bin/java -Djava.net.preferIPv4Stack=true -client -cp /nem/db:/usr/share/nativsystems-environment-manager-agent/liquibase/liquibase.jar

...

... -- LiquiBase version: 2.0-rc1

-- *********************************************************************

-- Lock Database

-- Changeset

/nem/db/liquibase/036_remove_extract_action_jsp.sql.xml::generated::antscript::(Checksum: 2:c37bc10575fc36d1b09257c3b6de4dc2)

UPDATE ACTION_TYPE SET JSP_NAME_ = 'file_action' where JSP_NAME_ = 'extract_action';

INSERT INTO `DATABASECHANGELOG` (`AUTHOR`, `COMMENTS`, `DATEEXECUTED`,

`DESCRIPTION`, `EXECTYPE`, `FILENAME`, `ID`, `LIQUIBASE`, `MD5SUM`, `ORDEREXECUTED`) VALUES ('ant script', '', NOW(), 'Custom SQL', 'EXECUTED','/nem/db/liquibase/036_remove_extract_action_jsp.sql.xml', 'generated', '2.0-rc1', '2:c37bc10575fc36d1b09257c3b6de4dc2', 37);

-- Release Database Lock

Stderr:

INFO 12/10/13 8:47 AM:liquibase: Successfully acquired change log lock

INFO 12/10/13 8:47 AM:liquibase: Reading from `DATABASECHANGELOG`

...

... INFO 12/10/13 8:47 AM:liquibase: ChangeSet /nem/db/liquibase/036_remove_extract_action_jsp.sql.xml::generated::ant script ran successfully in 0ms

INFO 12/10/13 8:47 AM:liquibase: Successfully released change log lock

Apply the patch by doing the following:

~$ sudo nem -f /nem/db/nem.conf run

NEM then presents a menu showing three options, select ‘liquibase (non-interactive)’ to apply all default changes or ‘liquibase’ to apply the changes one-by-one. The third option will start an interactive command line database session, which can be useful to debug database configuration or connectivity issues. A similar process is used for creating folders.

Flex Monitoring

This section suggests items that can be monitored, to verify the status of a Flex system.

Flex Manager Nodes

The Flex master nodes make use of Oracle Java, JBoss and the Apache web server, so standard monitoring tools can be used to test the items mentioned below:

HTTP checks

(mio.local is used as the example host name):

Flex component HTTP GET URL expected response code

console (Apache) https://mio.local/ 200

console (JBoss) http://mio.local:30044/ 200

As Flex checks the URL for the account that you’re trying to access, you will need to fake the Host header in order to be able to perform these checks on the Flex master nodes. Here’s a few examples:

$ curl -s -k -D- -H"Host: master.stage.mioeverywhere.tv" https://127.0.0.1|head -1

HTTP/1.1 200 OK

$ curl -s -D- -H"Host: master.stage.mioeverywhere.tv" 127.0.0.1:30044|head -1

HTTP/1.1 200 OK

The Following will NOT work:

$ curl -s -k -D- https://127.0.0.1|head -1

HTTP/1.1 404 Not Found

• No host correct header found, 404 returned.

$ curl -s -k -D- https://localhost|head -1

HTTP/1.1 500 Internal Server Error

Localhost used as the host header, 500 returned.

$ curl -s -k -D- -H"Host: stage.mioeverywhere.tv" https://127.0.0.1|head -1

HTTP/1.1 404 Not Found

• Correct Domain used but no account specified

NEM Modules

This section gives an overview of the functionality available within NEM. A detailed understanding of the NEM modules is not required for day-to-day operation of a Flex system, but may be useful in debugging issues found when starting, stopping or deploying Flex.

A number of modules are implemented which are intended to cover most of the routine tasks that need to be applied to environments. Module definition is done in ‘nem.conf’ files. For the most part Nativ will be responsible for the maintenance of these, but for minor changes clients and agencies are welcome to make changes. This section is not intended as a definitive guide, but rather an introduction of what's possible. For routine maintenance, knowledge of NEM modules is not required by clients and agencies, simply where to drop their new assets and configuration into Subversion.

The type of module used is set within a nem.conf file, for example:

~$ cat /nem/apache/nem.conf

[nem]

service = apache_01

depends = /nem/logs/apache/nem.conf

monitors = ~$

This file makes use of the Apache modules described in the following sections and sets up a dependency on a log folder under /nem/logs/apache/nem.conf.

Apache

Apache configuration can be done with the Apache service. This module simply includes given configuration onto a fairly basic Apache install. For routine maintenance of Apache please edit the relevant vhost.conf file. If you wish to have a new virtual host defined, please open a ServiceNow request, requesting Nativ to do this for you.

Flyway

The Flyway service is a wrapper around Flyway. Please see flywaydb.org/ for more information. This module allows us to assert that database updates are repeated over all environments.

SQL

Please note that the Liquibase database update service has been deprecated in the 5.1 series in favour of Flyway, however it still remains the default for the 5.0 series and is required to upgrade from 5.0.15 to 5.1.0.

The SQL service is a wrapper around Liquibase. Please see http://www.liquibase.org/ for more information. This module allows us to assert that database updates are repeated over all environments. Routine maintenance of an SQL module implementation this will typically require adding new ‘.sql’ files into Subversion.

It is possible to have a database dependency with no SQL module dependency defined if another process is used to track database changes. If changes are defined with the SQL module but not applied then NEM will refuse to upgrade.

If source files are later altered, rather than creating new changes, then this is not supported and will result in unpredictable behaviour.

Packages

The packages module can be used to request more Ubuntu/Red Hat packages be installed which are not included in the base install. Updates to this should be done by Nativ, and requested through the ticketing system.

Processes

The process module starts and stops daemons such as Tomcat or JBoss. It is a wrapper around start-stop-daemon.

Nativ have decided to do away with shell scripts for this task and this replacement has proven to be more consistent and reliable. In the ‘nem.conf’ parameters can be defined such as:

• user to run as

• Environment Variables

• Process arguments

• Java properties

It is intended that these should be intuitive enough to developers, but if it is not clear then please suggest what changes to ‘init scripts’ would be required in a ticket and submit it to our helpdesk. Routine maintenance of a process module typically involves updating a WAR file.

Folder

The folder module allows developers to define when data should be stored persistently on local disk or on a NAS.

Please contact the helpdesk if you require this.

Logs

A special type of folder can be created for logs. Logs are then stored in path "/var/nem-logs/<client>/<domain>/<service>/<hostname>/now" folder. In case of example configuration below, the logs could be found in "/var/nem-logs/nativ/master.example.com/jboss/<hostname>/now", where hostname would be the actual hostname of machine running the mio instance.

$ cat /nem/logs/jboss/nem.conf

[nem]

service = logfolder_01

depends =

monitors =

[logfolder]

client = nativ

domain = master.example.com

service = jboss

[folder]

owner = mio

Users

UNIX users can be created in the range 25000 to 29998. Ideally all daemon should run as different users for security reasons. Nativ will typically add new users here when new daemons are created.

Appendix

https://help.ooyala.com/sites/all/libraries/dita/en/media-logistics/flex/dev/51/nem_guide.html