Setting up Prometheus/Grafana Monitoring Dashboard for ForgeRock IDM

Exercise summary: Install Prometheus and Grafana, configure IDM to provide monitoring statistics and experiment with the sample dashboard available on backstage. 

Prometheus is a third-party tool used for gathering and processing monitoring data.
IDM 6.0 has implemented a metrics endpoint which Prometheus can use to gather information about your IDM installation. For more information about installing and running Prometheus, see the Prometheus documentation.

The monitoring exercise would require some data and it is suggested as a prerequisite,
that IDM is set up in a way similar to $OPENIDM/sample/sync-with-ldap or
$OPENIDM/sample/sync-with-csv. This allows you the ability to run a reconciliation against a few users in LDAP or CSV, that will generate some load that can be consumed by Prometheus/Grafana.  

1. On backstage Download the Monitoring Dashboard samples for IDM 6.0.
   
   DM-monitoring-dashboard-samples-6.0.0.zip
   
2. Install and download Prometheus and Grafana.

https://grafana.com/grafana/download?platform=mac

https://prometheus.io/download/

3. Now configure IDM6.0 to enable metrics pushed on the metrics endpoint. Edit $OPENIDM/conf/metrics.json and enable the metrics by setting enable to true.
   {

    "enabled" : true,

    "prometheusRole" : "&{openidm.prometheus.role}"

}

4. Once turned on you can test and see that you get metrics output on the endpoint with the following curl command.

curl -X GET \

 'http://localhost:8080/openidm/metrics/api?_queryFilter=true' \

 -H 'content-type: application/json' \

 -H 'x-openidm-password: openidm-admin' \

 -H 'x-openidm-username: openidm-admin'

Run Load Against IDM Server

To create metrics data on your IDM server, run a reconciliation on your

{project.location} directory. Should you need some data quickly, take a look at the $OPENIDM/sample/sync-with-ldap sample.

Prometheus Setup

Assuming Prometheus is installed as outlined in the product documentation, configure Prometheus, to pull metrics from your IDM server.

If you set up a new instance of an IDM server, you could use the following `scrape_configs` settings in your Prometheus configuration file, `prometheus.yml`:

...

scrape_configs:

   # The job name is added as a label `job=<job_name>` to any timeseries scraped from this config.

 - job_name: 'openidm'

   scrape_interval: 15s

   scrape_timeout: 5s

   metrics_path: 'openidm/metrics/prometheus'

   # scheme defaults to 'http'.

   scheme: http

   basic_auth:

     username: 'prometheus'

     password: 'prometheus'

   static_configs:

     - targets: ['localhost:8080']

...

Start Prometheus.

./prometheus

Once Prometheus starts, check that Prometheus does retrieve data. The default URL to access Prometheus is http://localhost:9090.

To troubleshoot any problems, try these steps:

• Check the Prometheus URL on a IDM server to make sure the metrics are published. For example, browse <http://localhost:8080/openidm/metrics/prometheus>, authenticating with username `prometheus` and password `prometheus`.
• Check that the IDM targets are up using the Prometheus targets page. The default URL to browse is <http://localhost:9090/targets>.
• Check that the metrics are found using the Prometheus graph page. The default URL to browse is <http://localhost:9090/graph>. You should find a long list of `idm_*` metrics in the drop-down list.

Grafana Setup

Assuming Grafana is installed according to the details on the Grafana website and you are on a mac having installed using Brew. Brew is easily installed using the following command:

/usr/bin/ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)"

Go ahead and start Grafana.

brew services start grafana

Grafana by default is located on http://localhost:3000. Point your browser there and login using admin:admin as credentials.

Add a default data source that scrapes data from Prometheus. If you used the default settings, then use these data source settings:

```

Name:     ForgeRockIDM

Type:     Prometheus

Default:  (checked)

URL:      http://localhost:9090

Access:   proxy

Auth:     Basic Auth (checked)

         username: prometheus

         password: prometheus

```

• Import the sample IDM dashboard into Grafana, selecting ForgeRockIDM as the data source.

• (Optional) Generate more load on the IDM server. The dashboard updates the graphs as the data changes.

A emailNotification workflow revisit in IDM 6.0

Workflow continues to be a topic that intrigues people i meet in the field deploying ForgeRock IDM.

This blog post will illustrate out to configure IDM 6.0 to enable the embedded workflow engine, discuss some tools that are available and also build and deploy a simple workflow that can be deployed that sends a simple email Notification.

First off, the workflow module is no longer enabled by default and requires you to configure IDM 6.0 to enable it. By default IDM 6.0 gets deployed using an embedded ForgeRock Directory Server and since the workflow engine can't persist workflow and business process state in the DS, it needs a separate RDBMS. For this purpose, the in-memory database H2 is still embedded and automatically gets utilized for this purpose.

IDM 6.0 does support using a external DS as repo, even in production, but if you are considering using the Workflow module in production, you should also ensure you have a supported RDBMS up and running to manage the persisting of workflows.

The Activiti implementation on IDM 6.0 is unfortunately based on an older version of Activiti and the documentation ForgeRock points out on Alfresco/Activiti's website reference the Activiti Desinger for Eclipise, which is no longer available. Community members have filed a request to get binaries back but so far only the source code is available and should you require this plugin, you need to build it from scratch.

The URL to access the Activiti Desinger is available on GitHub at https://github.com/Activiti/Activiti-Designer.

Despite the Activiti engine not being a par with the latest and greatest from Alfresco/Activiti it is still one of the most important and widely deployed components for ForgeRock IDM.

This Blog post deals with the topic of configuring IDM 6.0 to enable the Workflow engine and to build out a simple email Notification that connects to an SMTP and sends an email. A Launch form will be used where the business process invoking user is able to provide parameters to the workflow.

To follow this little exercise i'm assuming some prerequisites.

1.) ForgeRock IDM 6.0 is installed, up and running. 
Latest version should be available from http://backstage.forgerock.com

2.) You have a Fake SMTP server up and running on localhost. 
You just need a fake service that will accept emails. I would recommend getting FakeSMTP from http://nilhcem.com/FakeSMTP/download.html, install and start this.

3.) That a BPMN 2.0 Editor is installed and can be leveraged. 

Should you require a BPMN 2.0 editor i could recommend the Yaoqiang BPMN Editor that has some nice features and is easy to work with. Yaoqiang BPMN Editor can be found and downloaded for free from https://sourceforge.net/projects/bpmn/files/latest/download?source=typ_redirect

The steps we will do are:

1.) Enable and configure IDM 6.0 to enable the workflow engine.
2.) Create an emailNotification workflow
3.) Deploy and test the workflow in IDM 6.0


Enable and configure IDM 6.0 to enable the workflow engine. 
IDM 6.0 is up and running. Log in as the openidm-admin user and orient yourself via the Admin UI to System Preferences and enable workflow.

This creates two json config files that you can study.

1.  $OPENIDM/conf/workflow.json (This is the location where IDM picks up deployed workflows in .bar file format or .xml format)

{

    "useDataSource" : "default",

    "workflowDirectory" : "&{idm.instance.dir}/workflow"

}

2.  $OPENIDM/conf/datasource.jdbc-default.json (DB store for Workflow)

{

    "driverClass" : "org.h2.Driver",

    "jdbcUrl" : "jdbc:h2:file:&{idm.install.dir}/db/activiti/database;MVCC=FALSE;DB_CLOSE_DELAY=0",

    "databaseName" : "activiti",

    "username" : "sa",

    "password" : {

        "$crypto" : {

            "type" : "x-simple-encryption",

            "value" : {

                "cipher" : "AES/CBC/PKCS5Padding",

                "salt" : "XXXXXXXXXXXXXXXXX==",

                "data" : "XXXXXXXXXXXXXXXXX==",

                "iv" : "XXXXXXXXXXXXXXXXX==",

                "key" : "openidm-sym-default",

                "mac" : "XXXXXXXXXXXXXXXXX"

            }

        }

    },

    "connectionTimeout" : 30000,

    "connectionPool" : {

        "type" : "hikari",

        "minimumIdle" : 1,

        "maximumPoolSize" : 5

    }

}

Now create $OPENIDM/workflow directory that is being referred to in the $OPENIDM/conf/workflow.json.

The workflow engine is now enabled and the requirements there for you to be able to deploy workflows. 

Create an emailNotification workflow

Now lets create our simple emailNotification workflow. Purpose of this simple process is to provide a launchform that allows the invoking user to provide some parameters. In this simplistic illustration we will offer the user to provide a toEmail parameter. The rest of the data will be static and hard code. Should you want to you can at your own pace experiment with this sample process to expand on the launch form and the email task to include a more dynamic behavior. 

To create this workflow you need a Start Event, a Service Task and an End Event. If you are using the Yaoqiang BPMN Editor you can just drag these activities out and connect the transitions from Start to Service Task to End.

Rename the Service Task to Email Notification.

1.

2.

3.

Now you need to implement a Launch form. We do this in the Service Task that we have renamed Email Notification. Switch to "Source" mode and insert the necessary XML code to provide input via a form field. Our simple form will just provide a text field that takes input for the variable toEmail.

<startEvent id="startevent1" isInterrupting="true" name="Start" parallelMultiple="false">
      <extensionElements>
        <activiti:formProperty id="toEmail" name="To Email:" variable="toEmail" writable="true"/>
      </extensionElements>
      <outgoing>flow1</outgoing>
      <outputSet/>
    </startEvent>

Next up we want to set up the Service Task to send emails and define some the required parameters with some static values. 

<serviceTask activiti:type="mail" completionQuantity="1" id="mailtask1" implementation="##WebService" isForCompensation="false" name="Email Notification" startQuantity="1">
      <extensionElements>
        <activiti:field expression="${toEmail}" name="to"/>
        <activiti:field expression="no-reply@forgerock.com" name="from"/>
        <activiti:field name="text">
          <field>
            <activiti:string>
              <string><![CDATA[Here is a simple Email Notification from ForgeRock IDM.]]></string>
            </activiti:string>
          </field>
        </activiti:field>
        <activiti:field name="subject">
          <field>
            <activiti:string>
              <string><![CDATA[Simple Email Notification]]></string>
            </activiti:string>
          </field>
        </activiti:field>
      </extensionElements>
      <incoming>flow1</incoming>
      <outgoing>flow2</outgoing>

    </serviceTask>

Now we have fully implemented the necessary components for this exercise and we are ready to deploy and test the workflow. 

Save the file somewhere temporarily and then manually copy the file to $OPENIDM/workflow/.

Make sure the FakeSMTP is up and running. 

Login to the self-service interface of IDM 6.0 and you will discovered that the emailNotification workflow is available at the bottom of the dashboard. Expand details and take it for a spin!

If everything is correctly assembled, you should be getting an email to the specified email address you defined in the launch form when invoking the process. 

-=[ THE END ]=-

Appendix - The actual BPMN definition. 

<?xml version="1.0" encoding="UTF-8"?>

<definitions

 xmlns="http://www.omg.org/spec/BPMN/20100524/MODEL"

 xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"

 xmlns:activiti="http://activiti.org/bpmn"

 xmlns:bpmndi="http://www.omg.org/spec/BPMN/20100524/DI"

 xmlns:omgdc="http://www.omg.org/spec/DD/20100524/DC"

 xmlns:omgdi="http://www.omg.org/spec/DD/20100524/DI"

 typeLanguage="http://www.w3.org/2001/XMLSchema"

 expressionLanguage="http://www.w3.org/1999/XPath"

 targetNamespace="http://www.activiti.org/test">

 <process id="EmailNotification" name="emailNotification">

   <documentation>Simple Email Notification Task</documentation>

  

   <startEvent id="startevent1" name="Start">

   

   <extensionElements>

    <activiti:formProperty name="To Email:" id="toEmail" variable="toEmail" writable="true" />

  </extensionElements>

   

   </startEvent>

   <sequenceFlow id="flow1" name="" sourceRef="startevent1"

     targetRef="mailtask1"></sequenceFlow>

   <endEvent id="endevent1" name="End"></endEvent>

   <sequenceFlow id="flow2" name="" sourceRef="mailtask1"

     targetRef="endevent1"></sequenceFlow>

   <serviceTask id="mailtask1" name="Email Notification"

     activiti:type="mail">

     <extensionElements>

       <activiti:field name="to" expression="${toEmail}"></activiti:field>

       <activiti:field name="from" expression="no-reply@forgerock.com"></activiti:field>

        <activiti:field name="text">

          <activiti:string><![CDATA[Here is a simple Email Notification from ForgeRock IDM.]]></activiti:string>

        </activiti:field>

        <activiti:field name="subject">

          <activiti:string><![CDATA[Simple Email Notification]]></activiti:string>

        </activiti:field>

      </extensionElements>

   </serviceTask>

 </process>

 <bpmndi:BPMNDiagram id="BPMNDiagram_EmailNotification">

   <bpmndi:BPMNPlane bpmnElement="EmailNotification"

     id="BPMNPlane_EmailNotification">

     <bpmndi:BPMNShape bpmnElement="startevent1" id="BPMNShape_startevent1">

       <omgdc:Bounds height="35" width="35" x="170" y="250"></omgdc:Bounds>

     </bpmndi:BPMNShape>

     <bpmndi:BPMNShape bpmnElement="endevent1" id="BPMNShape_endevent1">

       <omgdc:Bounds height="35" width="35" x="410" y="250"></omgdc:Bounds>

     </bpmndi:BPMNShape>

     <bpmndi:BPMNShape bpmnElement="mailtask1" id="BPMNShape_mailtask1">

       <omgdc:Bounds height="55" width="105" x="250" y="240"></omgdc:Bounds>

     </bpmndi:BPMNShape>

     <bpmndi:BPMNEdge bpmnElement="flow1" id="BPMNEdge_flow1">

       <omgdi:waypoint x="205" y="267"></omgdi:waypoint>

       <omgdi:waypoint x="250" y="267"></omgdi:waypoint>

     </bpmndi:BPMNEdge>

     <bpmndi:BPMNEdge bpmnElement="flow2" id="BPMNEdge_flow2">

       <omgdi:waypoint x="355" y="267"></omgdi:waypoint>

       <omgdi:waypoint x="410" y="267"></omgdi:waypoint>

     </bpmndi:BPMNEdge>

   </bpmndi:BPMNPlane>

 </bpmndi:BPMNDiagram>

</definitions>

Allowing GDPR change the world into a better place

ForgeRock is in the business of providing the necessary enablers to safeguard consumers personal identity information, yet allowing companies to customize and tailor the user experience and establish and maintain trusted digital relationships with technologies such as progressive profiling during self-service registration and thru out each login.

In the light of all the recent scandals, its it an absolute must for companies to implement state of the art software technology to ensure that the data trusted with companies is safe, secure and declared what the intent behind the collection of this data is all about.

How many of you read the user agreements that big companies forces you to accept for services that you utilises? This could be companies like Spotify, Apple or Facebook.

Often we are talking about 35-40 pages of lawyer-talk that is anything from simple to read and would require most people, young and old, to use magnifying glasses since the font size is so small that its impossible to make any sense of it. Many of these agreements often signs away liability and the right to the photos of your children, the latest barbecue with friends or collected metrics on how fast you ran 5k's over the weekend to the companies behind these services.

In many of the agreement you give away the right to file lawsuits against these companies or engage in any form of class act lawsuits should there be a breach of trust or misuse of the data you have entrusted these companies with.

Cambridge Analytica managed to access not only the once who signed up for the "This is your digital life" application but also their friends connected to them on Facebook. They managed to harvest personal information of possibly somewhere in the range of 50 million identities when the first reports came back in March, and use that to target political campaigns and adds in the United States and Indonesia. Who knows where else these type of activities have been going on with collected data from unknowing users?

As i was reading last night the number of users caught up in the Cambridge Analytica scandal that kicked off the latest round of scrutiny into the world of social networking - now stands at a staggering 87 million. As for the market cap of Facebook, it shed 22 percent of its stock and lost over $100 billion in market cap since the Cambridge Analytica revelations and with it also affected other prominent technology stocks along with it.

The other day i got an email from TaskRabbit, who apparently had a major data breach where personal identity information had been leaked. I must have signed up long tie ago out of curiosity and totally forgot about it - yet trusted them with pockets of my personal digital identity information, which is now in the hands of third parties.

Is private information safe with companies such as Facebook? What have we actually agreed to share and with whom and for what purpose?

Your personal identity information is your asset and companies profit on that. Everywhere you go, you leave an exhaust of personal information. Your smartphone tracks every step you go, every place you visit combine that with wearables such as Fitbit and smart watches where vital body metrics get captured and sent off.

I would argue that no consumer knows what their data is used for and with whom its shared and trust in these companies are dropping everyday.

Currently there is a stand-off happening, but fortunately things are about to change, atleast for european consumers with the enforcement of GDRP (the General Data Protection Regulation) on May 25th, 2018. Hopefully this will stir up this stand of and give control back to consumers.

GDRP is a great reminder to businesses that people lend their information and that organizations have a responsibility to look after it. It is not just about confidentiality, it is also about integrity, accuracy and availability, all in all being summarized to what should be and in my ears, is good business practice. You want to do your business with companies that can be trusted - not the sketchy ones that sells your data or shares it with third party or hides what they collect about you.

Companies have purposefully harvested information from their own staff and their customers and included in small print and lawyer talk, their right to reuse that information for purposes not originally intended - this is of course NOT ok!

With all these scandals and breaches of trust occurring back to back on a more or less daily basis, awareness is building among consumers. People are in fact concerned about how their data is being used, that their data is secure and that they have the ability to review what is known about them and that they can choose to opt out as shown by a recent survey published in the Economist.

• 9 out of 10 are concerned about identity theft and fraud!
• 89% are worried abou tthat their data is not kept securely by the providers.
• 3 out of 4 are concerned that small invasions of privacy leads to loss of civil rights. 
• 89% are worried their data is being shared with third parties that they have not consented to share with. 

Consumers demand disclosure from companies on how and why their data is being collected, what its being used for and for what purpose.

Now, even if companies must adapt new processes and implement better controls which can be costly, it is a great opportunity. An opportunity to long term establish healthy and sound, trusted digital relationships with their customers.

• Clear, transparent and accessible information on how you process personal data will lead to public confidence in your organization. (People will tend to trust you more and also be more forgiving)  
• Data volume reduction - as GDRP is being enforced, there will be a drive to reduce the data volumes which will turn into vastly reduced cost and operational inefficiencies associated with keeping masses of redundant and obsolete emails and files on corporate servers - in the cloud or on prem. 
• Data quality - we all know data ages very quickly, even records that are months old can be completely out of date and storing and sifting through this mass of unstructured data mess wastes resource time and storage space. GDPR ensures information is only kept as long as it is valid and for the purpose it was gathered. 
• Security - With data breaches hitting the headlines daily, GDRP will ensure you must adopt better policies with Data under management - benefiting both your reputation and your endu users' data.
  
  and finally....
  
• Trust - As companies adopt better data policies to compy with GDRP, the overall trust level between companies dealing with each others' information will rise. 

So despite what we typically think about regulatory requrements, i personally believe that GDPR will bring something good to the table. Both for consumers who will enjoy a better protection knowing that its a legal requirement to manage their personal identity data safe, secure and with integrity but also from a corporate point of view who wants to act and be seen as serious companies that deal with their customers with out most respect. View GDPR as an opportunity to change the world into a better place!