Add Data to Discover

Discover is part of the Knowledge Discovery product suite and you add data using Knowledge Discovery NiFi Ingest. You can use a new or existing NiFi deployment.

IDOL NiFi Ingest includes a processor named "PutIDOLGraph" that accepts IDOL documents, such as those that are produced by Knowledge Discovery connectors. In your NiFi deployment, add this processor to your ingest pipeline. The processor runs an XSL transformation, to transform the documents it receives into XML that describes entities and links to add to Discover.

When configuring the PutIDOLGraph processor you must provide the host name and port of the Discover API. You must also configure standard properties such as an IDOL License Service. The following sections provide more detail about the XSL transformation that you must write, and about how to configure authentication with the Discover API.

The XSL Transformation

The XSL transformation runs once for each IDOL document that is received by the PutIDOLGraph processor. It must produce XML that looks like this:

Copy

<Entities>
    <Entity>
        <typeId>[Entity type]</typeId>
        <access>
            <public>[true|false]</public>
            <filter>typeId==TeamEntity and properties:TeamEntity:title=contains=Global</filter>
            <filter>...</filter>
            ...
            <id>[EntityID]</id>
            <id>...</id>
            ...
            <idolAcl>:U:...:G:...</idolAcl>
        </access>
        <properties>
            <[property name] type="[string(default)|number|boolean|null]" key="[true|false(default)]">[property value]</[property name]>
            ...
        </properties>
        <linkedEntity>
            <entity>
                <typeId>[Entity type]</typeId>
                <properties>
                    <[property name] type="[string(default)|number|boolean|null]" key="[true|false(default)]">[property value]</[property name]>
                    ...
                </properties>
            </entity>
            <link type="[upstream|downstream]">
                <typeId>Contains</typeId>
                <properties>
                    <[property name] type="[string(default)|number|boolean|null]">[property value]</[property name]>
                    ...
                </properties>
            </link>
        </linkedEntity>
        <file>
            <properties>
                <[property name] type="[string(default)|number|boolean|null]" key="[true|false(default)]">[property value]</[property name]>
                ...
            </properties>
            <link type="[upstream|downstream]">
                <typeId>Preview</typeId>
                <properties>
                    <[property name] type="[string(default)|number|boolean|null]">[property value]</[property name]>
                    ...
                </properties>
            </link>
        </file>
        ...
    </Entity>
</Entities>

The Entity element defines a single entity to add to the Discover graph. Your XSL transformation can create as many Entity elements as necessary. Within each Entity element:

  • The typeId element defines the entity type to add. This must match one of the entity types defined in your entity schema.
  • The access element defines who can access the new entity. This element is optional, because you can configure default access permissions in the processor properties. Setting this element through the XSL transform allows you to override the default access permissions for individual entities, based on data in the IDOL document. There are several child elements that you can use to define access permissions:

    • public - when this element has the value true, the new entity is public.
    • filter - a filter that selects a single entity, for the new entity to inherit access setting from.
    • id - use one or more id elements to provide a list of entity IDs for the new entity to inherit access settings from.
    • idolAcl - an IDOL ACL defining users and groups who can access the new entity. This cannot be used in conjunction with any other access properties.
  • The properties element defines properties to add to the new entity. The available properties and their types depend on the entity schema that you configured.
  • The linkedEntity element defines links to or from the new entity. Entities that you list here can be created if they do not already exist (this can be configured in the processor properties). You can assign additional properties to both the linked entities and the links themselves.
  • The file element is optional and can be used to created a linked file entity (an entity that can have an associated binary file).

Configure Authentication

To configure authentication with the Discover API, use the OAUTH SETUP tab in the advanced configuration dialog.

You must enter the following information:

  • AppKey - the value of the environment variable ISOL_AUTH_SERVICE_API_CLIENT from base.env (see Configure the Application).
  • AppSecret - the value of the environment variable ISOL_AUTH_SERVICE_API_CLIENT_SECRET from base.env (see Configure the Application).
  • Username and Password - the credentials of a user that you created in the system (see Manage Users).
  • TokenURL - the URL to use to obtain a token from KeyCloak. For example: https://discover-host.example.com:8010/realms/discover/protocol/openid-connect/token.

After entering these details, click SUBMIT. The processor will obtain and store OAuth tokens so that it can communicate with the Discover API.