Having two data sources with the same exact schema is rare. When you integrate vendor data into BriteCore, you need a way to map objects between the two domains and transform the data for integration and reporting purposes.

For example, BriteLines is essentially a product modeler for insurance products. Consider the schema as the visible data model you can view from the BriteLines interface, not the actual tables powering BriteLines in our own database. A rough and incomplete example of the BriteLines schema would be:

You might have a BriteCore field located under:

For the purposes of a VINMaster lookup, you may want to output the vehicle_identification_number field to vin for a specific application. Data maps that allow you to persistently map a field in BriteLines should be canonically treated as vin for a specific purpose.

Instead of hardcoding the data mapping into your UI, BriteDataMapping allows you to make API calls to encode and execute mapping logic over HTTP. This works by taking a source object, a mapping schema, running the source object through mapping, and finally producing the output object. It works on lists and deeply nested data structures as well as flat files.

There are three broad steps involved when using BriteDataMapping:

1. Map data objects between domains: Refer to the VINMaster lookup example above.
2. Encode the logic: You make POST /mappings/ requests to create a map of your data objects..
3. Execute the transformation: Then you make POST /map/?from=&to= requests to execute the logic against a given piece of input data.

The service basically transforms a given JSON object produced within a certain domain using a mapping to produce a JSON object for another domain.

This tutorial will walk you through the steps and considerations to apply when implementing BriteDataMapping for your integrations.

Step 1: Map the data objects between domains

To map the data objects, you need to:

1. Retrieve a JSON data object from the vendor.
2. Retrieve your BriteCore line output to compare with the vendor data object. You need to evaluate the field_answers from BriteLines interface; alternatively, you can make an API call to retrieve your product definition.
3. Compare the two JSON data objects.

Step 2: Use Data Mapping to create a map for your data objects (encode the logic)

There are two possible modes to map your data:

1. Default mapping mode: Defines a number of operations to apply to the data and returns all of the data. Ideal for basic data structures.
2. Template mode: Renders the data to a template. Recommended for most use cases and more complex data structures, such as lists and deeply nested data structures.

To differentiate between the two modes, add mode= to the query string of the map request.

Note: The default mapping mode will apply if you don’t specify the mode in your map request.

Example 1: Create a mapping in default mapping mode

Use the default mapping mode only for basic data structures, such as flat files to flat files; for example, if you have data from a payment system and want to store that data in your accounting system.

The payment object looks like:

However, the accounting system expects first_name to be in camel case, i.e., firstName.

You will need to transform the key of first_name to firstName and produce the following:

To create a mapping, we make a POST request to /mappings/ with the request body having the following schema:

{
  "data": {
    "type": "mapping",
    "attributes": {
        "from": "<from>",
        "to": "<to>",
        "mappings" : {
            "<path_expression>" : { "operation": "<operation>", "operand" : "<operand>",  "key" : "<key>",  "scope" : "<Transform>"}
        }
    }
  }
}

Sample request

The actual request to define the mapping would be:

{
  "data": {
    "type": "mapping",
    "attributes": {
        "from": "payment_system",
        "to": "accounting_system",
        "mappings" : {
            "first_name" : { "operation": "map_key", "operand" : "firstName" }
        }
    }
  }
}

The response sample for the above payload:

Example 2: Create a mapping in template mode

Template mode applies a Jinja template—specified in the template key of the mapping—to the provided data.

This mode provides you flexibility in defining completely new JSON data structures from the source data.

The template must be a valid Python data structure as a string. That is then evaluated with literal_eval and finally returned as JSON.

Example template:

Returned data:

Sample template mapping

{
  "data": {
    "type": "mapping",
    "attributes": {
        "from": "<from>",
        "to": "<to>",
        "mappings" : {
            "template" : "<template as a big string>"
        }
    }
  }
}

Jinja Environment details

• Your data is provided to the template as source.
• OrderedDict: The OrderedDict collection is available as a global.
• regex sub: The regex’s sub function is available as a filter, i.e., sub (“”, “”).

Sample Jinja details

"""\
{% set typeLabels = OrderedDict([
    ('type1', 'Label for type 1'),
    ('type2', 'Label for type 2'),
    ('type3', 'Label for type 3')
]) %}
{
    "name": "{{source.name}}",
    "things": [
        {% for item in source.things %}
          {% if item.a < 3 %}
            {{item.a}}{% if loop.nextitem is defined %},{% endif %}
          {% endif %}
        {% endfor %}
    ],
    "street": "{{source.address|sub("[^0-9]", "")|trim}}",
    "state": "{{source.state.split('_')[1][0:2].upper()}}",
    "claimAmt": "{{source.claims|sum(attribute="amt")}}",
    "claimTypes": [
        {% for claim in source.claims %}
          {% if claim.type %}
            {{typeLabels[claim.type]}},
          {% endif %}
        {% endfor %}
    ],
}"""

Use the from and to in the attributes when creating the mapping. You may include a version.

The returned data will be a transformed/mapped version of the data provided.

Data Mapping details

Versioning

• The id of each mapping consists of the from, to, and version attributes, separated by @. For example: my-payments-system@accounting@version@1
• Each version is a separate mapping.
• Create a new version by creating a new mapping using the same data, but with a different version.
• Once a mapping is created, changing the from, to, or version won’t change the id or be available to use.

Transforms

A mapping consists of key-value pairs that can be thought of as a single unit, called a Transform.

Each Transform has two objects:

• path_expression
• Operation

Path_expresion

A path_expression allows you to describe the location of the input data that you’re interested in mapping.

Operation

• <operation>: Describes the kind of transformation you can make to the input data.
• <operand>: A value that you provide to the operation.
• <scope>: A scope is another Transform that, after evaluation, is used as a value to the operation instead of the value of the path_expression value in the source data.
• <key>: The key at which you want to return this data.

Table 1 summarizes:

• Supported operations.
• Type of any required operand.
• Result of the operation on the example data.
  • Example data: {"some_field": "some_value"}

Table 1: BriteDataMapping supported operations

Example data:

{"some_field": "some_value"}

Return types

The service first converts the JSON data into Pythonic types and then operates on that data. Therefore, the return types are dependent upon the input data.

Recursive mapping

Using scopes, you can recursively transform the data. This allows you to combine operations as well.

For example, if you wanted to both prefix and convert to uppercase (upper), you need to use a scope and a special operation called recursive_map.

This will take three Transforms:

• Prefix Transform: {"a": {"operation": "prefix", "key": "C", "operand": "my_"}}
• Upper Transform:{"a": {"operation": "upper", "key": "C", "scope": }}
• Recursive Transform: {"a": {"operation": "recursive_map", "key": "B", "scope": }}

Example of recursive mapping

{
  "a": {
    "operation": "recursive_map",
    "key": "B",
    "scope": {
      "a": {
        "operation": "upper",
        "key": "C",
        "scope": {
          "a": {
            "operation": "prefix",
            "key": "C",
            "operand": "my_"
          }
        }
      }
    }
  }
}

Resulting data after mapping

Plugins

BriteCore’s UI Plugins client has a mapping request tool, BriteCorePluginRequest.makeMappingRequest (from, to, data, mode), which simplifies BriteDataMapping usage from plugins.