Very often values printed on documents are not in the desired format or contain unwanted characters such as spaces, commas, colons, etc. Given this, it may be beneficial to transform the values extracted by the AI instead of using the original ones contained in the document.

This is most commonly seen with the removal of non-alphanumerical characters from values such as the VAT, IBAN, and account number.

Value Transformations is a configurable extension that can help with this process by offering powerful configurable string manipulation features that use regular expressions to replace chosen string patterns automatically. Note that this extension cannot perform any arithmetic or date operations.

Advanced users can also chain defined transformations to cover more complex cases when one regular expression is insufficient. It is also possible to chain multiple actions where results of one are projected into the input of another. However, advanced technical expertise isn’t required to use the extension.

Configuration examples for common use cases

Given that it might be complicated to define transformation rules just by using the description of the available parameters (found later in this article), here are some configuration examples that can be copied and modified for an easier time setting up the extension.

Please note that backslashes in regular expressions must be escaped (doubled) in the extension configuration of the Rossum UI. The example configuration below already contains escaped regular expressions.

Also note that any string in action condition expression need to be enclosed in single quotes (see examples for more details).

Examples below show just a sample of capabilities of both regular expressions and python expressions this function used to define transformations and conditions respectively. Full documentation of python regular expressions can be found here and expression documentation can be found here.

Removal of non-alphanumeric characters

The Value Transformations extension with the configuration below removes all non-alphanumeric characters from the Vendor VAT Number and IBAN fields.

Example:

• Input: DE 12345-6789
• Output: DE123456789

{
  "actions": [
    {
      "transformations": [
        {
          "pattern_to_replace": "[^a-zA-Z\\d]",
          "value_to_replace_with": "",
          "replace_if_this_pattern_matches": "[^a-zA-Z\\d]"
        }
      ],
      "source_target_mappings": [
        {
          "source": "sender_vat_id",
          "target": "sender_vat_id_normalized"
        },
        {
          "source": "iban",
          "target": "iban_normalized"
        }
      ]
    }
  ]
}

Extracting and normalizing part of the line item description

Value Transformations with the configuration below uses two chained transformations to extract and normalize item code from the item description.

The first transformation removes everything after the first space character in the string. The second one removes all hyphens from the result of the first transformation.

Notice also that there is an action condition defined in this configuration. This action will only be performed when the Vendor Name is “Great Company“. The condition is optional.

Example:

• Input: 1234-567-89 This is a line item description with the code at the beginning.
• Output: 123456789

{
  "actions": [
    {
      "transformations": [
        {
          "pattern_to_replace": " ([\\s\\S]*)$",
          "value_to_replace_with": "",
          "replace_if_this_pattern_matches": " ([\\s\\S]*)$"
        },
        {
          "pattern_to_replace": "-",
          "value_to_replace_with": "",
          "replace_if_this_pattern_matches": "-"
        }
      ],      
      "action_condition": "{sender_name} == 'Great Company'",
      "source_target_mappings": [
        {
          "source": "item_description",
          "target": "item_code"
        }
      ]
    }
  ]
}

Copying item code to another field for certain line items

Configuration below uses an action condition to check if document_type header field contains value ‘tax_invoice’ and checks all line items if attribute item_code is either ‘M0061’ or ‘M0062’. For line items fulfilling this condition the transformation is applied.

The transformation then removes all other characters than numerical characters and stores the transformed values in a separate field call item_other for each of the matched line item.

Example:

• Input: ‘tax_invoice’ and [M006, M0062]
• Output: [0061, 0062]

{
  "actions": [
      {
        "queue_ids": ["123"],
        "action_condition": "{document_type} == 'tax_invoice' and {item_code} in ['M0061','M0062']",
        "transformations": [
            {
             "pattern_to_replace": "[^0-9]",
             "value_to_replace_with": "",
             "replace_if_this_pattern_matches": ".+",
            }
         ],
         "source_target_mappings": [
            {
             "source": "item_code",
             "target": "item_other"
            }
         ]
      }
   ]
}

Setting up the extension

Setting up the extension itself takes a few simple steps:

1. Prepare your queues and schemas
2. Activate Value Transformations in the Rossum Store
3. Specify the queue(s) the extension is going to be used for
4. Set up the actions and transformations

Step 1: Prepare your queues and schemas

The first step is identifying the queue(s) with the documents that require Value Transformations. Once that’s done, identify the schema IDs of the fields that will contain the extracted values set to be transformed by the extension.

If using the Dedicated Engine, make sure to create new schema fields that will store the results of the transformations (see the info panel below). If the Generic Engine is being used, configure the extension to modify the value of the field “in-place“ (same source and target field).

Please note that by using the Dedicated Engine and configuring the extension to modify the value of a particular field, the results of the accuracy calculation for that field will be significantly lower compared to the real accuracy. To avoid this, modifying the values extracted by the AI and OCR manually or programmatically is not recommended when using the Dedicated Engine.

Step 2: Activate Value Transformations in the Rossum Store

In order to activate Value Transformations, go to the Rossum application and:

1. Click the Extensions button in the main menu and you will be taken to the Rossum Store.
2. Once in the Rossum Store section, Value Transformations should be visible. If not, click “See all”.
3. Click the “Value Transformations” extension tile.
4. Click “Try extension”.

Step 3: Specify the queue(s) the extension is going to be used for

Once in the “Rossum Store extension settings”, scroll down to “Queues” and select the queue(s) that the extension should be used for.

Step 4: Set up the actions and transformations

The extension is configured through the configuration field in the UI or by using the settings attribute of the hook API object. The configuration is in JSON format (see the description of the available parameters below).

This configuration consists of a list of actions that can work with values from different fields in the schema. Each action has a set of transformations, source/target field definitions, and the condition under which the action will be performed.

The code from examples above can be pasted here in case of a matching use case. Then, only step needed is a simple replacement of the fields whose values are to be transformed.

The full list of available settings parameters in table below.

Root	Param name	Description
	actions	List of actions to be performed by the extension. Description of the action parameters is shown below.
actions	source_target_mappings	List of source and target field schema ids.
source_target_mappings	source	Source schema_id of the initial value to be transformed.
source_target_mappings	target	Target schema_id of the target field where the transformed value will be stored.
actions	transformations	List of transformations to be performed on the value of the source field. See description of the transformation parameters below.
actions	queue_ids	List of queue IDs where the particular action should be performed. It is possible to assign the extension to multiple queues and specify multiple actions for different queues in one instance. This parameter is optional. If it is not present in the configuration, then the action will be performed on all the queues that the extension is assigned to unless excluded_queue_ids is set.
actions	excluded_queue_ids	List of queue IDs where the particular action will not be performed. This parameter cannot be set with queue_ids also set. This parameter is optional.
actions	action_condition	Definition of a condition for a particular action. The condition needs to evaluate to bool – True or False. If defined, the action will be evaluated and if True, the action’s transformation will be applied, otherwise it will be skipped. The condition is a python condition where schema fields are referenced by their schema_id enclosed in {}. Example: {item_code} != '' and {document_type} == 'invoice' See python expressions documentation.
actions	additional_user_update_trigger	Additional list of schema_ids that will trigger the action if a field with such name was modified by a user.
actions	allow_target_update	If set to true prevents the action to overwrite the target value when user updates it manually. Default false.
transformations	pattern_to_replace	Regular expression which defines a pattern in the value to be found and replaced. See python regular expressions for details
transformations	value_to_replace_with	The value which will replace all occurrences of the pattern matching the regular expression defined in the pattern_to_replace parameter.
transformations	replace_if_this_pattern_matches	Regular expression which defines the condition for a transformation to be applied. The transformation will only be applied if the value matches the expression. See python regular expressions for details