Field mapping in Elasticsearch defines how a document is indexed and how its fields are indexed and stored. As such, accurate field mapping is crucial for effective analysis and visualization of the data.
Logz.io users can make full use of Elasticsearch’s automatic mapping (Dynamic Mapping) for automatic type and field mapping, and now they also have the ability to manually handle mapping with the introduction of the new Field Mapping feature.
Logz.io Field Mapping is especially useful for resolving mapping errors that may occur when inconsistencies are identified within the data, for example — when a field that was defined in existing mapping as an integer changes into a string.
When these inconsistencies are identified and occur too many times, a mapping error is reported and the problematic fields are flattened into a ‘logzio_removed_field’ field until this error is resolved.
You can view and manage field mapping on the new Field Mapping page, accessed via the Settings page. On this page, you’ll be able to see both existing field mapping definitions and any mapping errors that may have occurred.
Resolving Mapping Errors
Any existing issues are listed in the Mapping Errors tab on the new Field Mapping page, and how you resolve them depends on what caused them.
One case could be if the same field name is found in multiple log types. The resolution in this case is to enter a field alias for the problematic field for one of the log types. This can be done easily under the Field alias column. You can then change the corresponding field type as well under the Mapping column.
Another way of course, is to change the value type for the field in the logs themselves — in this case, let Logz.io know of the change by marking the error as resolved. To do this, just hover over the righthand side of the list and click ‘mark as resolved’. This releases the flattening rule applied to the problematic field.
Mapping errors can also be caused if a field type (e.g. boolean, long, integer) changed after Elasticsearch dynamic mapping occurred at the beginning of the day. You can resolve this by selecting a different mapping type under the Mapping column.
As in the first scenario, you can also apply changes to the logs themselves. Again, mark these errors as resolved once the change has been applied to release the flattening rule put in place by Logz.io.
Mapping changes are applied using the Apply mapping button. In both scenarios, please note that you can only apply field mapping changes up to five times a day.
Breaking and Forcing Changes
Changing field mapping can directly influence other log types, visualizations and dashboards. As such, we have placed safeguard measures to make sure these changes are applied carefully without causing damage:
- Changing a mapping type is disabled in cases where a specific field type exists in another log type in the same Logz.io account. To circumvent this block, you can either enter a field alias or force a change. Note, forcing the change may result in a parsing failure or break Kibana objects relying on this field.
- Entering a field alias is disabled in cases where only one field with this name is indexed.
To help you understand the impact of your mapping changes in linked Kibana objects, you can click the pie chart icon on the righthand side of the table to view a list of the affected objects.
Multi-Layered Data Enhancement
Enriching log data is a critical element in any logging pipeline. Logz.io provides automatic data parsing and enrichment for various log types but also provides users with a multi-layered system for manually massaging their log data as they see fit.
Data Parsing was introduced earlier this year and allows users to manually define parsing for logs. Data Parsing also defines field mapping but these two features should not be mistaken as alternatives to each other but as complementary tools.
Data Parsing was designed for setting up initial parsing rules for logs. Field Mapping was designed to help users deal with mapping conflicts following later changes to logs. In any case, the latest mapping configuration made takes precedence. For example, if you set up a parsing rule using Data Parsing, and subsequently changed a field name or type in Field Mapping, this latter change overrules the configurations made earlier.