Map functions ​

A Map is a set of key-value pairs: the WEL type for a JSON object. Map functions read its keys and values, combine one map with another, remove fields, and convert between a map and a list so that list functions can operate on it.

Keys are always String. Insertion order is preserved, so a map built in a deliberate order serializes in that order.

_ is the input your expression receives in these examples. Refer to Input variables for more information.

Call a map function ​

Call a function directly, or pipe a value into it with >>:

keys(_.customer)
_.customer >> keys

Read a field with a period (.), or with brackets ([]) when the key is computed or isn't a bare identifier:

_.customer.email
_.customer['email']

Reading a key that doesn't exist raises E104. Use has_key? to test first, or the try-fallback operator |? to supply a default:

_.customer.nickname |? 'none'

Read keys and values ​

The following functions read a map's keys and values without changing it:

keys ​

Returns the keys as a list, in insertion order.

keys(map)

keys({sku: 'WID-1', qty: 3})

["sku", "qty"]

values ​

Returns the values as a list, in insertion order. Values that are Skip are omitted.

values(map)

values({sku: 'WID-1', qty: 3})

["WID-1", 3]

has_key? ​

Returns true if the key is present.

A test for the key isn't the same as a test for a value. A key present with a null value returns true here. Use present? instead to test for a usable value.

has_key?(map, key)

has_key?({sku: 'WID-1'}, 'sku')

true

has_key?({sku: 'WID-1'}, 'qty')

false

Choose fields ​

The following functions narrow a map to a smaller, safer set of fields:

pluck ​

Returns a map containing only the named keys. Keys that aren't present are skipped rather than raising an error.

Use it to narrow a large inbound record to the fields a destination actually accepts.

pluck(map, keys)

pluck({id: 1, secret: 'x', name: 'a'}, ['id', 'name'])

{id: 1, name: "a"}

pluck({id: 1}, ['id', 'missing'])

{id: 1}

except ​

Returns a map with the named keys removed.

pluck and except are opposites, and the choice between them matters for safety. pluck is an allowlist, so a new field appearing upstream is dropped by default. except is a denylist, so a new field is forwarded by default. Prefer pluck when leaking a newly added field is the concern.

except(map, keys)

except({id: 1, secret: 'x', name: 'a'}, ['secret'])

{id: 1, name: "a"}

Use case: Drop sensitive fields before forwarding ​

An inbound user record carries credentials that shouldn't reach the destination system. Use except to drop the sensitive fields before forwarding the record:

Input

{
  "user": {
    "id": "U-1",
    "email": "[email protected]",
    "password": "hunter2",
    "ssn": "111-22-3333"
  }
}

Formula

_.user >> except(['password', 'ssn'])

Output

{"id": "U-1", "email": "[email protected]"}

This use of except is the denylist form, and it's only safe while you know every field the source can send. pluck(['id', 'email']) is the safer expression of the same intent if the source may add fields.

Combine maps ​

The following function combines two maps into one:

merge ​

Combines two maps. The second wins when both define the same key.

merge(first, second)

merge({sku: 'WID-1', qty: 1}, {qty: 5})

{sku: "WID-1", qty: 5}

Use case: Apply overrides to a set of defaults ​

A set of defaults needs caller-supplied overrides applied on top, so anything supplied wins and anything omitted falls back. Use merge with the defaults first and the caller's values second:

Input

{
  "overrides": {"currency": "EUR", "notify": true}
}

Formula

merge(
  {currency: 'USD', locale: 'en-US', notify: false},
  _.overrides
)

Output

{"currency": "EUR", "locale": "en-US", "notify": true}

merge combines one level. A nested map in the second argument replaces its counterpart outright rather than being merged into it.

Convert between maps and lists ​

These four functions are how a map reaches the list functions. entries and pairs go one way. from_entries and from_pairs come back.

entries ​

Converts a map to a list of {key, value} maps.

entries(map)

entries({sku: 'WID-1', qty: 3})

[{key: "sku", value: "WID-1"}, {key: "qty", value: 3}]

from_entries ​

Converts a list of {key, value} maps back to a map. The inverse of entries.

from_entries(list)

from_entries([{key: 'sku', value: 'WID-1'}])

{sku: "WID-1"}

pairs ​

Converts a map to a list of two-element [key, value] lists. Pairs whose value is Skip are omitted.

pairs(map)

pairs({sku: 'WID-1', qty: 3})

[["sku", "WID-1"], ["qty", 3]]

from_pairs ​

Converts a list of two-element [key, value] lists back to a map. The inverse of pairs.

from_pairs(list)

from_pairs([['sku', 'WID-1']])

{sku: "WID-1"}

{a: 1, b: 2} >> filter_by((k, v) ~> v > 1)

Flatten nested maps ​

The following function collapses a nested map into a single level, joining the path into one key:

flatten_map ​

Collapses nested maps into a single level, joining the path into one key.

The walk is depth-first in insertion order. Only maps are descended into. A list is treated as a leaf value and kept whole. An empty nested map is also a leaf.

Two nested keys can collapse onto the same flattened key. The last write wins and an E229 warning is emitted when that happens, so the collision is visible in the job rather than silent.

flatten_map(map, options)

An unrecognized option key raises E222 rather than being ignored.

flatten_map({customer: {name: 'Nur', address: {city: 'London'}}})

{"customer.name": "Nur", "customer.address.city": "London"}

flatten_map({a: {b: 1}}, {sep: '_'})

{a_b: 1}

Use case: Flatten a record for a CSV row ​

A CSV destination needs one column per field, but the source is nested two levels deep. Use map functions to flatten the record into a single level:

Input

{
  "record": {
    "id": "C-1",
    "customer": {
      "name": "Nur",
      "address": {"city": "London", "postcode": "E1"}
    }
  }
}

Formula

_.record >> flatten_map

Output

{
  "id": "C-1",
  "customer.name": "Nur",
  "customer.address.city": "London",
  "customer.address.postcode": "E1"
}

The flattened keys are exactly the column headers the destination expects. Pass {sep: '_'} if the target doesn't accept dots in header names.

Related ​

• List functions: What to do with entries, pairs, keys, and values.
• Deep map functions: Reach through nesting rather than flattening it.
• Data types: Map, Skip, and null.
• Error codes: Troubleshoot job failures such as E104, E222, and E229.