Unicode functions ​

Unicode functions handle text based on how readers perceive it rather than how systems store it. Use these functions for non-English text, strings that look identical but aren’t equal, and potentially deceptive text from external sources.

Grapheme clusters ​

Grapheme clusters represent user-perceived characters rather than individual code points. A family emoji displays as one character but consists of several code points. An accented letter can be stored as one code point or as a letter and a combining mark.

Use grapheme functions for displays and human-facing character counts, such as when truncating text or enforcing a character limit on an input field.

grapheme_length ​

Counts user-perceived characters (grapheme clusters).

grapheme_length(text)

grapheme_length('👨‍👩‍👧')

1

grapheme_substring ​

Extracts text by grapheme-cluster index without splitting characters that contain multiple code points.

A negative start counts from the end. A start exactly at the end returns an empty string. Any other out-of-range start raises E201.

grapheme_substring(text, start, length)

grapheme_substring('héllo', 0, 2)

hé

grapheme_reverse ​

Reverses a string by grapheme clusters.

grapheme_reverse(text)

grapheme_reverse('abc')

cba

grapheme_lpad and grapheme_rpad ​

Pad a string to a width measured in grapheme clusters.

Use these rather than lpad and rpad for fixed-width output containing non-ASCII text, so the columns line up for a reader.

grapheme_lpad(text, width, pad)
grapheme_rpad(text, width, pad)

grapheme_lpad('42', 5, '0')

00042

grapheme_rpad('42', 5, '0')

42000

Normalization ​

Systems can store the same visible text in different ways. For example, café can contain a single é or an e followed by a combining accent. These representations look identical but aren't equal. This difference can cause a lookup to fail for an existing record.

Normalization converts text to a canonical form so comparisons work.

Normalize to nfc when data arrives, and comparisons downstream behave.

unicode_normalize ​

Converts a string to the given normalization form.

unicode_normalize(text, form)

length('café')

4

length(unicode_normalize('café', 'nfd'))

5

The decomposed form is one unit longer because the accent becomes a separate combining mark. The two forms appear identical on screen.

unicode_normalized? ​

Returns true if the string is already in the given form.

unicode_normalized?(text, form)

unicode_normalized?('café', 'nfc')

true

unicode_normalized?(unicode_normalize('café', 'nfd'), 'nfc')

false

unicode_compare ​

Compares two strings after normalizing both, returning -1, 0, or 1.

unicode_compare is the reliable way to compare text from two different systems. Plain == compares what is stored. unicode_compare compares what is written.

unicode_compare(first, second, form)

unicode_compare('café', unicode_normalize('café', 'nfd'))

0

The two arguments are stored differently and compare as equal, which is the entire point.

Detect deceptive text ​

Text arriving from outside can be built to mislead a person reading it. These functions surface that intent.

unicode_contains_suspicious? ​

Detects text designed to deceive, such as bidirectional overrides, zero-width characters, confusable lookalike characters, Zalgo stacking, or private-use code points.

unicode_contains_suspicious?(text)
unicode_contains_suspicious?(text, options)

unicode_contains_suspicious?('paypal.com')

false

unicode_contains_suspicious?('pаypal.com')

true

unicode_contains_bidi? ​

Detects right-to-left characters.

Right-to-left characters are legitimate in Arabic and Hebrew text. They are also the mechanism behind attacks that make a string display in a different order from how it's stored, so treat a true result as a signal to look rather than an error on its own.

unicode_contains_bidi?(text)

unicode_contains_bidi?('hello')

false

unicode_contains_emoji? ​

Detects emoji.

unicode_contains_emoji?(text)

unicode_contains_emoji?('ok 👍')

true

unicode_contains_emoji?('plain')

false

Japanese text ​

Japanese systems distinguish two kana scripts and two character widths, and often require a specific combination. These six functions convert between them.

hiragana_to_katakana and katakana_to_hiragana ​

Convert between the two kana scripts.

hiragana_to_katakana(text)
katakana_to_hiragana(text)

hiragana_to_katakana('ひらがな')

ヒラガナ

katakana_to_hiragana('カタカナ')

かたかな

to_fullwidth_ascii and to_halfwidth_ascii ​

Convert ASCII characters between full-width and half-width forms.

to_fullwidth_ascii(text)
to_halfwidth_ascii(text)

to_fullwidth_ascii('AB1')

ＡＢ１

to_halfwidth_ascii('ＡＢ１')

AB1

to_fullwidth_katakana and to_halfwidth_katakana ​

Convert katakana between full-width and half-width forms.

to_fullwidth_katakana(text)
to_halfwidth_katakana(text)

to_fullwidth_katakana('ｶﾀｶﾅ')

カタカナ

to_halfwidth_katakana('カタカナ')

ｶﾀｶﾅ

Use case: Normalize names before matching across systems ​

Two systems hold the same customer, but one stored the name decomposed. Use Unicode functions to normalize both names so the match succeeds, and flag anything that shouldn't be matched automatically:

Input

{
  "crm_name": "café",
  "erp_name": "café"
}

Formula

{
  equal_as_stored: _.crm_name == _.erp_name,
  equal_normalized: unicode_compare(_.crm_name, _.erp_name) == 0,
  needs_review: unicode_contains_suspicious?(_.erp_name)
}

Output

{
  "equal_as_stored": false,
  "equal_normalized": true,
  "needs_review": false
}

The two names in the input are not the same bytes. The CRM stored é as one character and the ERP stored it as e and a combining accent. They render identically, and no amount of looking at them will reveal the difference.

A plain == therefore says these are different customers. That incorrect result is the bug, because it creates a duplicate record that then diverges.

Related ​

• String functions: length, substring, lpad and their byte-oriented behavior.
• Encoding functions: Convert text to bytes in a named encoding.
• Common functions: Information about length across data types.
• Error codes: Troubleshoot job failures such as E201.