Encoding functions ​

Encoding functions convert between text and bytes using a named character encoding, and handle the encoded-word form used in email headers.

These functions matter when a system doesn't speak UTF-8, such as a mainframe export in Shift_JIS, a legacy CSV in Windows-1252, or an email header carrying an accented name. A wrong encoding produces the garbled text that shows up as é where an é should be.

Text to bytes and back ​

The following functions convert between text and bytes with a specified character encoding:

encode_string ​

Converts a string to Binary using a named encoding.

Use encode_string instead of Binary() when text can contain non-ASCII characters. Binary() raises E008 for non-ASCII input instead of selecting an encoding.

encode_string(text, encoding)

encode_string('café', 'UTF-8')

0x"636166C3A9"

The é occupies two bytes, C3A9, so the 4-code-point string café encodes to 5 bytes. length counts code points, grapheme_length counts user-perceived characters, and byte_length counts bytes. The three can differ for anything outside ASCII.

decode_string ​

Converts Binary to a string using a named encoding.

decode_string(binary, encoding, options)

An invalid sequence raises an error by default. Silently replacing bytes instead would turn a data problem into a corrupted record that nobody notices.

decode_string(encode_string('café', 'UTF-8'), 'UTF-8')

café

decode_string_by_content_type ​

Decodes bytes using the charset named in an HTTP Content-Type header, rather than one you supply.

Use it when handling an HTTP response whose encoding you don't control.

decode_string_by_content_type(binary, content_type, options)

Check before converting ​

The following functions test whether text or bytes support a specified encoding:

ascii_string? ​

Returns true if the string contains only ASCII characters.

ascii_string?(text)

ascii_string?('plain')

true

ascii_string?('café')

false

convertible_to_encoding? ​

Returns true if the given encoding can represent the string without losing anything.

Check convertibility before sending text to a system with a restricted character set. It's better to reject a record here than to deliver one with characters silently replaced.

convertible_to_encoding?(text, encoding)

convertible_to_encoding?('café', 'ASCII')

false

valid_string? ​

Returns true if the bytes are valid in the given encoding and can be decoded.

valid_string?(binary, encoding)

valid_string?(Binary('AB'), 'UTF-8')

true

encodings ​

Lists the supported encoding names. Pass true to include synonyms.

encodings(include_synonyms)

Email headers ​

An email header can only carry ASCII. RFC 2047 wraps anything outside that range in encoded-words that look like =?UTF-8?B?...?=.

encode_mime_header ​

Encodes text as RFC 2047 encoded-words for an email header.

encode_mime_header(text, encoding, mode)

encode_mime_header('café', 'UTF-8')

=?UTF-8?B?Y2Fmw6k=?=

decode_mime_header ​

Decodes RFC 2047 encoded-words in an email header back to text.

decode_mime_header(text)

decode_mime_header('=?UTF-8?B?SGVsbG8=?=')

Hello

Use case: Prepare a name for a legacy ASCII-only system ​

The destination exclusively accepts ASCII. Use convertible_to_encoding? to check each name before sending, and route the ones that won't fit rather than delivering corrupted text:

Input

{
  "names": ["Kalani Park", "Noam Gupta", "José Álvarez"]
}

Formula

let parts = _.names >> partition_by(n ~> convertible_to_encoding?(n, 'ASCII'))
do {
  send: parts >> at(0),
  needs_review: parts >> at(1)
}

Output

{
  "send": ["Kalani Park", "Noam Gupta"],
  "needs_review": ["José Álvarez"]
}

Check convertibility before transmission so a person can correct the record instead of receiving corrupted text such as Jos? lvarez.

Related ​

• Binary functions: Base64 and hex, which need no character encoding.
• Unicode functions: Normalization and grapheme-aware text handling.
• Escape functions: Make a value safe inside another format.
• Error codes: Troubleshoot job failures such as E008.