Hidden Character Detector

What Are Hidden Unicode Characters?

Hidden characters are Unicode characters that have no visible glyph but still exist within text. They consume data storage, influence text layout and rendering, and can cause hard-to-diagnose bugs in programming, data processing, and security systems.

Unicode defines hundreds of control and formatting characters, but the most commonly encountered fall into three groups: zero-width characters (invisible spacers), special whitespace variants, and bidirectional (bidi) control characters used for right-to-left scripts.

Common Hidden Character Types

1. Zero-Width Characters

U+200B (Zero-Width Space), U+200C (ZWNJ), and U+200D (ZWJ) are the three most dangerous invisible characters. They take up zero visual space but alter text processing behavior. ZWJ is legitimately used in compound emoji sequences (family groups, flag sequences), but it is also exploited to create phishing URLs that appear identical to legitimate ones.

2. Byte Order Mark (BOM)

U+FEFF was originally designed to signal byte order in UTF-16 files. Today, it most often appears accidentally at the start of UTF-8 files saved by certain Windows text editors. A BOM in a JSON or CSV file causes parser errors that are extremely confusing because the first character of the file is invisible rather than the expected { or header text.

3. Bidirectional (Bidi) Characters

RTL Override (U+202E) forces text to render right-to-left. This is a well-known attack vector: a file named invoice_‮fdp.exe displays as invoice_exe.pdf on many operating systems, tricking users into executing a malicious binary they believe is a harmless PDF.

4. Special Whitespace

Non-Breaking Space (U+00A0), En Space, Em Space, Thin Space, and Hair Space look like regular spaces but have different widths and break behaviors. They commonly appear when copying text from PDFs, Word documents, or richly formatted web pages. String comparison fails silently: "hello world" with NBSP is not equal to "hello world" with a regular space, despite looking identical on screen.

Hidden Characters in Security

Invisible characters are a favored weapon in phishing and social engineering attacks. Common techniques include:

• Homograph attacks: Combining hidden characters with Unicode look-alikes of Latin letters to forge domain names.
• RTL override: Reversing the displayed file extension to hide the true format (.exe masquerading as .pdf).
• Zero-width injection: Inserting ZWSP into usernames or passwords to bypass blacklists or security filters.
• Steganographic watermarking: Embedding a unique combination of invisible characters in documents to trace the source of leaks.

Prevention in Code

• Always normalize input with String.prototype.normalize() before processing.
• Use a regex to strip control characters: /[​-‍­ ]/g
• Validate file BOM before parsing JSON/CSV — strip  from the first byte.
• Display codepoints (U+XXXX) alongside each character when debugging text processing.
• Use an editor with visible whitespace mode (VS Code: Toggle Render Whitespace).
• Add a CI/CD pipeline step to check for hidden characters in source code and config files.