PEP 672 — Unicode-related Security Considerations for Python

Most of this information is directly from Python.org specifically here. But I felt like the more people that read this – the quicker this issue will be resolved. Some of the information here can be daunting and complicated; but if you read carefully you will understand it completely. Also, much of this information is specific to Python although it may be present in other programming languages – this is specifically regarding Python as Python is our preferred language here at IA.

Python & Unicode has discovered that there may be a bit of a security problem with malicious individuals that may use the freedom that Unicode offers a user to in fact be able to write code that fools a person that may be conducting a code review on something that has just been introduced to a system.

Unicode is the universal character encoding standard that provides the basis for processing, storing of text data and the interchange that takes place on various platforms throughout the world. It performs this function regardless of language type and regardless of software being utilized when interacting with information technology protocols. Unicode encompasses all characters and all writing systems used through out the world – all of them – old as well as new systems. The intention of the Unicode Standard is to support the needs of all types of users, be it in business or academia and using either mainstream or minority scripts. As you can imagine, the number of languages and their variations would be difficult to keep track of. Unicode encodes the characters for a script, and not the languages themselves. But it basically covers the entire world of script and text.

If you wish to see which languages are supported in more detail just click here.

In addition to the scripts listed above, a large number of other collections of characters are also encoded by Unicode. These collections include the following:

• Numbers
• General Diacritics
• General Punctuation
• General Symbols
• Mathematical Symbols (Western and Arabic)
• Musical Symbols (Western, Byzantine, Ancient Greek, and other)
• Technical Symbols
• Emoji: For details, see Emoji Versions
• Dingbats
• Arrows, Blocks, Box Drawing Forms, and Geometric Shapes
• Game Symbols
• Miscellaneous Symbols
• Presentation Forms
• Kangxi and other CJK radicals

Keep in mind, Unicode works with script and not languages. So it’s power to encompass virtually any language via texted scripts naturally makes it a broadly used and universally accepted system for encoding languages. It really is a great resource.

When writing systems for more than one language, according the Unicode website :

When writing systems for more than one language share sets of graphical symbols that have historically related derivations, the union of all of them is treated as a single collection of characters for encoding and is viewed (identified ) as a single script – which in turn serves as an inventory of graphical symbols, which are drawn upon for the writing systems of any particular language. So you will see – on occasion – an instance where a spoken/written language will make use of only one script because the symbols are exclusively implied to be specific to that same language; Hangul, is one such case – which is typically only used to write the Korean Language. Where as writing systems for some languages use more than just one script – for example Japanese, which traditionally uses Han (Kanji), Hiragana, and Katakana scripts, and modern Japanese usage commonly mixes in Latin script as well.

So one can see just how very versatile and ingenious Unicode really is.

As advanced and developed as Unicode is and as truly impressive as it is – much work is still being done to complete it. It still requires development tools such as keyboards, fonts language data (dates and such) and language data translation as well. So still a work in progress – if you can believe that.

So the very thing that gives Unicode it’s versatility and power is the same thing that allows for a malicious actor to take advantage of security flaws – because it allows writing virtually any type of code and this fact can lead to confusion for a code reviewer – as an example. And if it looks confusing to a code reviewer, it is possible that that same person reviewing the code will mistakenly allow it and approve it for their system or systems.

Apparently the best course of action is not for Python developers to re-write python code as this would leave far too many restrictions and probably would be much less viable for use with many different data sets and information or code. Awareness seems to be the correct strategy here. That is, the programmers and people that are reviewing the code should approach this by solving the issues that may be present when malicious code is introduced and by simply enforcing project-specific policies using their code editors.

This makes perfect sense – if you ask me.

Malicious code that can – at first glance – appears to be harmless can be missed by the individual that is reviewing the code. This is a possibility and not the norm. Awareness of this possibility was first marked as a potential problem by CVE-2021-42574 :

An issue was discovered in the Bidirectional Algorithm in the Unicode Specification through 14.0. It permits the visual reordering of characters via control sequences, which can be used to craft source code that renders different logic than the logical ordering of tokens ingested by compilers and interpreters. Adversaries can leverage this to encode source code for compilers accepting Unicode such that targeted vulnerabilities are introduced invisibly to human reviewers.

Now, one must understand that this does not mean that anything like this has taken place or that there are individuals out there working on something that targets this vulnerability specifically – this is only an indication that it is possible.

As you can imagine – if you put all of the languages of the world together, there are going to be a number of differences between the different languages of the world when it comes to the script that is used that translates each individual language into Unicode. So there is bound to be some confusion and some mismatches in the way a script may be interpreted. One other thing that needs to mentioned once again is that we are referring specifically to Python and how it affects Python in particular. Although the effect of this security issue may be germane to other languages due to the way a script interprets certain symbols within – if any problem is present in any language – for the most part the problem is the same one and the systems merely need to adapt – that is the programmers, code editors and code reviewers need to stay on high alert and be on the look out for code that is poorly written as a matter of consequence or one that is deliberately written in a malicious way. Either one can be harmful – So the issue needs to be dealt with by the common problem that is innate to any one particular language due to it’s interpretation of any given symbol compared to the universal interpretation of a symbol that resembles it so closely that it may go unnoticed by an individual – and so it behooves us to solve this problem with the code editor during development. Just add to the already vast capabilities that is already present in code editors. This will make the job of anyone involved so much easier.

Some characters look alike. Before the age of computers, many mechanical typewriters lacked the keys for the digits 0 and 1: users typed O (capital o) and l (lowercase L) instead. Human readers could tell them apart by context only. In programming languages, however, distinction between digits and letters is critical — and most fonts designed for programmers make it easy to tell them apart.

Similarly, in fonts designed for human languages, the uppercase “I” and lowercase “l” can look similar. Or the letters “rn” may be virtually indistinguishable from the single letter “m”. Again, programmers’ fonts make these pairs of confusables noticeably different.

However, what is “noticeably” different always depends on the context. Humans tend to ignore details in longer identifiers: the variable name accessibi1ity_options can still look indistinguishable from accessibility_options, while they are distinct for the compiler. The same can be said for plain typos: most humans will not notice the typo in responsbility_chain_delegate.

accessibi1ity_options

accessibility_options

# Don't call this function:
fire_the_missiles()

# Don't call this function:⬛fire_the_missiles()

>>> int('৪୨')
42
>>> '{٥}'.format('zero', 'one', 'two', 'three', 'four', 'five')
five

s = "X" * 100 #    "X" is assigned

s = "א" * 100 #    "א" is assigned

s = "x‏" * 100 #    "‏x" is assigned

>>> xⁿ = 8
>>> xn
8

>>> class Test:
...     def finalize(self):
...         print('OK')
...
>>> Test().finalize()
OK
>>> Test().finalize()
OK
>>> getattr(Test(), 'finalize')
Traceback (most recent call last):
  ...
AttributeError: 'Test' object has no attribute 'finalize'