Decompress Binary

Name: Decompress Binary
Author: clickhouse

clickhouse/clickhouse

10 installs
49.1k repo stars
Updated August 5, 2026
clickhouse/clickhouse

Extract the inner ELF from a self-extracting ClickHouse binary, including across architectures, so gdb/lldb can load real symbols for CI or release cores.

About

Extracts the inner ELF from a self-extracting ClickHouse binary, including cross-architecture cases like loading an aarch64 core dump on x86. A developer uses it when a debugger needs real symbols from a downloaded CI or release binary.

Handles architecture mismatch (aarch64 core on x86)
Provides real symbols for gdb/lldb

Decompress Binary by the numbers

10 all-time installs (skills.sh)
Ranked #425 of 596 Debugging skills by installs in the Skillselion catalog
Data as of Aug 5, 2026 (Skillselion catalog sync)

npx skills add https://github.com/clickhouse/clickhouse --skill decompress-binary

Add your badge

Show developers this skill is listed on Skillselion. Paste this into your README.

[![Listed on Skillselion](https://skillselion.com/badge/skills/clickhouse/clickhouse/decompress-binary.svg)](https://skillselion.com/skills/clickhouse/clickhouse/decompress-binary)

Installs	10
repo stars	★ 49.1k
Last updated	August 5, 2026
Repository	clickhouse/clickhouse ↗

What it does

Extract the inner ELF from a self-extracting ClickHouse binary, including across architectures, so gdb/lldb can load real symbols for CI or release cores.

Files

SKILL.mdMarkdownGitHub ↗

Decompress a ClickHouse Binary (Cross-Architecture)

ClickHouse release and CI clickhouse binaries are self-extracting: a small decompressor stub, followed by the zstd-compressed real ELF and a trailer.

The normal way to decompress is to run the binary once: it extracts the inner ELF in place and re-execs it. That works on the binary's own architecture, and also on a foreign architecture if `qemu` user-mode emulation for it is installed (e.g. qemu-aarch64 to run an aarch64 binary on x86). When qemu for the target is not available, you cannot run the binary at all and must extract the payload offline.

This skill extracts the inner ELF without executing anything, on any host.

When to use

You downloaded a CI/release clickhouse and gdb/lldb shows no real symbols

(it only sees the decompressor stub's tiny symbol table).

The binary's architecture differs from the host (cannot self-extract by running).
You need the inner ELF to load a core dump (see ci/decrypt-cores.md for the

matching core-dump decryption procedure).

Format

See utils/self-extracting-executable/types.h:

[ decompressor ELF ]
[ compressed file blobs ]
[ FileData[] ]                 # one per packed file, each followed by its name
[ MetaData (16 bytes) @ EOF ]

MetaData { uint64 number_of_files; uint64 start_of_files_data; }
FileData { uint64 start, end, name_length, uncompressed_size, umask; bool exec; }

MetaData sits at the very end of the file. start_of_files_data points at the FileData array; each 48-byte FileData is followed by the file name. The compressed bytes for a file are input[start:end] (zstd, possibly multi-frame). The packed clickhouse ELF is the entry with exec = true.

Steps

1. Download the binary from the build job for the exact commit, for example:

   curl -s "https://clickhouse-builds.s3.amazonaws.com/PRs/<pr>/<sha>/build_<arch>_<sanitizer>/clickhouse" -o clickhouse.sfx

Find the precise URL in the build job's artifact_report_build_*.json, or via .claude/tools/fetch_ci_report.js "<pr-url>". Download in the foreground (a killed/resumed curl can append garbage past EOF and break the trailer; verify the size matches Content-Length).

2. Extract the inner ELF:

   python3 .claude/skills/decompress-binary/extract_self_extracting.py clickhouse.sfx clickhouse.elf

3. Verify it is the right build and has symbols:

   file clickhouse.elf                                          # ELF ..., not stripped, with debug_info
   llvm-objdump -s -j .note.gnu.build-id clickhouse.elf | tail  # must match the core's build id

The build id must equal the one in the crash report / core. A mismatched binary yields unusable backtraces.

4. Use it with the core dump:

   gdb clickhouse.elf core.<pid>      # or: lldb clickhouse.elf -c core.<pid>

gdb and lldb read foreign-architecture cores fine for backtraces and memory inspection (you are not executing the target).

Notes

A truncated or corrupted download is the most common failure: if the script

reports an implausible number_of_files, re-download cleanly and check the size.

The inner ELF is large (several GB for sanitizer builds, unstripped). Make sure

there is enough disk.

Shortcut when you can run the binary: if the host matches the binary's

architecture, or qemu user-mode emulation for it is installed, just run ./clickhouse once to self-extract in place. This skill is for the case where neither is possible.

#!/usr/bin/env python3
"""Extract the inner ELF from a ClickHouse self-extracting executable.

ClickHouse release/CI `clickhouse` binaries are self-extracting: a small
decompressor stub followed by the zstd-compressed real ELF and a trailer.
Running the binary self-extracts by re-`exec`-ing itself, which only works on
the binary's own architecture. To inspect a foreign-arch binary on your host
(e.g. load an aarch64 CI core dump on an x86 workstation) you must extract the
payload offline, which is what this script does.

Layout (see utils/self-extracting-executable/types.h):

    [ decompressor ELF ]
    [ compressed file blobs ]
    [ FileData[] ]                      # one per packed file, each followed by its name
    [ MetaData (16 bytes) @ EOF ]

    MetaData { uint64 number_of_files; uint64 start_of_files_data; }
    FileData { uint64 start, end, name_length, uncompressed_size, umask; bool exec; }
              # 41 payload bytes, padded to 48; name (name_length bytes) follows each FileData
    Compressed blob for a file is input[start:end] (zstd, possibly multi-frame).

Usage:
    extract_self_extracting.py <clickhouse-self-extracting> [output.elf]
"""
import struct
import subprocess
import sys

FILEDATA_FMT = "<QQQQQ?"  # start, end, name_length, uncompressed_size, umask, exec
FILEDATA_SIZE = 48        # struct is padded to 48 bytes in C++


def main() -> int:
    if len(sys.argv) < 2:
        sys.exit(__doc__)
    path = sys.argv[1]
    out = sys.argv[2] if len(sys.argv) > 2 else "clickhouse.elf"

    data = open(path, "rb").read()
    number_of_files, start_of_files = struct.unpack_from("<QQ", data, len(data) - 16)
    print(f"number_of_files={number_of_files} start_of_files_data={start_of_files}")
    if not 0 < number_of_files < 1000:
        sys.exit("implausible file count; not a self-extracting binary or it is truncated")

    extracted = False
    pos = start_of_files
    for i in range(number_of_files):
        start, end, name_len, usize, _umask, is_exec = struct.unpack_from(FILEDATA_FMT, data, pos)
        name = data[pos + FILEDATA_SIZE : pos + FILEDATA_SIZE + name_len].decode("utf-8", "replace")
        print(f"  file[{i}] name={name!r} exec={is_exec} comp=[{start},{end}) ({end - start} B) uncompressed={usize}")
        if is_exec:
            blob = data[start:end]
            res = subprocess.run(["zstd", "-d", "-f", "-o", out], input=blob,
                                 stdout=subprocess.PIPE, stderr=subprocess.PIPE)
            if res.returncode != 0:
                sys.exit("zstd failed: " + res.stderr.decode()[:500])
            print(f"  -> extracted inner ELF to {out}")
            extracted = True
        pos += FILEDATA_SIZE + name_len

    if not extracted:
        sys.exit("no executable file found inside the archive")
    return 0


if __name__ == "__main__":
    raise SystemExit(main())