Misc. bug: segfault downloading models

[WhyNotHugo]

Name and Version

$ llama-cli --version
version: 8611 (d43375f)
built with GNU 15.2.0 for Linux x86_64

Operating systems

Linux

Which llama.cpp modules do you know to be affected?

llama-server

Command line

llama-server --hf-repo unsloth/Qwen3.5-9B-GGUF:Qwen3.5-9B-Q5_K_M.gguf

Problem description & steps to reproduce

Downloading models segfaults with b8611.

First Bad Commit

I bisected this to 8c7957c

Relevant log output

$ llama-server --sleep-idle-seconds 300 --host 0.0.0.0 --hf-repo HauhauCS/Qwen3.5-2B-Uncensored-HauhauCS-Aggres
sive:Q6_K
common_download_file_single_online: HEAD failed, status: 404
no remote preset found, skipping
Segmentation fault (core dumped)

backtrace:

0x0000555555740dab in std::__detail::_Executor<__gnu_cxx::__normal_iterator<char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::__cxx11::sub_match<__gnu_cxx::__normal_iterator<char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > >, std::__cxx11::regex_traits<char>, true>::_M_dfs(std::__detail::_Executor<__gnu_cxx::__normal_iterator<char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::__cxx11::sub_match<__gnu_cxx::__normal_iterator<char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > >, std::__cxx11::regex_traits<char>, true>::_Match_mode, long) ()
(gdb) bt
#0  0x0000555555740dab in std::__detail::_Executor<__gnu_cxx::__normal_iterator<char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::__cxx11::sub_match<__gnu_cxx::__normal_iterator<char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > >, std::__cxx11::regex_traits<char>, true>::_M_dfs(std::__detail::_Executor<__gnu_cxx::__normal_iterator<char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::__cxx11::sub_match<__gnu_cxx::__normal_iterator<char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > >, std::__cxx11::regex_traits<char>, true>::_Match_mode, long) ()
#1  0x0000555555741518 in std::__detail::_Executor<__gnu_cxx::__normal_iterator<char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::__cxx11::sub_match<__gnu_cxx::__normal_iterator<char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > >, std::__cxx11::regex_traits<char>, true>::_M_rep_once_more(std::__detail::_Executor<__gnu_cxx::__normal_iterator<char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::__cxx11::sub_match<__gnu_cxx::__normal_iterator<char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > >, std::__cxx11::regex_traits<char>, true>::_Match_mode, long) ()
#2  0x0000555555740be9 in std::__detail::_Executor<__gnu_cxx::__normal_iterator<char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::__cxx11::sub_match<__gnu_cxx::__normal_iterator<char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > >, std::__cxx11::regex_traits<char>, true>::_M_dfs(std::__detail::_Executor<__gnu_cxx::__normal_iterator<char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::__cxx11::sub_match<__gnu_cxx::__normal_iterator<char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > >, std::__cxx11::regex_traits<char>, true>::_Match_mode, long) ()
#3  0x0000555555740dfe in std::__detail::_Executor<__gnu_cxx::__normal_iterator<char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::__cxx11::sub_match<__gnu_cxx::__normal_iterator<char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > >, std::__cxx11::regex_traits<char>, true>::_M_dfs(std::__detail::_Executor<__gnu_cxx::__normal_iterator<char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::__cxx11::sub_match<__gnu_cxx::__normal_iterator<char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > >, std::__cxx11::regex_traits<char>, true>::_Match_mode, long) ()
#4  0x0000555555741518 in std::__detail::_Executor<__gnu_cxx::__normal_iterator<char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::__cxx11::sub_match<__gnu_cxx::__normal_iterator<char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > >, std::__cxx11::regex_traits<char>, true>::_M_rep_once_more(std::__detail::_Executor<__gnu_cxx::__normal_iterator<char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::__cxx11::sub_match<__gnu_cxx::__normal_iterator<char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > >, std::__cxx11::regex_traits<char>, true>::_Match_mode, long) ()
#5  0x0000555555740be9 in std::__detail::_Executor<__gnu_cxx::__normal_iterator<char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::__cxx11::sub_match<__gnu_cxx::__normal_iterator<char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > >, std::__cxx11::regex_traits<char>, true>::_M_dfs(std::__detail::_Executor<__gnu_cxx::__normal_iterator<char const*, std::__cxx11::basic_string<char--Type <RET> for more, q to quit, c to continue without paging--

[WhyNotHugo]

If I understand correctly, the default thread stack size is ~130KB, and std::regex uses deep recursion for backtracking, which exceeds this stack.

[WhyNotHugo]

Apparently, the default stack size for new threads on musl is 130KB, but on glibc it's a whooping 8MB, so the issue would be musl-specific.

[angt]

Hey!
Could you please share how you built it or where you got the binary ?

[WhyNotHugo]

The build is via the Alpine downstream package (which I'm maintaining): https://gitlab.alpinelinux.org/alpine/aports/-/blob/afdf37b577c3fe070b65686e000723d494803248/testing/llama.cpp/APKBUILD

You can build this manually in an Alpine environment:

• Install build deps: apk add build-base cmake glslang-dev jq linux-headers openblas-dev openssl-dev samurai shaderc vulkan-loader-dev
• Use the same cmake commands.

[angt]

Just setup a bigger the stack, this works:

FROM alpine:latest AS builder

RUN apk add --no-cache \
    git \
    cmake \
    openssl-dev \
    build-base \
    linux-headers

RUN git clone https://github.com/ggml-org/llama.cpp
WORKDIR /llama.cpp

RUN cmake -B build -DBUILD_SHARED_LIBS=OFF -DCMAKE_EXE_LINKER_FLAGS="-Wl,-z,stack-size=$((8*1024*1024))"
RUN cmake --build build -j32 -t llama-server
RUN build/bin/llama-server -hf HauhauCS/Qwen3.5-2B-Uncensored-HauhauCS-Aggressive:Q6_K

[angt]

I also use musl in https://github.com/angt/installama.sh without issues, it's a default config to change.

[WhyNotHugo]

-DCMAKE_EXE_LINKER_FLAGS="-Wl,-z,stack-size=$((8*1024*1024))" works, thanks!

I didn't occur to me that the stack size could be changed at compile time, but it's obvious in hindsight.

Do we want to incorporate this into CMakeList.txt, or shall we just close this?

[WhyNotHugo]

I also use musl in https://github.com/angt/installama.sh without issues, it's a default config to change.

I'll keep this in mind if I hit any similar issues :)

[angt]

I think we can close, the fix is easy in your case because we know the os and the compiler. A generic solution in cmake would be fragile and not so useful i believe.

Now I’m going to open a PR in cpp-httplib to fix the root cause i belive 😬