Contents

  1. Sending Bug Reports to the Squid Team
    1. crashes and core dumps
    2. Resource Limits
    3. Debugging Symbols
    4. Coredump Location
    5. Using gdb debugger on Squid
    6. Using gdb debugger on a live proxy (with minimal downtime)
    7. Attaching gdb debugger to and already running Squid
  2. Debugging Squid
    1. Full Debug Output
    2. Debug Sections
    3. Debug Levels

Sending Bug Reports to the Squid Team

Bug reports for Squid should be registered in our bug database.

Any bug report must include

  • The Squid release version
  • Your Operating System type and version
  • A clear description of the bug symptoms.
  • If your Squid crashes the report must include a coredumps stack trace as described below

Please note that:

  • bug reports are only processed if they can be reproduced or identified in the current STABLE or development versions of Squid.
  • If you are running an older version of Squid the first response will be to ask you to upgrade unless the developer who looks at your bug report immediately can identify that the bug also exists in the current versions.
  • It should also be noted that any patches provided by the Squid developer team will be to the current STABLE version even if you run an older version.

crashes and core dumps

There are two conditions under which squid will exit abnormally and generate a coredump. First, a SIGSEGV or SIGBUS signal will cause Squid to exit and dump core. Second, many functions include consistency checks. If one of those checks fail, Squid calls abort() to generate a core dump.

If you have a core dump file then use gdb to extract a stack trace from the core as follows: 

% gdb /usr/local/squid/sbin/squid core
gdb> backtrace

Many people report that Squid doesn't leave a coredump anywhere. This may be due to one of the following reasons:

  • Resource Limits
    • The shell has limits on the size of a coredump file. You may need to increase the limit using ulimit or a similar command (see below)
  • sysctl options
    • On FreeBSD, you won't get a coredump from programs that call setuid() and/or setgid() (like Squid sometimes does) unless you enable this option: 

# sysctl -w kern.sugid_coredump=1
  • No debugging symbols
    • The Squid binary must have debugging symbols in order to get a meaningful coredump. The debugging traces we need look something like this: 

Core was generated by `(squid) -D'.
Program terminated with signal 6, Aborted.

(gdb) bt
#0  0x006ad7a2 in _dl_sysinfo_int80 () from /lib/ld-linux.so.2
#1  0x006ed7a5 in raise () from /lib/tls/libc.so.6
#2  0x006ef209 in abort () from /lib/tls/libc.so.6
#3  0x0806b987 in xassert (msg=Could not find the frame base for "xassert".
) at debug.c:514
#4  0x0808170b in httpBuildRequestHeader (request=0x10791f40,
orig_request=0x10791f40, entry=0xfc6ba30, hdr_out=0xbfed04f0, flags=
      {proxying = 0, keepalive = 1, only_if_cached = 0, keepalive_broken = 0,
abuse_detected = 0, request_sent = 0, front_end_https = 0, originpeer = 0}) at
http.c:1195
...

if you find the trace contains a lot of lines with ?? and mentions no symbols found. It is usually useless and you will need to run a version of Squid where the debug symbols have not been removed.

  • Threads and Linux
    • On Linux, threaded applications do not generate core dumps. When you use the aufs cache_dir type, it uses threads and you can't get a coredump.
  • It did leave a coredump file, you just can't find it.

Resource Limits

These limits can usually be changed in shell scripts. The command to change the resource limits is usually either ulimit or limits. Sometimes it is a shell-builtin function, and sometimes it is a regular program. Also note that you can set resource limits in the /etc/login.conf file on FreeBSD and maybe other systems.

To change the coredumpsize limit you might use a command like: 

limits coredump unlimited

Debugging Symbols

To see if your Squid binary has debugging symbols, use this command: 

% nm /usr/local/squid/bin/squid | head

The binary has debugging symbols if you see gobbledegook like this: 

0812abec B AS_tree_head
080a7540 D AclMatchedName
080a73fc D ActionTable
080908a4 r B_BYTES_STR
080908bc r B_GBYTES_STR
080908ac r B_KBYTES_STR
080908b4 r B_MBYTES_STR
080a7550 D Biggest_FD
08097c0c R CacheDigestHashFuncCount
08098f00 r CcAttrs

There are no debugging symbols if you see this instead: 

/usr/local/squid/bin/squid: no symbols

Debugging symbols may have been removed by your install program. If you look at the squid binary from the source directory, then it might have the debugging symbols.

Coredump Location

The core dump file will be left in one of the following locations:

  1. The coredump_dir directory, if you set that option.

  2. The first cache_dir directory if you have used the cache_effective_user option.

  3. The current directory when Squid was started

Recent versions of Squid report their current directory after starting, so look there first: 

2000/03/14 00:12:36| Set Current Directory to /usr/local/squid/cache

If you cannot find a core file, then either Squid does not have permission to write in its current directory, or perhaps your shell limits are preventing the core file from being written.

Often you can get a coredump if you run Squid from the command line like this (csh shells and clones): 

% limit core un
% /usr/local/squid/bin/squid -NCd1

Once you have located the core dump file, use a debugger such as dbx or gdb to generate a stack trace: 

% gdb /usr/local/squid/sbin/squid core
gdb> backtrace

If possible, you might keep the coredump file around for a day or two. It is often helpful if we can ask you to send additional debugger output, such as the contents of some variables. But please note that a core file is only useful if paired with the exact same binary as generated the corefile. If you recompile Squid then any coredumps from previous versions will be useless unless you have saved the corresponding Squid binaries, and any attempts to analyze such coredumps will most certainly give misleading information about the cause to the crash.

Using gdb debugger on Squid

If you CANNOT get Squid to leave a core file for you then one of the following approaches can be used

First alternative is to start Squid under the contol of GDB 

% gdb /path/to/squid
handle SIGPIPE pass nostop noprint
handle SIGTERM pass nostop noprint
handle SIGUSR1 pass nostop noprint
handle SIGSEGV stop
handle SIGABRT stop
run -DNYCd3
[wait for crash]
backtrace
quit

Using gdb debugger on a live proxy (with minimal downtime)

The drawback from the above is that it isn't really suitable to run on a production system as Squid then won't restart automatically if it crashes. The good news is that it is fully possible to automate the process above to automatically get the stack trace and then restart Squid. Here is a short automated script that should work: 

trap "rm -f $$.gdb" 0
cat <<EOF >$$.gdb
handle SIGPIPE pass nostop noprint
handle SIGTERM pass nostop noprint
handle SIGUSR1 pass nostop noprint
handle SIGSEGV stop
handle SIGABRT stop
run -DNYCd3
backtrace
quit
EOF
while sleep 2; do
  gdb -x $$.gdb /path/to/squid 2>&1 | tee -a squid.out
done

Other options if the above cannot be done is to:

  1. Build Squid with the --enable-stacktraces option, if support exists for your OS (exists for Linux glibc on Intel, and Solaris with some extra libraries which seems rather impossible to find these days..)
  2. Run Squid using the "catchsegv" tool. (Linux glibc Intel)

    {i} these approaches do not by far provide as much details as using gdb.

Attaching gdb debugger to and already running Squid

First locate the PID number for the particular Squid worker you are wanting to debug. 

% gdb /path/to/squid
handle SIGPIPE pass nostop noprint
handle SIGTERM pass nostop noprint
handle SIGUSR1 pass nostop noprint
handle SIGSEGV stop
handle SIGABRT stop

attach [worker PID]
[wait for crash]
backtrace
detach
quit

Debugging Squid

If you believe you have found a non-fatal bug (such as incorrect HTTP processing) please send us a section of your cache.log with debugging to demonstrate the problem. The cache.log file can become very large, so alternatively, you may want to copy it to an FTP or HTTP server where we can download it.

Once you have the debugging captured to cache.log, take a look at it yourself and see if you can make sense of the behavior which you see. If not, please feel free to send your debugging output to the squid-users or squid-bugs mailing lists.

Full Debug Output

It is very simple to enable full debugging on a running squid process. 

squid -k debug

This causes every debug() statement in the source code to write a line in the cache.log file. You also use the same command to restore Squid to normal debugging level.

cache.log has to be opened after Squid is already running, so to debug the very first and very last operations Squid does on startup/shutdown you will need to use the -X option when starting Squid: 

squid -X

This will dump the initial details to either stdout for your immediate viewing or to one of the system logs (usually syslog or daemon.log).

Debug Sections

To enable selective debugging (e.g. for one source file only), you need to edit squid.conf and add to the debug_options line. Every Squid source file is assigned a debugging section. The debugging section assignments can be found by looking at the top of individual source files, by reading the file debug-sections.txt, or looking at KnowledgeBase/DebugSections.

Debug Levels

You also specify the debugging level to control the amount of debugging. Higher levels result in more debugging messages. For example, to enable full debugging of Access Control functions, you would use: 

debug_options 28,9

Then you have to restart or reconfigure Squid.

Back to the SquidFaq

SquidFaq/BugReporting (last edited 2012-02-08 11:59:26 by Amos Jeffries)