MultiCpuSystem

Warning: Any example presented here is provided "as-is" with no support or guarantee of suitability. If you have any further questions about these examples please email the squid-users mailing list.

  • /!\ {i} Squid-3.2 has now had experimental multi-process SMP support merged. It is designed to operate with a similar but different configuration to these while reducing much of the complexity of process instance management.

Outline

Squid-3.1 and older do not scale very well to Multi-CPU or Multi-Core systems. Some of its features do help, such as for example DiskDaemon, or COSS, or the ability to delegate parts of the request processing to external helpers such as Authenticators or other auxiliary software. Still Squid remains to this day very bound to a single processing core model. There are plans to eventually make Squid able to effectively use multicore systems, but something may be done already, by using a fine-tuned MultipleInstances setup.

/!\ Notice

This setup has been designed with a recent version of Squid in mind. It has been tested with Squid-3.1, but it should work with Squid-3.0 and Squid-2.7 as-is as well. Squid-2.6 and earlier can be coaxed to work, but it will be harder to setup and maintain.

It is also geared at expert system-administrators. MultipleInstances is not easy to manage and run, and system integration depends on the specific details of the operating system distribution of choice.

The setup laid out in this configuration example aims at creating on a system multiple running squid processes:

  • a 'front-end' process which does
    • authentication
    • authorization
    • logging, delay pools etc.
    • in-memory hot-object caching
    • load-balancing of the backend processes
    • redirection etc.
  • a 'back-end' processes farm, whose each does
    • disk caching
    • do the network heavy lifting

While this setup is expected to increase the general throughput of a multicore system, the benefits are anyways constrained, as the frontend process is still expected to be the bottleneck. Should anyone put this in production, he's encouraged to share the results to help others evaluate the effectiveness of the solution.

Squid Configuration File

For a 2-backends system, there are 5 configuration files to be used. You can click below each file on its filename to download it, no need to copy and paste. The .txt extension an artifact, please remove it.

acl

This file contains the ACL's that are common to all running instances. This allows to change cluster-wide parameters without needing to touch each instance's. Each instance will still have to be reconfigured individually.

   1 acl manager proto cache_object
   2 acl localhost src 127.0.0.1/32
   3 acl to_localhost dst 127.0.0.0/8
   4 acl localnet src 192.168.0.0/24
   5 acl SSL_ports port 443
   6 acl Safe_ports port 80      # http
   7 acl Safe_ports port 21      # ftp
   8 acl Safe_ports port 443     # https
   9 acl Safe_ports port 70      # gopher
  10 acl Safe_ports port 210     # wais
  11 acl Safe_ports port 1025-65535  # unregistered ports
  12 acl Safe_ports port 280     # http-mgmt
  13 acl Safe_ports port 488     # gss-http
  14 acl Safe_ports port 591     # filemaker
  15 acl Safe_ports port 777     # multiling http
  16 acl CONNECT method CONNECT
  17 
  18 
  19 
  20 cachemgr_passwd somepassword all
common.acl.conf.txt

common backend parameters

Backends share most of the configuration, it makes sense to also join those

   1 #you want the backend to have a small cache_mem
   2 cache_mem 4 MB
   3 
   4 refresh_pattern ^ftp:		1440	20%	10080
   5 refresh_pattern ^gopher:	1440	0%	1440
   6 refresh_pattern -i (/cgi-bin/|\?) 0	0%	0
   7 refresh_pattern .		0	20%	4320
   8 
   9 shutdown_lifetime 3 second
  10 #debug_options all,8
  11 
  12 # add user authentication and similar options
  13 http_access deny !Safe_ports
  14 http_access deny CONNECT !SSL_ports
  15 http_access allow localhost
  16 http_access deny all
common.backend.conf.txt

frontend

   1 # acl are shared among instances
   2 include /usr/local/etc/lab/common.acl.conf
   3 
   4 http_port 3128
   5 
   6 #add backends
   7 cache_peer localhost parent 4001 0 carp login=PASS name=backend-1
   8 cache_peer localhost parent 4002 0 carp login=PASS name=backend-2
   9 
  10 #you want the frontend to have a significant cache_mem
  11 cache_mem 512 MB
  12 
  13 refresh_pattern ^ftp:		1440	20%	10080
  14 refresh_pattern ^gopher:	1440	0%	1440
  15 refresh_pattern -i (/cgi-bin/|\?) 0	0%	0
  16 refresh_pattern .		0	20%	4320
  17 
  18 shutdown_lifetime 3 second
  19 #debug_options all,8
  20 
  21 # change /tmp to your own log directory, e.g. /var/log/squid
  22 access_log /var/log/squid/frontend.access.log
  23 cache_log /var/log/squid/frontend.cache.log
  24 pid_filename /var/log/squid/frontend.pid
  25 
  26 # add user authentication and similar options
  27 http_access allow manager localhost
  28 http_access deny manager
  29 http_access deny !Safe_ports
  30 http_access deny CONNECT !SSL_ports
  31 http_access allow localnet
  32 http_access allow localhost
  33 http_access deny all
frontend.conf.txt

backend 1

   1 # acl are shared among instances
   2 include /usr/local/etc/lab/common.acl.conf
   3 
   4 http_port 127.0.0.1:4001
   5 visible_hostname backend-1
   6 unique_hostname backend-1
   7 cache_dir aufs /mnt/cache-1 10240 128 128
   8 
   9 # change /var/log/squid to your own log directory
  10 access_log /var/log/squid/backend-1.access.log
  11 cache_log /var/log/squid/backend-1.cache.log
  12 pid_filename /var/log/squid/backend-1.pid
  13 
  14 include /usr/local/etc/lab/common.backend.conf
backend-1.conf.txt

backend 2

   1 # acl are shared among instances
   2 include /usr/local/etc/lab/common.acl.conf
   3 
   4 http_port 127.0.0.1:4002
   5 visible_hostname backend-2
   6 unique_hostname backend-2
   7 cache_dir aufs /mnt/cache-2 10240 128 128
   8 
   9 # change /var/log/squid to your own log directory
  10 access_log /var/log/squid/backend-2.access.log
  11 cache_log /var/log/squid/backend-2.cache.log
  12 pid_filename /var/log/squid/backend-2.pid
  13 
  14 include /usr/local/etc/lab/common.backend.conf
backend-2.conf.txt


CategoryConfigExample

ConfigExamples/MultiCpuSystem (last edited 2010-07-09 07:20:10 by FrancescoChemolli)