Using eCAP with Squid 3.x/4.x
Using eCAP for GZip support with Squid 3.x/4.x
by YuriVoinov
Warning: Any example presented here is provided "as-is" with no support or guarantee of suitability. If you have any further questions about these examples please email the squid-users mailing list.
Contents
Outline
Since Squid does not support runtime content compression with GZip, we will be used existing eCAP support and GZip eCAP module.
Usage
This configuration is very useful to reduce external Internet traffic from proxy and good caching compressed data.
Build eCAP library
We are uses two different libraries for different branches of Squid. 0.2.0 for Squid 3.4.x or 1.0.0/1.0.1 for Squid 3.5.x/4.x.x
Build and install library accordingly your Squid 32-bit or 64-bit versions:
## 32-bit GCC ./configure 'CXXFLAGS=-O2 -m32 -pipe' 'CFLAGS=-O2 -m32 -pipe' ## 64-bit GCC ./configure 'CXXFLAGS=-O2 -m64 -pipe' 'CFLAGS=-O2 -m64 -pipe' gmake gmake install-strip
Then rebuild your Squid with --enable-ecap configure option. To do that you may need to add PKG_CONFIG_PATH to your configure options:
./configure '--enable-ecap' 'PKG_CONFIG_PATH=/usr/local/lib/pkgconfig'
PKG_CONFIG_PATH pointed to libecap pkgconfig file.
Patch and build squid-ecap-gzip
To build squid-ecap-gzip with corresponding eCAP library, you need apply patch for 0.2.0 or 1.0.0 first.
Note: Patch 1.0.0 appropriate also when using libecap 1.0.1.
Then build squid-ecap-gzip:
## 32 bit GCC ./configure 'CXXFLAGS=-O2 -m32 -pipe' 'CFLAGS=-O2 -m32 -pipe' 'LDFLAGS=-L/usr/local/lib' ## 64 bit GCC ./configure 'CXXFLAGS=-O2 -m64 -pipe' 'CFLAGS=-O2 -m64 -pipe' 'LDFLAGS=-L/usr/local/lib' gmake gmake install-strip
- Note: It is important to choose identical 32 or 64 bit (like your Squid) build mode for eCAP library and squid-gzip-ecap.
Squid Configuration File
Paste the configuration file like this:
ecap_enable on acl HTTP_STATUS_OK http_status 200 loadable_modules /usr/local/lib/ecap_adapter_gzip.so ecap_service gzip_service respmod_precache ecap://www.vigos.com/ecap_gzip bypass=off adaptation_access gzip_service allow HTTP_STATUS_OK
Also you can add next lines to your squid.conf:
# Replace Accept-Encoding to support compression via eCAP request_header_access Accept-Encoding deny all request_header_replace Accept-Encoding gzip
to adapt Accept-Encoding to set gzip support first.
Finally, restart your Squid and enjoy.
Note: Don't specify Accept-Encoding in request_header_replace like this: gzip;q=1.0, identity; q=0.5, *;q=0. This is correct, but Yahoo experienced known problems with this encoding specifications. Also note - identity is default and you do not required to specify it.
Support compression all text/* or extended text content types
To support compression not only text/html, but also all text/* (i.e. text/javascript, text/plain, text/xml, text/css) types you must patch squid-ecap-gzip with this one:
--- src/adapter_gzip.cc 2011-02-13 17:42:20.000000000 +0300
+++ src/adapter_gzip.cc 2012-02-26 03:37:26.000000000 +0400
@@ -353,17 +353,19 @@
-* At this time, only responses with "text/html" content-type are allowed to be compressed.
+* At this time, only responses with "text/*" content-type are allowed to be compressed.
*/
static const libecap::Name contentTypeName("Content-Type");
-
+
// Set default value
this->requirements.responseContentTypeOk = false;
if(adapted->header().hasAny(contentTypeName)) {
const libecap::Header::Value contentType = adapted->header().value(contentTypeName);
-
+ std::string contentTypeType; // store contenttype substr
+
if(contentType.size > 0) {
std::string contentTypeString = contentType.toString(); // expensive
+ contentTypeType = contentTypeString.substr(0,4);
- if(strstr(contentTypeString.c_str(),"text/html")) {
+ if(strstr(contentTypeType.c_str(),"text")) {
this->requirements.responseContentTypeOk = true;
}
}- Note: This is not all possible text types in modern Web. If you want to achieve less disk cache and a bit more delivery speed, you can apply another patch against previous:
--- src/adapter_gzip.cc Tue Jun 21 03:20:48 2016
+++ src/adapter_gzip.cc Tue Jun 21 03:24:34 2016
@@ -367,7 +367,6 @@
/**
* Checks the Content-Type response header.
- * At this time, only responses with "text/html" content-type are allowed to be compressed.
*/
static const libecap::Name contentTypeName("Content-Type");
@@ -376,13 +375,27 @@
if(adapted->header().hasAny(contentTypeName)) {
const libecap::Header::Value contentType = adapted->header().value(contentTypeName);
-
+
+ std::string contentTypeType; // store contenttype substr
+
if(contentType.size > 0) {
std::string contentTypeString = contentType.toString(); // expensive
-
- if(strstr(contentTypeString.c_str(),"text/html")) {
+ contentTypeType = contentTypeString.substr(0,4);
+ if(strstr(contentTypeType.c_str(),"text")) {
this->requirements.responseContentTypeOk = true;
}
+ else if(strstr(contentTypeString.c_str(),"application/xml")) {
+ this->requirements.responseContentTypeOk = true;
+ }
+ else if(strstr(contentTypeString.c_str(),"application/javascript")) {
+ this->requirements.responseContentTypeOk = true;
+ }
+ else if(strstr(contentTypeString.c_str(),"application/x-javascript")) {
+ this->requirements.responseContentTypeOk = true;
+ }
+ else if(strstr(contentTypeString.c_str(),"application/x-protobuffer")) {
+ this->requirements.responseContentTypeOk = true;
+ }
}
}
@@ -410,7 +423,7 @@
adapted->header().add(name, value);
- // Add "Vary: Accept-Encoding" response header if Content-Type is "text/html"
+ // Add "Vary: Accept-Encoding" response header if Content-Type is supported type
if(requirements.responseContentTypeOk) {
static const libecap::Name varyName("Vary");
const libecap::Header::Value varyValue = libecap::Area::FromTempString("Accept-Encoding");After applying this patch has the meaning to change access to adapter as follows:
ecap_enable on acl HTTP_STATUS_OK http_status 200 loadable_modules /usr/local/lib/ecap_adapter_gzip.so ecap_service gzip_service respmod_precache ecap://www.vigos.com/ecap_gzip bypass=off acl allowedmime rep_mime_type -i (text\/|javascript|xml|application\/x-protobuffer) adaptation_access gzip_service allow allowedmime adaptation_access gzip_service allow HTTP_STATUS_OK
to prevent adapter overloading with unsupported types.
- Note: To prevent possible memory leaking during adapter running, you can also use this patch:
--- src/adapter_gzip.cc Wed Jun 8 21:21:10 2016
+++ src/adapter_gzip.cc Sat Jun 18 22:32:09 2016
@@ -548,7 +548,7 @@
Must(receivingVb == opOn);
- receivingVb = opComplete;
+ stopVb();
if (sendingAb == opOn) {
hostx->noteAbContentDone(atEnd);
sendingAb = opComplete;
@@ -611,7 +611,7 @@
// if the host does not know that already
void Adapter::Xaction::stopVb() {
if (receivingVb == opOn) {
- hostx->vbStopMaking();
+ hostx->vbStopMaking(); // we will not call vbContent() any more
receivingVb = opComplete;
} else {
// we already got the entire body or refused it earlierThis patch is based on original Alex Rousskov patch from here.
Using eCAP for antivirus checking with Squid 3.x/4.x
Outline
Using eCAP for antivirus checking, like C-ICAP, may be more effective. You avoiding usage intermediate services (C-ICAP and clamd itself, module uses libclamav), and, therefore, can do antivirus checking more quickly. This is reduces total Squid installation latency and memory consumption as a whole.
Build eCAP ClamAV adapter
First you need to download eCAP ClamAV adapter from here.
Then you need to compile and install adapter:
## 32 bit GCC ./configure 'CXXFLAGS=-O3 -m32 -pipe' 'CFLAGS=-O3 -m32 -pipe' 'LDFLAGS=-L/usr/local/lib' PKG_CONFIG_PATH=/usr/local/lib/pkgconfig:/usr/local/clamav/lib/pkgconfig 'CPPFLAGS=-I/usr/local/clamav/include -I/usr/include' 'LDFLAGS=-L/usr/local/lib -L/usr/local/clamav/lib' ## 64 bit GCC ./configure 'CXXFLAGS=-O3 -m64 -pipe' 'CFLAGS=-O3 -m64 -pipe' 'LDFLAGS=-L/usr/local/lib' PKG_CONFIG_PATH=/usr/local/lib/pkgconfig:/usr/local/clamav/lib/pkgconfig 'CPPFLAGS=-I/usr/local/clamav/include -I/usr/include' 'LDFLAGS=-L/usr/local/lib -L/usr/local/clamav/lib/amd64' gmake gmake install-strip
- Note: To use adapter with 64-bit Squid, you need also to compile ClamAV and libecap also with 64 bit. Also use appropriate adapter version for interoperability with your Squid version and used libecap.
- Note: On some platforms (i.e. Solaris) you may need to add #include <unistd.h> to src/Gadgets.h to avoid compilation error due to lack of unlink subroutine.
Squid Configuration File
Paste the configuration file like this:
ecap_enable on # Bypass scan mime-types acl bypass_scan_types_req req_mime_type -i ^text/ acl bypass_scan_types_req req_mime_type -i ^application/x-javascript acl bypass_scan_types_req req_mime_type -i ^application/x-shockwave-flash acl bypass_scan_types_req req_mime_type -i ^image/ acl bypass_scan_types_req req_mime_type -i ^video acl bypass_scan_types_req req_mime_type -i ^audio acl bypass_scan_types_req req_mime_type -i ^.*application\/x-mms-framed.*$ acl bypass_scan_types_rep rep_mime_type -i ^text/ acl bypass_scan_types_rep rep_mime_type -i ^application/x-javascript acl bypass_scan_types_rep rep_mime_type -i ^application/x-shockwave-flash acl bypass_scan_types_rep rep_mime_type -i ^image/ acl bypass_scan_types_rep rep_mime_type -i ^video acl bypass_scan_types_rep rep_mime_type -i ^audio acl bypass_scan_types_rep rep_mime_type -i ^.*application\/x-mms-framed.*$ loadable_modules /usr/local/lib/ecap_clamav_adapter.so ecap_service clamav_service_req reqmod_precache uri=ecap://e-cap.org/ecap/services/clamav?mode=REQMOD bypass=off ecap_service clamav_service_resp respmod_precache uri=ecap://e-cap.org/ecap/services/clamav?mode=RESPMOD bypass=on adaptation_access clamav_service_req allow !bypass_scan_types_req all adaptation_access clamav_service_resp allow !bypass_scan_types_rep all
- Note: On some setups you may need to create symbolic link in $prefix/clamav/share to DatabaseDirectory path, specified in clamd.conf. I.e, for example:
ln -s /var/lib/clamav /usr/local/clamav/share/clamav
This is due to semi-hardcoded db path in libclamav. Otherwise adaptation module will be crash Squid itself in current releases.
Co-existing both services in one setup
Both services can co-exists in one squid instance:
ecap_enable on # Bypass scan mime-types acl bypass_scan_types_req req_mime_type -i ^text/ acl bypass_scan_types_req req_mime_type -i ^application/x-javascript acl bypass_scan_types_req req_mime_type -i ^application/x-shockwave-flash acl bypass_scan_types_req req_mime_type -i ^image/ acl bypass_scan_types_req req_mime_type -i ^video acl bypass_scan_types_req req_mime_type -i ^audio acl bypass_scan_types_req req_mime_type -i ^.*application\/x-mms-framed.*$ acl bypass_scan_types_rep rep_mime_type -i ^text/ acl bypass_scan_types_rep rep_mime_type -i ^application/x-javascript acl bypass_scan_types_rep rep_mime_type -i ^application/x-shockwave-flash acl bypass_scan_types_rep rep_mime_type -i ^image/ acl bypass_scan_types_rep rep_mime_type -i ^video acl bypass_scan_types_rep rep_mime_type -i ^audio acl bypass_scan_types_rep rep_mime_type -i ^.*application\/x-mms-framed.*$ loadable_modules /usr/local/lib/ecap_clamav_adapter.so ecap_service clamav_service_req reqmod_precache uri=ecap://e-cap.org/ecap/services/clamav?mode=REQMOD bypass=off ecap_service clamav_service_resp respmod_precache uri=ecap://e-cap.org/ecap/services/clamav?mode=RESPMOD bypass=on adaptation_access clamav_service_req allow !bypass_scan_types_req all adaptation_access clamav_service_resp allow !bypass_scan_types_rep all acl HTTP_STATUS_OK http_status 200 loadable_modules /usr/local/lib/ecap_adapter_gzip.so ecap_service gzip_service respmod_precache ecap://www.vigos.com/ecap_gzip bypass=off adaptation_access gzip_service allow HTTP_STATUS_OK
BEWARE: Order is important! eCAP ClamAV adapter should precede Vigos adapter!