Squid Web Cache wiki

Squid Web Cache documentation

🔗 Reworking the forwarding chain

Currently there is confusion within squid between protocols that we can have in a request:

And protocols we have a server implementation of:

And protocols we have a client implementation of:

Theres a patch to break out the server implementations - HTTPS, HTTP, ICP, HTCP. This possibly needs more work to be really polished, and is slated for 3.1.

Some work has been done on breaking out the protocols we can have in a request into a single clean set of classes, making it modular, but its not finished - and probably cant be until the protocols we implement clients of, and the connection between having a request object and actually handing it off to an external server, are decoupled.

This proposal tries to provide a consistent api for doing this that will allow:

🔗 Design

There are three key ways that requests are handled within squid:

The scheme field of each request is a reasonably good key to drive this: scheme->startFetch() will hand the request processing of to whatever the module wants to do. The module will be responsible for ensure that errors etc all occur correctly at this point.

Internally handled URL schemes will implement startFetch themselves. Most schemes however are not internal, and will instead be a subclass of ForwardableURLScheme. ForwardableURLScheme, for its startFetch method, creates a FwdState and initiates the existing FwdState process.

That is, a set of peers are selected, based on the peers configuration data - acls, netdb, and our data about the peer - cache digests etc, and possibly via an ICP lookup on the wire to the cache.

We’ll modify this lookup slightly. Firstly, we’ll add a method to URLScheme - the type of request->scheme - called ‘protocolClientAvailable’. This will return true if and only if squid has a native implementation of that protocol. If it returns false, we’ll require that it get forwarded via a peer. (Consider for instance WAIS, which we do not natively implement. We can only satisfy WAIS via a peer that does implement WAIS.)

At this point we have a set of peers to try. For each peer (including the direct-access one if that was permitted):

🔗 Examples


The ConnectURLScheme will be a subclass of ForwardableURLScheme Connect will return true for protocolClientAvailable() calls. The protocolClientFactory for connect will go straight into tunnel mode and tell the forward state that its sending data to the client and cannot be reforwarded. There is no CONNECT peer type.


The HTTPUrlScheme will be a subclass of ForwardableURLScheme HTTPUrlScheme will return true for protocolClientAvailable() calls. The protocolClientFactory for http will be an instance of HTTPClientFactory. This will return an HTTPClient, which is roughly what HTTPServerState is today, but only the wire level aspects.

There will be an HTTP Peer subclass. its protocolClientFactory attribute will be the same as the one for HTTPUrlScheme by default.


The HTTPSUrlScheme will be a subclass of ForwardableURLSche,e HTTPUrlSchceme will return true for protocolClientAvailable() calls. HTTPSUrlScheme will only be compiled in when SSL support is enabled. The protocolClientFactory will be a SSLClientFactory parameterised with a HTTPClientFactory instance. SSLClientFactory’s take a socket and perform SSL handshaking, after which they call the factory they were parameterised with - so the sequence is:

  1. aSSLClientFactory(fd, ProtocolClientRequest *requestor)
  2. Create a SSLClientEndpoint(fd)
  3. Create a SSLConnectionRequest(this, requestor)
  4. call endpointer->connect(theSSLConnectionRequest)
  5. Handshaking occurs
  6. theSSLConnectionRequest->connected() is called when the handshaking completes, or ->failed() if it fails to complete.
  7. on error we create return a global instance - FailedSSLProtocolClient, which will generate errors.
  8. on connected() we then call this->nestedProtocolClientFactory(theSSLClientEndpoint->fd, theSSLConnectionRequest). This will create the nested ProtocolClient - i.e. for HTTPS, this creates the HTTPClient, that is actually connected to the SSL client.
  9. The SSLClientEndpoint is given the HTTPClient to own as a reference, like the Socket owns the SSLClientEndpoint.
  10. The SSLClientEndpoint then calls the original Requestor with the HTTPClient.

There will be no HTTPSPeer subclass. Instead HTTPS peers will be an instance of HTTPPeer with the protocolClientFactory set to an instance of SSLClientFactory parameterised with a HTTPClientFactory.

This means that only one code path will create HTTPS wrappers for clients. This also means we can do Gophers or other such things like tls for any protocol.

🔗 Gopher

The GopherURLScheme will be a subclass of ForwardableURLScheme Connect will return true for protocolClientAvailable() calls. The protocolClientFactory for gopher will return a GopherClient object. There is no Gopher peer type, because our gopher implementation cannot forward other protocols.

Categories: WantedFeature

Navigation: Site Search, Site Pages, Categories, 🔼 go up