Squid Build Farm

The Squid project supports a big number of OSes (SquidFaq/AboutSquid has the short list), but it lacks the resources to test all of them, relying on user feedback instead. Aim of the Build Farm effort is to have a greater number of OS platforms directly available for unit-testing and development. The backbone of the build farm will be composed by a number of virtual machines running on eu.squid-cache.org.

FrancescoChemolli is leading this effort.

note: this information is mostly outdated. As of 2014-12-17 the build farm consists mostly of Docker-based containers running on top of VM donated by RackSpace. Project sysadmins can find documentation at /SystemAdministration and on the Docker farm at /DockerBuildFarm. An /AutoScale build farm is in the works.

The currently maintained farm consists of:

OS

Arch

Ver

system-name

Status

CentOS

x64

5.8

eu.squid-cache.org

{OK} Sponsored by Messagenet

x64

6.3

rs-centos-6

{OK} Sponsored by RackSpace; Installation guide

Fedora

x64

19

rs-fedora-19.squidfarm.kinkie.it

{OK} Sponsored by RackSpace

Ubuntu

x64

12.04 (precise)

rs-ubuntu-precise.squidfarm.kinkie.it

{OK} Sponsored by RackSpace

x64

14.04 (trusty)

rs-ubuntu-trusty.squidfarm.kinkie.it

{OK} Sponsored by RackSpace

x64

14.10 (utopic)

rs-ubuntu-utopic.squidfarm.kinkie.it

{OK} Sponsored by RackSpace

OpenSUSE

x64

12.1

opensuse-x64

{OK}

x64

12.3

rs-suse-12.squidfarm.kinkie.it

{OK} Sponsored by RackSpace

Mandriva

x32

2011.0

mandriva-x86

{OK}

Debian

ARM

6.0 (squeeze)

sheeva-debian-squeeze

{OK}

x64

wheezy

rs-debian-wheezy.squidfarm.kinkie.it

{OK} Sponsored by RackSpace

x64

sid

rs-debian-unstable.squidfarm.kinkie.it

{OK} Sponsored by RackSpace

x32

sid

rs-debian-unstable-x86.squidfarm.kinkie.it

{OK} Sponsored by RackSpace

OpenBSD

x64

4.8

obsd-48-x64

{OK} Installation guide

FreeBSD

x64

9.1

rs-fbsd-91.squidfarm.kinkie.it

{OK} Sponsored by RackSpace; Installation guide

MS Windows

MinGW x32

5.2 (2003 Server)

mswin

Sponsored.

MinGW x64

6.1 (7 Server)

mswin7

Sponsored.

Visual

Sponsored. Need setting up.

MacOSX

x64

thinkwell

Availability unreliable :(

Volunteer Help Sought:

32-bit systems running Linux flavors

Volunteers sought

Darwin and/or MacOS X

Volunteers sought

MS Windows Cygwin

Needed

AIX

Volunteers sought, PPC hardware sought

Donations of disk space and CPU time on non-x86 systems are welcome and encouraged.

The actual testing will be coordinated by Jenkins. Our instance

Slave Node software requirements

Since the test nodes are doing more than just building Squid from a prepared tarball there are additional development tools required:

Basics:

  • Java 1.6 or higher (required to run the Hudson slave)
  • lsb_release (for linux slaves)

Squid-3 Testing:

  • Bazaar
  • At least one C/C++ compiler: g++, clang++, and whatever the regular one for the OS is.
  • autoconf
  • automake / make
  • libtool (including libltdl-dev)
  • ed (scriptable command line editor)
  • CppUnit

  • OpenSSL development library (libssl-dev)
  • Optional extras for better testing:
    • libcap (version 2.09+)
    • po2html
    • po4a
    • libxml2
    • expat / libexpat

Registering a slave machine (as the machine owner)

  • Send an email to noc at squid dash cache dot org containing the following details:

    1. A public FQDN for your machine
    2. The operating system name, version, and bit-type (32-bit, 64-bit)
    3. The CPU type(s) of that machine
  • The address you send this email from is assumed to be your contact.

You will be contacted back by one of the administrators with details required to complete the activation sequence below.

Activating a slave machine (as the machine owner)

  • {i} $NODENAME is a name for your bot which is assigned by the build farm administrators.

  • Install java
  • Add a useraccount for the builds to run under
  • As that user, down load the slave jar to slave.jar

  • For unix: java -jar slave.jar -jnlpUrl http://build.squid-cache.org/computer/$NODENAME/slave-agent.jnlp. Put this in rc.local or something, to run on every startup.

  • For windows visit the slave web page and click on 'java web start', then in the java applet that starts, click the create a service option.

Setting up a hudson account

  • /!\ needed by administrators only

  • create a usercode (Use a simple username like 'myname' no spaces etc.

  • Grant access to this usercode to administer hudson

Setting up a slave machine (to run test builds)

  • /!\ needed by administrators only

  • Add the slave node. Use a simple name, doesn't need to be a hostname.

  • Configure job testnode to ensure the slave node is available and active with correct filesystem privileges. This must pass before its worth continuing to the remaining steps.

  • Configure job test-builddeps to ensure that the slave has the correct dependencies to actually build the source. This must pass before its worth continuing to the remaining steps. Note that this is still based on the real HEAD sources, so its best to run this when HEAD is expected to build clean.

When the node passes both of those tests:

/!\ Not needed anymore - labels should be automatically generated.

  • Configure the node labels list for the machine to have $ARCH $OS $VERSION $ARCH-$OS $OS-$VERSION $ARCH-$OS-$VERSION

If there is no job already present for this ARCH-OS-VERSION combination, AND the node is confirmed to be able to build the sources:

  • Create a new job Select copy-from existing and put in the name of one of the existing jobs for the same squid branch as the job to copy from. Give your new job a name like "3.HEAD-$ARCH-$OS-$VERSION". E.g. "3.HEAD-i386-FreeBSD-6.4".

  • Configure your new job, and change the tie this job to a node to select the ARCH-OS-VERSION of the new slave node.

Virtual nodes on eu.squid-cache.org

/!\ Administrator info

  • on the host /home/hudson/shared is dedicated to providing homedirs for hudson slaves
  • each slave mounts /home/hudson/shared/$nodename at /home/hudson
  • such clients should set /home/hudson as their root in the node config page.


  • Discuss this page using the "Discussion" link in the main menu

See Discussed Page

Wishlist for BuildFarm:

  1. have it properly interact with libvirt to start/stop slaves (FrancescoChemolli)

    • currently hard-stops rather than freezing.
  2. add build using ksh shell to detect bashism bugs
  3. setup Windows VisualStudio testing

  4. faster ARM slave
  5. develop matrix parametrized builds available (e.g. to test icc on centos) (AmosJeffries)

  6. find a way to embed the Hudson "traffic-light" and "weather" results live into this and other wiki pages.

BuildFarm (last edited 2015-12-10 22:07:47 by FrancescoChemolli)