HowTo Build NST 20

Building NST 20

The following page provides details on building the NST 20 yum repository and ISO images directly from source code. The procedure boils down to the following steps:

• Setting up a development machine.
• Checking out copy of the source code.
• Installing the necessary packages to build.
• Building the yum repository and ISO image.

There appears to be issue with building the NST ISO image if you are using a GNOME 3 desktop and logged in as the root user. You are strongly discouraged from attempting to build the NST ISO image when logged into a GNOME 3 desktop.

Do not use sudo when invoking commands to build the NST. If you are logged in as a non-root user, the build process will automatically insert the necessary sudo invocations as needed.

Setting Up A Development Machine

In order to set up a development machine, you will need to download either the 32 bit or 64 bit Live Installable version of Fedora 20 (http://fedoraproject.org/get-fedora). Choose the 32 bit version if you want to build the 32 bit version of NST and choose the 64 bit version if you want to build the 64 bit version of NST. If you want to build both a 32 bit version and 64 bit version of the NST, you will need to set up two development machines.

You can use a virtual environment as your development machines. If you do this, I'd recommend allocated at least 2GB of RAM and 40 GB of disk space and two CPUs. Even though it takes longer to build within a virtual environment, I find that the convenience is typically worth it. I typically use VirtualBox (https://www.virtualbox.org/) for my personal builds. The NST project uses VMware (http://www.vmware.com/) environments for nightly test builds.

Be aware that there are parts of the build process which require root access. Because of this, it is recommended that you set up a dedicated machine for NST development.

Regardless of whether you are using real hardware or virtual hardware, after you install Fedora 20, make sure you do the following:

• Update the /etc/sudo file such that your account (or wheel group) can run the sudo command without being prompted for a password (NOTE: you can skip this step if you will be using the root account).

sudo visudo

• Update all of the packages on the system:

sudo yum update

• Reboot the system to load the newest kernel:

sudo reboot

• Install/update any device drivers for you hardware or virtual environment.
• Make sure that the subversion package is installed.

sudo yum install subversion 

• Reboot the system to make sure the device drivers load properly.

sudo reboot

Check your PATH setting and make certain that /usr/bin appears before the /bin directory (or omit the /bin directory entirely). Here is the PATH setting we use on our NST nightly build machines:

[root@dev20-64 ~]# echo $PATH
/usr/lib64/qt-3.3/bin:/usr/lib64/ccache:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin
[root@dev20-64 ~]# 

We are uncertain as to why this is required, but do not skip this step as we have systems where the RPM builds had dependencies on files like /bin/perl and /bin/python. The RPMs would build cleanly with these dependencies, but the livecd-creator tool would then fail to build the ISO image when it tried to resolve the /bin dependencies.

Check Out The Source Code

Once you have your Fedora 20 development machine set up, you can check out a copy of the NST source code. It is recommended that you check out the repo area from subversion as this is the active area that is kept up to date.

The following commands will check out a copy of the repo area to the $HOME/repo directory on your system:

cd $HOME
svn checkout http://svn.code.sf.net/p/nst/code/repo nst

The repo area is the more stable area of the NST source code and the development area we use when we build ISO releases or RPM packages that are suitable for pushing out to other machines. If you prefer to live a little closer on the edge, you can use the dev area which heavy (think daily) changes are occurring. To check out the dev area instead of the repo area, use the following command:

cd $HOME
svn checkout http://svn.code.sf.net/p/nst/code/dev/20 nst

It may take a bit, but the above should get a current copy of what we are currently using to build the current version of the NST distribution.

Initial Configuration

When you run the top level ./configure script, the configuration script may automatically be able to determine what packages are missing and add them to your development system. In order for the script to do this, you must either be logged in as the root user or using a account which has sudo privileges. To check whether or not your account has sudo privileges, try running the following:

sudo tail /var/log/messages

If the above command fails, it means you skipped a step during the development machine setup section. You will need to fix this before continuing.

If the above command passes (or you are logged in as root), you should be able to run the following commands:

cd $HOME/nst
./configure

The first invocation will take a long time as many packages will need to be installed onto your development machine in order to make it capable of building the NST.

If the configuration script does not complete successfully, it indicates one of the following issues:

• A core package needs to be installed by hand. In this case, a error message should be displayed showing you what yum command to run in order to install the core package.
• There was a networking issue in downloading and installing one of the required packages.
• One of the packages that needed to be built and installed locally failed. This typically indicates that the remote server where the source is downloaded from was temporarily down (or that we need to update the source URL for the package in the NST source code).

Initial Build

Once you have successfully completed the configuration process, you should be able to build the NST yum repository and NST ISO image using the following commands:

cd $HOME/nst
make

If things work correctly, this should take a long time (many hours) and produce:

livecd/nst-20.x86_64.iso or livecd/nst-20.i686.iso

The 64 bit or 32 bit ISO image which can be burned to DVD or installed on a USB memory stick (using the live-iso-to-disk command).

yum/repo

The directory containing the NST yum repository. You can push this out to a web server if you want to maintain your own yum repository for your deployed NST systems to get updates from.

Subsequent Builds

Subsequent builds will run much faster (less than two hours) as less things should need to be downloaded, installed and built. Use the following commands to update your subversion area to the latest code and rebuild the NST yum repository and ISO image:

cd $HOME/nst
svn update && ./configure && make

Troubleshooting Notes

A build may fail for one of the following reasons:

• One of the web servers (there are many) that the build downloads source code is unreachable at the time of your build. For example, if the http://www.thc.org/ web site is down, you won't be able to download the source code for the hydra package. In this case the build will fail if you don't already have a copy of the hydra source code downloaded to your yum/cache/downloads directory.
• The download URL for a source package is no longer valid. For example if there is a new release of the wireshark source code and current release the NST is trying to download a older release which is no longer available from the wireshark web site. In this case the build will fail if you don't already have a copy of the wireshark source code downloaded to your yum/cache/downloads directory.
• You have not set up your development machine correctly.
• You did not check out the correct area of the source code (this is the area we actively maintain).
• Your source code is not up to date (svn update && ./configure).
• We made a mistake when merging updates into the repo area. If you could notify us via the NST forum (http://sourceforge.net/p/nst/discussion/361524/) we would appreciate it.

If the build fails because it was unable to download the source code for a particular package it will probably tell you the name of the file it was trying to download. If you can find this file elsewhere on the Internet (or in the yum/cache/downloads directory of another of your NST build machines), you can copy it into the yum/cache/downloads directory on your build machine to work through the process.

Unfortunately, the easiest method to get a successful build often requires some patience. If a build fails one day, it may just be easier to wait another day, update, configure and try again. Once you get your first successful build things typically go much more smoothly (and quickly) as files are cached.

Customizing Your Build

There are many command line options you can specify to the top level ./configure script. To see the various options that can be specified, run:

./configure --help | less

Instead of specifying a long set of parameters each time when invoking the ./configure command, it is recommended that you use standard configuration files to configure your build. The remaining sections will describe the files which can be used to customize your build.

Create the Configuration Directory ($HOME/.nst)

By default, when your run the ./configure script, it will assume the configuration directory is: $HOME/.nst. You can create this directory with the following command:

mkdir $HOME/.nst

$HOME/.nst/configure.sh

This file will be sourced by the ./configure script and allow you to override variables which change the build. The following shows a sample template of the most common variables you might want to adjust:

# Set the default password
NSTPASSWD="myNewPasswd";

#
# These are only used if your run "make test" or "make probe-check"
# from the top level directory
#

# Set the TEST_EXTENDED variable to "true" to enable test that take a long time to complete
TEST_EXTENDED="false";

# Following are related to tests requiring SMTP server access (tests that generate email)
TEST_SMTP_SERVER="smtp.linux.bogus";
TEST_SMTP_DOMAIN="linux.bogus";
TEST_SENDMAIL_FROM="first.last@linux.bogus";
TEST_SENDMAIL_TO="first.last@destination.bogus";

There used to be a lot more variables that could be "tweaked" in this file, but most of them have become deprecated since version 2.11.0 of the NST. You can review the file include/data/configure.in for a list of some of the other variables that can be overridden.

$HOME/.nst/custom.ks

This file (if present) is "tacked" onto the kickstart file used to create the NST live ISO image. If you are familiar with kickstart files, you can use this to add other packages to your NST distribution.

For example, lets assume that you wanted the following things to be done in your custom NST build:

• Include both the gcc and kernel-devel packages (as well as all the packages they depend on).
• Transfer the contents of your $HOME/.ssh configuration from the development system to the ISO image being built.
• Set the default proxy configuration for your local area network.

To accomplish this, you would create the file: $HOME/.nst/custom.ks with the following contents:

# Make google yum repo available for ISO build
repo --name=google-chrome --baseurl=http://dl.google.com/linux/chrome/rpm/stable/x86_64

# This %packages block is used to indicate what extra packages we
# would like to include in the Live ISO image. The packages listed
# here along with any required dependencies will be added to the ISO.

%packages
# Include google chrome in custom build
google-chrome-beta

# Pre-install gcc and kernel-devel to simplify installing VirtualBox
# guest additions
gcc
kernel-devel

# You can add additional packages here as long as they are part of the
# Fedora or NST yum repositories, if they are not there, you will
# need to add "repo" line similar to what is shown above for Google Chrome
%end


# This code runs non-chrooted. This allows us to transfer files from
# the host development system into the build image under $INSTALL_ROOT
#
# Useful variables:
#
# LIVE_ROOT - Top level directory of all the file that will appear in the
# ISO image.
#
# INSTALL_ROOT - The directory that is used to make the compressed file system
# image (/) that you see after booting (you'll find: /etc, /var, /usr, ... under here).

%post --nochroot

# Install our ssh keys
if [ -d /root/.ssh ]; then
  [ -d ${INSTALL_ROOT}/root/.ssh ] && /bin/rm -fr ${INSTALL_ROOT}/root/.ssh;
  cp -rp /root/.ssh ${INSTALL_ROOT}/root/.ssh;
fi

%end


# This %post block does not include the --nochroot option. It
# runs in a chroot environment under file system being created
# for the live boot (its as if you are running inside the live boot
# and unable to see the full development filesystem).

%post

# Set default proxy configuration
nstsetproxy --http-host 192.168.1.2 --http-port 3128

%end

This simple $HOME/.nst/custom.ks file provides the necessary hooks to allow you to extend, customize and tweak your NST ISO build as much as you want.

For more information about kickstart files, see: http://fedoraproject.org/wiki/Anaconda/Kickstart.

NOTE: The $HOME/.nst/custom.ks feature did not become available until version 2.13.0 of the NST.

Customizing The NST WUI CSS

If you would like to customize the look of the HTML produced by the nstweb and nstwui packages, you can do so by creating the file $HOME/.nst/site.css. The CSS commands which you specify within this file will override the default CSS provided by the packages.

For example, if you prefer to have less color in your tooltips, create the $HOME/.nst/site.css file and add the following:

/* Reduce the number of colors used in the tooltips */

div.niceTitle .ttCommand,
div.niceTitle .ttDocument,
div.niceTitle .ttSite,
div.niceTitle .ttNav,
div.niceTitle .ttValue,
div.niceTitle .ttAction,
div.niceTitle .ttAcronym,
div.niceTitle .ttEmphasis,
div.niceTitle .ttEmphasisPFont,
div.niceTitle .ttNote,
div.niceTitle .ttNotePFont {
  color: white;
}

When the nstweb and nstwui packages are built, the contents of $HOME/.nst/site.css will be appended to /usr/share/nstweb/css/domtt.css and /usr/share/nstwui/domtt.css as these normally the last CSS files are included on most of the HTML pages.