Analysis: Build PBS using CMake

Links

• Link to discussion on Developer Forum: <link to your project's discussion>

• Link to dev branch: https://github.com/minghui-liu/openpbs/tree/cmake

Overview

This page documents an exploration project of building PBS using CMake. This document begins with a brief introduction to CMake, followed a comparison of CMake with build tools we use today. Finally it details the changes that we need to make to build PBS using CMake.

What is CMake?

CMake is an open source system for managing the build and packaging process of software in a compiler-independent manner. It is designed to generate build files (makefiles on Unix/Linux and projects/workspaces in Windows MSVC) and be used with native build tools. Once integrated with PBS it will replace autotools which we currently use. It also ships with a packaging tool, CPack, that generates packages for different Unix/Linux distros and even windows installer. For more information please read Cmake website https://cmake.org/overview/

CMake vs Autotools

Summary of Changes

To integrate CMake we need the following changes:

1. Create a CMakeLists.txt file under each directory of PBS

2. Create a .cmake directory for custom CMake modules

3. Modify all files with .in extension, and replace configurable values with CMake style variable placeholders (@VARIABLE@)

4. Modify RPM scripts (pbs_postinstall, pbs_preunstainll, etc.) to accommodate CPack

5. Remove autotools files: configure.ac, autogen.sh, “.m4” file, Makefile.am, Makefile.in,

6. Remove RPM spec file (openpbs.spec.in)

Build PBS using CMake

CMake uses a hierarchy of configuration files named CMakeLists.txt (C and L must be capitalized). So one is created under each directory of PBS that contains something to be built or installed. Higher level CMakeLists.txt files include its subdirectories by using add_subdirectory() function. Here is an example of our root level CMakeLists.txt including the src and doc directories. 

# add the src subdirectory
add_subdirectory(src)
add_subdirectory(doc)

Variables and Options

CMake uses its own language that consists of functions (cmake calls them “commands”) and variables. Variables can be defined by using set() function. For example, in the top level CMakeLists.txt file, we create a variable of string type that stores the home directory of PBS with a default value and a description string. The “CACHE” keyword tells CMake to cache this variable between builds. Cached variables are stored in a file named CMakeCache.txt under the build directory.

set(PBS_HOME_DIR /var/spool/pbs CACHE STRING "PBS home directory")

Options are boolean variables that can be turned ON or OFF. They are usually used to enable or disable features (by turning them into C proprocessor definitions and use #ifdef in source code to conditionally compile features). For example, in the top level CMakeLists.txt we have an option to enable PTL with a default value of ON.

option(ENABLE_PTL "Enable PTL" ON)

Both variables and options can be overridden through the command line or graphical tools before a build.

Dependency Packages

PBS requires a number of packages to be installed on the system before it can be built. To find these libraries and packages we use the find_package() function and give it the name of the package. CMake has a list of common packages it knows how to find. CMake achieves this with build-in cmake modules, each one responsible for finding a package. However some of our dependencies, such as UndoLR, Editline and Ical, are not on this list. Therefore we created our own cmake module to find them. These custom modules were placed under the .cmake directory of PBS.

find_package(Python3 COMPONENTS Interpreter Development REQUIRED)

CMake can find some packages by component. Here we require both the interpeter (python3) and the development (python3-devel) package of Python3.

The find_package function and modules store result in variables. By convention these variables are prefixed with the name of the package. For example, after we tell cmake to find python3, variables including Python3_FOUND, and Python3_INCLUDE and Python3_LIBRARIES etc. will be set, the first containing whether Python3 was found, the later two containing the path to Python header files and Python3 libraries. These variables will be useful later when we conditional compile Python features, and need to link to Python3.

System Introspection

Some PBS features require certain system capabilities. CMake offers many ways to let us inspect the system for features, such as checking if a header file exist on the system, or checking whether a symbol exists in a header file. We can use them to find out if features that PBS depend on, such as evenfd and epoll is available on the system. Below is an example of checking for sys/eventfd.h and store the resulst in HAVE_SYS_EVENTFD_H vairable.

check_include_file(sys/eventfd.h HAVE_SYS_EVENTFD_H)

And below is how we figure out if the system has epoll by checking for epoll_create and epoll_pwait in the sys/epoll.h header, and store the result in PBS_HAVE_EPOLL and PBS_HAVE_EPOLL_PWAIT.

check_symbol_exists(epoll_create sys/epoll.h PBS_HAVE_EPOLL)
check_symbol_exists(epoll_pwait sys/epoll.h PBS_HAVE_EPOLL_PWAIT)

check_type_size() lets us to check if a type size is defined.

check_type_size(socklen_t HAVE_SOCKLEN_T)

If an os features cannot be figured out from header files and symbols, CMake has allows us to check features by testing if a block of c code compiles or runs. For example, here is a another way of checking if the system has epoll, and store the result in PBS_HAVE_EPOLL.

check_c_source_compiles("
#include <sys/epoll.h>
int main()
{
return ((epoll_create(100) == -1) ? -1 : 0);
}
" PBS_HAVE_EPOLL)

All of these checks are placed in /src/include/CMakeLists.txt.

File Generation

With all the information about packages and system features stored in cmake variables, we can now use them to generate headers, C source files and other files that are needed to build PBS.

For generating header files, cmake offers two handy functions, #cmakedefine and #cmakedefine01 that turn cmake variables and options into C preprocessor definitions. #cmakedefine #define or #undef a definition depending on if the cmake variable has a non-false value. #cmakedefine01 always defines a definition but gives it value 0 or 1 depending on if the cmake variable is set to non-false value. For example in pbs_config.h.in:

/* This will be converted to
 *   #define PBS_HAVE_EPOLL
 * or 
 *   #undef PBS_HAVE_EPOLL
 */
#cmakedefine PBS_HAVE_EPOLL

/* This will be converted to
 *   #define PBS_HAVE_EPOLL 1
 * or 
 *   #define PBS_HAVE_EPOLL 0
 */
#cmakedefine01 PBS_HAVE_EPOLL

For other types of files, and for using values of cmake variables in files directly, CMake has two formats of variable placeholders, @VARIABLE@ and ${VARIABLE}. The first one is prefered because the second one conflicts with shell script variables.

To generate (cmake calls this step 'configure') files and replace cmake variables with values, we can use configure_file() function in the CMakeLists.txt file under the same directory as the file to be generated. To tell cmake to ignore the ${VARIABLE} format we can append @ONLY at the end. For example, in src/include/CMakeLists.txt we ask cmake to generate pbs_config.h from pbs_config.h.in, and only replace variables in the @VARIABLE@ format.

configure_file(pbs_config.h.in pbs_config.h)

Besides generating files by variable replacement from an input file, CMake also supports custom file generation by using add_custom_command() and add_custom_target(). We need to supply the dependency (input file), output file name and the formula for generating the file.

The example below shows how to generate libattr header files from xml attribute defintions using attr_parser.py. Instead of calling add_custom_command for each attr def file, a function named gen_attrdef_file is created and and called with the file names to save repeatition.

The example below is how we generate job_attr_def.c from master_job_attr_def.xml by using buildutils/attr_parser.py.

add_custom_command(OUTPUT job_attr_def.c
  COMMAND ${Python3_EXECUTABLE}
  ${CMAKE_SOURCE_DIR}/buildutils/attr_parser.py
  -m ${CMAKE_SOURCE_DIR}/src/lib/Libattr/master_job_attr_def.xml
  -s ${CMAKE_CURRENT_BINARY_DIR}/job_attr_def.c
  MAIN_DEPENDENCY ${CMAKE_SOURCE_DIR}/src/lib/Libattr/master_job_attr_def.xml
)

Since we do this for all the attribute definition files under src/lib/libattr, we can create a function for this to avoid repeating.

function(gen_attrdef_file name)
	add_custom_command(OUTPUT ${name}.c
		COMMAND ${Python3_EXECUTABLE}
			${CMAKE_SOURCE_DIR}/buildutils/attr_parser.py
			-m ${CMAKE_CURRENT_SOURCE_DIR}/master_${name}.xml
			-s ${CMAKE_CURRENT_BINARY_DIR}/${name}.c
		MAIN_DEPENDENCY ${CMAKE_CURRENT_SOURCE_DIR}/master_${name}.xml)
endfunction()
gen_attrdef_file(job_attr_def)
gen_attrdef_file(node_attr_def)
gen_attrdef_file(queue_attr_def)
gen_attrdef_file(resc_def_all)
gen_attrdef_file(resv_attr_def)
gen_attrdef_file(sched_attr_def)
gen_attrdef_file(svr_attr_def)

Note that cmake does not place generate files under the same directory, but creates a output tree (binary tree in cmake terms) under your build folder where all generated files and built executables and libraries go. So if you called cmake .. from a folder called build, cmake will place the output of src/include/pbs_config.h.in to build/src/include/pbs_config.h . To distingushi between the source tree and the binary tree, we can use ${CMAKE_SOURCE_DIR} and ${CMAKE_BINARY_DIR} prefix.

Now we can create a build directory, cd into it and call cmake [path to root dir of PBS] under build directory.

mkdir build && cd build
cmake ..

You should see the generate CMake files, and the binary tree.

Targets

With all the required files generated, we list all targets that need to be built. Targets can be anything. It is simply an abstraction of the build system for tracking dependencies. But typically we list libraries and executables and targets. The example below is how the static library liblog is built, and how include directory and linker information is given to cmake. Note that CMake automatically appends lib prefix to a library name. 

add_library(log STATIC
  pbs_log.c
  chk_file_sec.c
  log_event.c
  pbs_messages.c
  setup_env.c
)
target_include_directories(log 
  PRIVATE
  ${CMAKE_SOURCE_DIR}/src/include
  ${CMAKE_BINARY_DIR}/src/include
  ${CMAKE_BINARY_DIR}/src/lib/Libattr
)
target_link_libraries(log
  PUBLIC
  ${CMAKE_BINARY_DIR}/src/lib/Libnet/libnet.a
  ${CMAKE_BINARY_DIR}/src/lib/Libutil/libutil.a
  ${CMAKE_BINARY_DIR}/src/lib/Libpbs/libpbs.a
)

Add_library() tells cmake about a library target and its source files. target_include_directories() tells CMake its include directories (where to look for header files, think the -I option of gcc), and target_link_libraries() tells CMake where to find the libraries the target links to (-L of gcc). The PRIVATE and PUBLIC are inheritance keywords (there is a third one, INTERFACE). They determine how a target's property transfers to other targets that link to it.

Executable targets are created by using add_executable() function in a similarly way. Here is how we build pbs_probe in src/tools/CMakeLists.txt.

add_executable(pbs_probe
  pbs_probe.c
  ${CMAKE_SOURCE_DIR}/src/lib/Libcmds/cmds_common.c
)
target_include_directories(pbs_probe
  PRIVATE
  ${COMMON_INCLUDES}
  ${Python3_INCLUDE_DIRS}
)
target_link_libraries(pbs_probe
  PRIVATE
  ${COMMON_LIBS}
  ${Python3_LIBRARIES}
  ${CMAKE_DL_LIBS}
)

In this example we use many variable instead of hardcoded paths. The ${Python3_INCLUDE_DIRS} and ${Python3_LIBRARIES} variables were from find_package(Python3 ...) . ${COMMON_INCLUDES} and ${COMMON_LIBS} are two we created that contain lists of libraries and include files that all executables under src/tools share.

After this step, we will be to run cmake and then make (on Linux) to build PBS.

Install Rule

Now that libraries and excutables targets are listed, we need to decide how they are installed. The install rules for targets and files are given to CMake by the install() function. These decide where things get installed to and what permission they are set to. They apply both to manual install (make install on Unix/Linux) and to package install.

As an example, here we install binaries under src/tools to sbin directory.

install(TARGETS
  pbs_ds_monitor
  pbs_idled
  pbs_probe
  pbs_upgrade_job
DESTINATION ${CMAKE_INSTALL_SBINDIR})

DESTINATION can be relative path, absolute path or variable. Here we use GNU install path variables as relative path. They are basically string variables that contain relative paths for different types of files, such as doc, lib, sbin, etc. For a comprehensive list of all the available GNU install directory variables please see here. All relative paths are relative to ${CMAKE_INSTALL_PREFIX}, which is set to a default of value of /opt/pbs in our top level CMakeLists.txt.

Besides targets, install() can be used to install files. There are different file type keywords, such as FILES for normal non-executable files, PROGRAMS for executable files, and DIRECTORY for directories. Here is an example of how we install shell scripts under src/cmds/scripts to ${CMAKE_INSTALL_PREFIX}/sbin. Because we used PROGRAMS keyword, CMake will correctly install these scripts with executable permission.

install(PROGRAMS
  pbs_dataservice
  pbs_ds_password
  pbs_server
  pbs_snapshot
  DESTINATION ${CMAKE_INSTALL_SBINDIR}
)

In the example below we install modulefile to ${CMAKE_INSTALL_PREFIX}/etc.

install(FILES
  ${CMAKE_CURRENT_BINARY_DIR}/modulefile
  DESTINATION ${CMAKE_INSTALL_SYSCONFDIR}
)

At this point, we can install PBS by make install under the build directory.

Packaging

CMake has a package generation tool name CPack that can generate packages in multiple formats at once. To use CPack we need to give CMake information about packaging by setting variables. There are variables generic to all package generators, and generator specific variables. For example, below are generic variable such as package name, vendor, version, description etc. At the end we set the default list of package formats to only RPM.

set(CPACK_PACKAGE_VENDOR "Altair Inc.")
set(CPACK_PACKAGE_VERSION_MAJOR "${PBS_MAJOR_VERSION}")
set(CPACK_PACKAGE_VERSION_MINOR "${PBS_MINOR_VERSION}")
set(CPACK_PACKAGE_VERSION_PATCH "${PBS_PATCH_VERSION}")
set(CPACK_RESOURCE_FILE_LICENSE "${CMAKE_CURRENT_SOURCE_DIR}/LICENSE")
set(CPACK_RESOURCE_FILE_README "${CMAKE_CURRENT_SOURCE_DIR}/README.md")
set(CPACK_PACKAGE_DESCRIPTION "OpenPBS descriotion")
set(CPACK_PACKAGE_DESCRIPTION_SUMMARY "Summary placeholder")
set(CPACK_PACKAGE_HOMEPAGE_URL "https://www.openpbs.org")
set(CPACK_PACKAGING_INSTALL_PREFIX "/opt/pbs") # or optionally, set CPACK_SET_DESTDIR to ON
set(CPACK_GENERATOR "RPM")

The following are variables we set that are specific to the RPM generator.

# RPM package specific variables
set(CPACK_RPM_POST_INSTALL_SCRIPT_FILE ${CMAKE_BINARY_DIR}/rpm/server/pbs_server_postinstall)
set(CPACK_RPM_POST_TRANS_SCRIPT_FILE ${CMAKE_BINARY_DIR}/rpm/server/pbs_server_posttrans)
set(CPACK_RPM_PRE_UNINSTALL_SCRIPT_FILE ${CMAKE_BINARY_DIR}/rpm/server/pbs_server_preuninstall)

To generate packages, call cpack from the build directory. Or do make package or

cmake --build . --target package

To generate the source package, call cmake --build . --target package_source or make source.

Demo

If you want to try building PBS using CMake yourself, please git clone the development branch and do the following.

cd openpbs
mkdir build
cd build
cmake ..
make
make install
// or call cpack, then yum install the generated RPM package

Future Work

1. Complete all PBS build options and system features tests

2. Correctly mark all target attributes with inheritance keywords. Now most of them are marked as PRIVATE.

3. Generate multiple RPM packages. Currently only a single RPM package that contains all PBS files is generated. We need to split it into server, client, execution, devel and ptl packages.

4. Generate package in more formats, such as DEB and Windows installer.

5. Test on more platforms. Currently building PBS using CMake is only tested on CentOS 7.

OSS Site Map

Project Documentation Main Page

Developer Guide Pages