Page Comparison

Overview:

This is a new proposed feature providing Kerberos support with GSS-API layer in PBS Pro. The Kerberos support includes distributing and renewing of Kerberos credentials for users although the credentials itself are provided by an external renew-tool on demand. The renew-tool is configurable and is not part of this work. The passwordless access between nodes (and frontend) for users can be ensured by Kerberos credentials with this feature.

...

First, GSS-API in short: GSS-API provides authentication by default and besides that, it can provide encryption between client and server. So, the identity is always verified by GSS-API and we can simply check the obtained principal with ACL. The encryption needs a GSS context on both the client and the server sides. The GSS context is used for further messages encryption. For acquiring the GSS context, we need to have valid credentials on both sides. These credentials are used for the handshake. The handshake means that the client and the server exchange messages - so-called 'tokens' as long as they need for establishing the GSS context. The client starts to establish the GSS context and sends the first token to the server. The server reads it, processes it, and responds with another token (if needed). The token exchange lingers in a loop or in some asynchronous exchange until the GSS context is established on both the client and the server. Once we have the GSS context, we can use it for encryption for further messages - it is called wrapping a message.

The GSS part has similar logic on both the TCP and the TPP. It uses the same pbs_gss* routines for establishing context and for un/wrapping messages. These routines can be found in Libutil/pbs_gss.c and use the gss_extra structure. This structure is added either to TCP connection structure or to TPP connection structure and it holds the information about GSS (like the GSS context).

TCP part:
With TCP, the GSS context is established in a tight loop on the client-side. On the server-side, the GSS tokens with context are received asynchronously. It means that the client initiates the handshake, sends the first token and if another token is needed, the client waits for it. The server asynchronously reads the token and processes it and if needed sends reply token. TCP supports both the cleartext and encryption. Once the GSS context is established then only encrypted messages can be exchanged with a particular client.
- Implementation notes:
  - A new cn_ready_func function is added. This function handles the handshake and the function process_request() is called only if cn_ready_func returns true, which means that the handshake is finished and some data for process_request() are ready.
  - The regular dis_* handlers are replaced with new dis_gss_* handlers and the dis_gss_* handlers call the regulars handlers. This way the GSS layer is isolated and dis_gss_* is stacked on regular dis_*. The dis_gss_wflush is called before dis_wflush and it wraps the message before sending.
TPP part:
With TPP, the message exchange is asynchronous on both sides. The communication is always encrypted and no cleartext is supported once the GSS is enabled. The pbs_comm is the GSS server. PBS server, PBS mom, and the scheduler are the GSS clients. The GSS encryption also works between pbs_comms (in that case the comm who initiates the connection is the GSS client).
- Implementation notes:
  - The GSS layer replaces the tpp handlers - leaf or router - (pkt_presend_handler, pkt_postsend_handler, pkt_handler, close_handler, post_connect_handler, timer_handler) with gss_* handlers and the regular leaf or router tpp_handlers all called within gss_* handlers.
  - The handshake is asynchronous and once the handshake is finished then the leaf or router post_connect_handler is called and the encrypted communication begins.

...

Kerberos server setup:

Install Kerberos server packages:
[root@pbspro-mit ~]# yum -y install krb5-libs krb5-server krb5-workstation

Config /etc/krb5.conf. The example is:

Code Block

[logging]
 default = FILE:/var/log/krb5libs.log
 kdc = FILE:/var/log/krb5kdc.log
 admin_server = FILE:/var/log/kadmind.log

[libdefaults]
 dns_lookup_realm = false
 dns_lookup_kdc = false
 ticket_lifetime = 8h
 renew_lifetime = 8h
 forwardable = true
 rdns = false
 pkinit_anchors = /etc/pki/tls/certs/ca-bundle.crt
 default_realm = PBSPRO

[realms]
 PBSPRO = {
  kdc = pbspro-mit
  admin_server = pbspro-mit
 }

[domain_realm]
 pbspro-mit = PBSPRO

Config /var/kerberos/krb5kdc/kdc.conf The example is:

Code Block

[kdcdefaults]
 kdc_ports = 88
 kdc_tcp_ports = 88

[realms]
 PBSPRO = {
  acl_file = /var/kerberos/krb5kdc/kadm5.acl
  dict_file = /usr/share/dict/words
  admin_keytab = /var/kerberos/krb5kdc/kadm5.keytab
  supported_enctypes = aes256-cts:normal aes128-cts:normal des3-hmac-sha1:normal arcfour-hmac:normal camellia256-cts:normal camellia128-cts:normal des-hmac-sha1:normal des-cbc-md5:normal des-cbc-crc:normal
 }

Now, the database needs to be created. Make up your own db_password.
[root@pbspro-mit ~]# kdb5_util create -s -P <db_password>
Loading random data
Initializing database '/var/kerberos/krb5kdc/principal' for realm 'PBSPRO',
master key name 'K/M@PBSPRO'
Once the database is created, we will create the principals for the host and for the test user.
[root@pbspro-mit ~]# kadmin.local addprinc -randkey host/$(hostname -f)
[root@pbspro-mit ~]# kadmin.local addprinc -pw <user_password> test
Now, we need to add the existing principals to the keytab. The user principal in the keytab is used by the testing renew-tool. It is possible to use different keytab for users, but the host principal is expected in the default keytab on each PBS node (including the PBS server).
[root@pbspro-mit ~]# kadmin.local ktadd -norandkey -k /etc/krb5.keytab host/$(hostname -f)
Entry for principal host/pbspro-mit ...
[root@pbspro-mit ~]# kadmin.local ktadd -norandkey -k /etc/krb5.keytab test
Entry for principal test ...
Enable and start kdc server:
root@pbspro-mit ~]# systemctl enable krb5kdc.service
Created symlink from /etc/systemd/system/multi-user.target.wants/krb5kdc.service to /usr/lib/systemd/system/krb5kdc.service.
[root@pbspro-mit ~]# systemctl start krb5kdc.service

Kerberos client:
- Install Kerberos client packages:
  [root@pbspro-mit ~]# yum -y install krb5-libs krb5-server krb5-workstation
- Test of getting credentials by the test user:
  [root@pbspro-mit ~]# su test
  [test@pbspro-mit root]$ cd
  [test@pbspro-mit ~]$ kinit
  Password for test@PBSPRO:
  [test@pbspro-mit ~]$ klist
  Ticket cache: FILE:/tmp/krb5cc_1000
  Default principal: test@PBSPRO
  Valid starting Expires Service principal
  02/05/2019 09:25:18 02/05/2019 17:25:18 krbtgt/PBSPRO@PBSPRO
Installation PBS Pro with Kerberos support:
1. Install packages requested by PBS Pro.
2. Install packages requested for building PBS Pro with Kerberos support:
  [root@pbspro-mit ~]# yum -y install krb5-libs krb5-devel libcom_err libcom_err-devel
3. Checkout PBS Pro with Kerberos support:
  [root@pbspro-mit ~]# git clone https://github.com/PBSPro/pbspro.git
4. Build and install PBS Pro with Kerberos support:
  [root@pbspro-mit pbspro]# ./autogen.sh
  [root@pbspro-mit pbspro]# CFLAGS="-g -ggdb -Wall -Werror" ./configure --prefix=/opt/pbs --with-krbauth PATH_KRB5_CONFIG=/usr/bin/krb5-config
  [root@pbspro-mit pbspro]# make
  [root@pbspro-mit pbspro]# make install
  [root@pbspro-mit pbspro]# chmod 4755 /opt/pbs/sbin/pbs_iff /opt/pbs/sbin/pbs_rcp
  [root@pbspro-mit pbspro]# /opt/pbs/libexec/pbs_postinstall
5. Enable pbs_mom:
  [root@pbspro-mit ~]# sed -i 's/PBS_START_MOM=0/PBS_START_MOM=1/' /etc/pbs.conf
6. Set PBS_AUTH_METHOD to GSS:
  [root@pbspro-mit ~]# echo "PBS_AUTH_METHOD=GSS" >> /etc/pbs.conf
7. Restart PBS service:
  [root@pbspro-mit ~]# systemctl restart pbs
8. Use qmgr to create default queue and add pbspro-mit as a pbs node.
Build a PBS renew-tool. The renew-tool is used for obtaining user credentials. You are free to develop, use, and share your own renew-tool. This simple renew-tool is for testing purpose only and it obtains credentials from keytab. Note: It is possible to provide a guide on how to use a proper renew-tool - krb525_renew (https://github.com/CESNET/krb525) but this tool consists of a client part and server part and the server part requests the KDC server to be Heimdal. This limitation is the reason why a simple tool is used. The guide for krb525_renew will be created on demand - if requested.
1. Clone the tool:
  [root@pbspro-mit ~]# git clone https://github.com/PBSPro/pbspro.git
2. Build the tool:
  [root@pbspro-mit ~]# cd pbspro/src/unsupported/renew-test/
  [root@pbspro-mit renew-test]# make
Configure Kerberos in PBS Pro:
1. Use following (or similar) qmgr commands:
  [root@pbspro-mit ~]# PBSPRO_IGNORE_KERBEROS= qmgr
  Max open servers: 49
  Qmgr: set server acl_krb_realm_enable = True
  Qmgr: set server acl_krb_realms = *@PBSPRO
  Qmgr: set server acl_krb_submit_realms = *@PBSPRO
  Qmgr: set server cred_renew_enable = True
  Qmgr: set server cred_renew_tool = "/root/pbspro/src/unsupported/renew-test//renew-test"
  Qmgr: set server cred_renew_period = 05:00:00
  Qmgr: set server cred_renew_cache_period = 10:00:00
Submit a job as the user 'test' and check the availability of credentials within the job:
[test@pbspro-mit ~]$ kinit
Password for test@PBSPRO:
[test@pbspro-mit ~]$ qsub -I
qsub: waiting for job 0.pbspro-mit to start
qsub: job 0.pbspro-mit ready
[test@pbspro-mit ~]$ klist
Ticket cache: FILE:/tmp/krb5cc_pbsjob_0.pbspro-mit
Default principal: test@PBSPRO
Valid starting Expires Service principal
02/05/2019 10:00:45 02/05/2019 18:00:45 krbtgt/PBSPRO@PBSPRO
Add another pbs node into this infrastructure:
1. Assuming the hostname of a new node is pbspro-mit-node1
2. Use the KDC server to add a new principal into the database:
  [root@pbspro-mit ~]# kadmin.local addprinc -randkey host/pbspro-mit-node1@PBSPRO
3. Create keytab for this host (use different location of keytab):
  [root@pbspro-mit ~]# kadmin.local ktadd -norandkey -k /etc/krb5-node1.keytab host/pbspro-mit-node1@PBSPRO
4. Copy the keytab to the default location on the new node (be careful to preserve the file permission):
  [root@pbspro-mit ~]# scp /etc/krb5-node1.keytab pbspro-mit-node1:/etc/krb5.keytab
5. Copy krb5.conf to the new node:
  [root@pbspro-mit ~]# scp /etc/krb5.conf pbspro-mit-node1:/etc/krb5.conf
6. Install clients packages for Kerberos on the new node:
  [root@pbspro-mit-node1 ~]# yum -y install krb5-libs krb5-workstation
7. Start the pbs service.

New Interfaces:

Interface: New option '--with-krbauth' to configure

Visibility: Public
Change Control: Stable
Synopsis: Compilation option
Details: This configure option enables the Kerberos and GSS code in the configuration of the PBS Pro compilation. The code is behind macro PBS_SECURITY == KRB5. It may be necessary to provide the correct path to Kerberos libraries. There is an option for this purpose, which is variable PATH_KRB5_CONFIG. PATH_KRB5_CONFIG should be set to config binary of either Heimdal or MIT Kerberos.
Examples:
./configure --prefix=/usr --with-krbauth PATH_KRB5_CONFIG=/usr/bin/krb5-config.heimdal
./configure --prefix=/usr --with-krbauth PATH_KRB5_CONFIG=/usr/bin/krb5-config.mit

...

Visibility: Public
Change Control: Stable
Synopsis: Server attribute. Settable via qmgr.
Details: This is an ACL list of realms allowed to access the server in some way. The attribute is a list. Managers are allowed to set this attribute. For being a manager/operator appropriate principal must be specified in managers and its realm must be in acl_krb_realms.
Example:
set server managers = vchlum@ADMIN.REALM
set server acl_krb_realms = *@REALM
set server acl_krb_realms += *@ADMIN.REALM

Interface: New server attribute 'acl_krb_submit_realms'

Visibility: Public
Change Control: Stable
Synopsis: Server attribute. Settable via qmgr.
Details: This is an ACL list of realms allowed to submit a job into the system. This attribute is a list. This list should be a subset of acl_krb_realm. The necessity for having both the acl_krb_submit_realms and acl_krb_realm is that you are able to access the PBS Pro server with the host credentials or with credentials of other services, which are not suitable for submitting jobs. Usually, the computation node running mom is not a good identity for submitting jobs, but it is a suitable identity for accessing the server (e.g. the pbs_python hooks will need to be specified here in the future). Managers are allowed to set this attribute.
Example:
set server acl_krb_submit_realms = *@REALM

Interface: New server attribute 'cred_renew_enable'

...

Visibility: Public
Change Control: Stable
Synopsis: Server attribute. Settable via qmgr.
Details: This attribute sets the renew-tool which provides Kerberos credentials data in base64 in a specific format. This attribute is a string. Managers are allowed to set this attribute.
- The tool is supposed to take one argument, which is the user's principal.
- The tool is supposed to print to stdout in this way: The first line is 'Type: <credentials type>', the second line is 'Valid until: <timestamp in epocha>', the third line is credential data in base64.
- The credentials data will be read by krb5_rd_cred() after decoding base64. The data type is 'krb5_data'. Please see the Kerberos documentation for more info.
- The tool is not part of this work.
Example of the tool:
- ~# ./krb525_renew vchlum@REALM
  Type: Kerberos
  Valid until: 1536960989
  doICPDCCAjigAwI<... just base64 here ...>GqAjAA
- set server cred_renew_tool = /usr/bin/krb525_renew

...

Versions Compared

Old Version 30

New Version Current

Key

Overview: