Planet OSG

Spinning: Condor Week 2012

Fri, 11 May 2012 20:17:29 +0000

Condor Week 2012 was last week. As in past years there was great representation from the research community. We learned how research of all sizes and shapes is benefiting from high throughput computing resources.

A number of enterprises also turned up to talk about their uses of Condor, even if they didn’t bring slides. DreamWorks Animation was present to tell us how our favorite movies are being created with Condor.

I got to talk about Condor’s developer community, and still want to hear from anyone who has tried or would like to contribute to the Condor project. That probably means you!

Condor Project News: Condor 7.8.0 released! (May 10, 2012)

Thu, 10 May 2012 05:00:00 +0000

The Condor team is pleased to announce the release of Condor 7.8.0. This is the first entry in a new stable series, contains all the features and bug fixes in the 7.7 development series. See the version History for a complete list of changes.

Inside OSG Ops.: CondorWeek 2012

noreply@blogger.com (K) — Mon, 07 May 2012 09:32:49 +0000

During the week of May 1-4, I attended CondorWeek 2012 in Madison, WI. The event was hosted in the Wisconsin Institutes for Discovery at the University of Wisconsin-Madison.

The first day consisted of tutorials explaining the basics of Condor usage, workflows, administration, and security. Having spent more time doing general support with OSG and working with the support workflow, I haven't spent as much time as I'd like with Condor, but the tutorials did show me some things that I would like to try. I think that next year, these tutorials would be even more useful for me after trying some new things with Condor in the coming months.

The final three days consisted of researchers, admins, and companies showing and discussing off the multitude of ways that they use Condor to enhance and extend their workflows. Of particular interest to me was the way that DreamWorks used Condor to send jobs to their rendering software to create their animated films. They showed the trailer for Madagascar 3, which is their first movie to be created using Condor from beginning to end in the rendering process.

It was interesting to hear other outfits explain how they have had to work around some things Condor currently can't do and see how the Condor team was curious to see if these things could be implemented into coming versions of Condor. The team was very interested to expand and enhance Condor, which is already a very capable product.

Overall it was a good experience that made me aware of all the things that can be done with Condor. I look forward to trying some of these things for my own, and hopefully returning next year with a much better understanding of Condor as a whole so that I can gain even more information from the talks and tutorials.

Inside OSG Ops.: RabbitMQ & CometD

noreply@blogger.com (Soichi Hayashi) — Mon, 07 May 2012 07:53:54 +0000

We've been experimenting with following new services in the last few month here at GOC.

event.grid.iu.edu (RabbitMQ/AMQP Server)
comet.grid.iu.edu (CometD Server)

RabbitMQ (event.grid.iu.edu) allows GOC, and OSG users to publish and/or subscribe to various messages generated by our services, and OSG in general. Currently, we receive following messages.

RSV status changes
OIM updates
GOC Ticket updates
GIP information changes (prototype)

Anyone can subscribe to these messages in XML format, and be notified in real time using AMQP messaging client. APIs are available in many languages including, Java, PHP, Python, etc..

CometD (comet.grid.iu.edu) allows us to push messages to our various web applications. For example, GOC Ticket uses it to display users who are currently viewing a ticket. If someone updates a ticket while someone else is viewing, it will send page refresh request to all viewers. CometD can also be used to implement features such as chat, shared editing, and other functionalists.

CometD itself is a Java application framework where we can implement various services that client (web browsers) can make requests to. CometD acts as a glue between RabbitMQ and the web browsers. For example, "GOC event service" in comet.grid.iu.edu subscribes to RSV, OIM, and GOC tickets events, and pools all recent events. A web browsers can then make a request to download these events during the initial loading of a page, and it will subscribe to "new event" queue on comet in order to receive new events in real time until user closes the page.

By using Event & Comet services, we can implement interesting features such as Realtime GOC event (prototype) in MyOSG. My current goal is to continue experimenting with RabbitMQ/CometD and see what I can (and can not) accomplishing using these tools.

If anyone has an idea about what we can do with these tools, please feel free to send me a message.

Condor Project News: Condor 7.6.7 released (May 2, 2012)

Wed, 02 May 2012 05:00:00 +0000

The Condor team is pleased to announce the newest release in our development series, 7.6.7. This release fixes several important bugs, and we believe this will be the last release in the 7.6 series. Please see the release notes for a complete list. Condor binaries and source code are available from our Downloads page.

Condor Project News: 50,000-Core Condor cluster provisioned by Cycle Computing (May 2, 2012)

Wed, 02 May 2012 05:00:00 +0000

for the Schroedinger Drug Discovery Applications Group. See the article in Hostingtecnews.com, or the article in BioIT World. And, Condor helped enable this research.

Condor Project News: Condor 7.7.6 released (April 24, 2012)

Tue, 24 Apr 2012 05:00:00 +0000

The Condor team is pleased to announce the newest release in our development series, 7.7.6. This release represents the release candidate for Condor version 7.8, and it is the last release in the 7.7 development series. Please see the release notes for a complete list. Condor binaries and source code are available from our Downloads page.

Derek's Blog: Developments at Nebraska

noreply@blogger.com (Derek Weitzel) — Fri, 20 Apr 2012 13:18:58 +0000

I thought I would do a quick post about recent developments at Nebraska.

Tusker & OSG

New Tusker Cluster

We recently received a new cluster, Tusker. It is our newest in a line of clusters that prioritize memory per core, and cores per node over clock speed. Therefore, the cluster is 104 nodes, 102 of which are 64 core, 256GB nodes.

The goal of this cluster is to enable higher throughput of local user jobs while enabling backfilling of grid jobs. The current breakdown of local and grid jobs can be found on the hcc monitoring page.

A common complaint among our local users is interference between processes on the nodes. To address this, we patched torque to add cgroups support for cpu isolation. Memory isolation should come into production in the next few weeks. This will affect grid jobs by locking down their usage to only a single core.

Nebraska's goal is to support all OSG VO's, and give them equal priority (albeit lower than local users). All OSG VO's are welcome to run on Tusker.

Nebraska's Contribution to OSG Opportunistic Usage

Opportunistic usage by Site (source)

Nebraska resources have become the largest contributor to opportunistic resources. Easily over 1/4 of opportunistic usage is happening at Nebraska. We are #1 (Tusker), #2 (prairiefire), and after adding Firefly's different CE's, #7. We are very proud of this contribution, and hope it continues.

Stay tuned, the next year should be exciting...

Derek's Blog: BOSCO + Campus Factory

noreply@blogger.com (Derek Weitzel) — Mon, 16 Apr 2012 16:36:34 +0000

Checklist while implementing CF + Bosco integration

For the last several months, the campus infrastructure team has worked on software that will help users create a larger, more inclusive campus grid. The goal largely has been to make the software easier to install and expand.

Integrating the Campus Factory (which is already used on many campuses) with Bosco has been a key goal for this effort. Last week, I finally integrated the two (Beta Install Doc). This will have many benefits for the user over both the current Campus Factory and current Bosco.

Feature	Campus Factory	Bosco v0	Campus Factory + Bosco
Installation	Large installation/configuration instructions. Install Condor and campus factory on every cluster.	Install on a central submit node. Configuration is handled automatically.
Adding Resources	Install Condor and the campus factory on every cluster. Configure to link to other submit and campus factory clusters.	Run the command bosco_cluster -add Installation and configuration handled auto-magically.
File Transfer	Using Condor file transfer, can transfer input and output.	Manually with scp by the user. No stderr or stdout from job.	Using Condor file transfer, can transfer input and output.
Job State	Accurate user job state.	Delayed user job state. No exit codes from the user jobs.	Accurate user job state.

As you can see from the above table, the combined Campus Factory + Bosco takes the best from both technologies.

Spinning: Service as a Job: HDFS NameNode

Mon, 16 Apr 2012 10:44:13 +0000

Scheduling an HDFS DataNode is a powerful function. However, an operational HDFS instance also requires a NameNode. Here is an example of how a NameNode can be scheduled, followed by scheduled DataNodes, to create an HDFS instance.

From here, HDFS instances can be dynamically created on shared resources. Workflows can be built to manage, grow and shrink HDFS instances. Multiple HDFS instances can be deployed on a single set of resources.

The control script is based on hdfs_datanode.sh. It discovers the NameNode’s endpoints and chirps them.

hdfs_namenode.sh

#!/bin/sh -x

# condor_chirp in /usr/libexec/condor
export PATH=$PATH:/usr/libexec/condor

HADOOP_TARBALL=$1

# Note: bin/hadoop uses JAVA_HOME to find the runtime and tools.jar,
#       except tools.jar does not seem necessary therefore /usr works
#       (there's no /usr/lib/tools.jar, but there is /usr/bin/java)
export JAVA_HOME=/usr

# When we get SIGTERM, which Condor will send when
# we are kicked, kill off the namenode
function term {
   ./bin/hadoop-daemon.sh stop namenode
}

# Unpack
tar xzfv $HADOOP_TARBALL

# Move into tarball, inefficiently
cd $(tar tzf $HADOOP_TARBALL | head -n1)

# Configure,
#  . fs.default.name,dfs.http.address must be set to port 0 (ephemeral)
#  . dfs.name.dir must be in _CONDOR_SCRATCH_DIR for cleanup
# FIX: Figure out why a hostname is required, instead of 0.0.0.0:0 for
#      fs.default.name
cat > conf/hdfs-site.xml <<EOF
<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
  <property>
    <name>fs.default.name</name>
    <value>hdfs://$HOSTNAME:0</value>
  </property>
  <property>
    <name>dfs.name.dir</name>
    <value>$_CONDOR_SCRATCH_DIR/name</value>
  </property>
  <property>
    <name>dfs.http.address</name>
    <value>0.0.0.0:0</value>
  </property>
</configuration>
EOF

# Try to shutdown cleanly
trap term SIGTERM

export HADOOP_CONF_DIR=$PWD/conf
export HADOOP_LOG_DIR=$_CONDOR_SCRATCH_DIR/logs
export HADOOP_PID_DIR=$PWD

./bin/hadoop namenode -format
./bin/hadoop-daemon.sh start namenode

# Wait for pid file
PID_FILE=$(echo hadoop-*-namenode.pid)
while [ ! -s $PID_FILE ]; do sleep 1; done
PID=$(cat $PID_FILE)

# Wait for the log
LOG_FILE=$(echo $HADOOP_LOG_DIR/hadoop-*-namenode-*.log)
while [ ! -s $LOG_FILE ]; do sleep 1; done

# It would be nice if there were a way to get these without grepping logs
while [ ! $(grep "IPC Server listener on" $LOG_FILE) ]; do sleep 1; done
IPC_PORT=$(grep "IPC Server listener on" $LOG_FILE | sed 's/.* on \(.*\):.*/\1/')
while [ ! $(grep "Jetty bound to port" $LOG_FILE) ]; do sleep 1; done
HTTP_PORT=$(grep "Jetty bound to port" $LOG_FILE | sed 's/.* to port \(.*\)$/\1/')

# Record the port number where everyone can see it
condor_chirp set_job_attr NameNodeIPCAddress \"$HOSTNAME:$IPC_PORT\"
condor_chirp set_job_attr NameNodeHTTPAddress \"$HOSTNAME:$HTTP_PORT\"

# While namenode is running, collect and report back stats
while kill -0 $PID; do
   # Collect stats and chirp them back into the job ad
   # Nothing to do.
   sleep 30
done

The job description file is standard at this point.

hdfs_namenode.job

cmd = hdfs_namenode.sh
args = hadoop-1.0.1-bin.tar.gz

transfer_input_files = hadoop-1.0.1-bin.tar.gz

#output = namenode.$(cluster).out
#error = namenode.$(cluster).err

log = namenode.$(cluster).log

kill_sig = SIGTERM

# Want chirp functionality
+WantIOProxy = TRUE

should_transfer_files = yes
when_to_transfer_output = on_exit

requirements = HasJava =?= TRUE

queue

In operation -

Submit the namenode and find its endpoints,

$ condor_submit hdfs_namenode.job
Submitting job(s).
1 job(s) submitted to cluster 208.

$ condor_q -long 208| grep NameNode       
NameNodeHTTPAddress = "eeyore.local:60182"
NameNodeIPCAddress = "eeyore.local:38174"

Open a browser window to http://eeyore.local:60182 to find the cluster summary,

1 files and directories, 0 blocks = 1 total. Heap Size is 44.81 MB / 888.94 MB (5%)
  Configured Capacity                   :        0 KB
  DFS Used                              :        0 KB
  Non DFS Used                          :        0 KB
  DFS Remaining                         :        0 KB
  DFS Used%                             :       100 %
  DFS Remaining%                        :         0 %
 Live Nodes                             :           0
 Dead Nodes                             :           0
 Decommissioning Nodes                  :           0
  Number of Under-Replicated Blocks     :           0

Add a datanode,

$ condor_submit -a NameNodeAddress=hdfs://eeyore.local:38174 hdfs_datanode.job 
Submitting job(s).
1 job(s) submitted to cluster 209.

Refresh the cluster summary,

1 files and directories, 0 blocks = 1 total. Heap Size is 44.81 MB / 888.94 MB (5%)
  Configured Capacity                   :     9.84 GB
  DFS Used                              :       28 KB
  Non DFS Used                          :     9.54 GB
  DFS Remaining                         :   309.79 MB
  DFS Used%                             :         0 %
  DFS Remaining%                        :      3.07 %
 Live Nodes                             :           1
 Dead Nodes                             :           0
 Decommissioning Nodes                  :           0
  Number of Under-Replicated Blocks     :           0

And another,

$ condor_submit -a NameNodeAddress=hdfs://eeyore.local:38174 hdfs_datanode.job
Submitting job(s).
1 job(s) submitted to cluster 210.

Refresh,

1 files and directories, 0 blocks = 1 total. Heap Size is 44.81 MB / 888.94 MB (5%)
  Configured Capacity                   :    19.69 GB
  DFS Used                              :       56 KB
  Non DFS Used                          :    19.26 GB
  DFS Remaining                         :   435.51 MB
  DFS Used%                             :         0 %
  DFS Remaining%                        :      2.16 %
 Live Nodes                             :           2
 Dead Nodes                             :           0
 Decommissioning Nodes                  :           0
  Number of Under-Replicated Blocks     :           0

All the building blocks necessary to run HDFS on scheduled resources.

Spinning: Service as a Job: HDFS DataNode

Wed, 04 Apr 2012 09:15:10 +0000

Building on other examples of services run as jobs, such as Tomcat, Qpidd and memcached, here is an example for the Hadoop Distributed File System‘s DataNode.

Below is the control script for the datanode. It mirrors the memcached’s script, but does not publish statistics. However, datanode statistic/metrics could be pulled and published.

hdfs_datanode.sh

#!/bin/sh -x

# condor_chirp in /usr/libexec/condor
export PATH=$PATH:/usr/libexec/condor

HADOOP_TARBALL=$1
NAMENODE_ENDPOINT=$2

# Note: bin/hadoop uses JAVA_HOME to find the runtime and tools.jar,
#       except tools.jar does not seem necessary therefore /usr works
#       (there's no /usr/lib/tools.jar, but there is /usr/bin/java)
export JAVA_HOME=/usr

# When we get SIGTERM, which Condor will send when
# we are kicked, kill off the datanode and gather logs
function term {
   ./bin/hadoop-daemon.sh stop datanode
# Useful if we can transfer data back
#   tar czf logs.tgz logs
#   cp logs.tgz $_CONDOR_SCRATCH_DIR
}

# Unpack
tar xzfv $HADOOP_TARBALL

# Move into tarball, inefficiently
cd $(tar tzf $HADOOP_TARBALL | head -n1)

# Configure,
#  . dfs.data.dir must be in _CONDOR_SCRATCH_DIR for cleanup
#  . address,http.address,ipc.address must be set to port 0 (ephemeral)
cat > conf/hdfs-site.xml <<EOF
<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
  <property>
    <name>fs.default.name</name>
    <value>$NAMENODE_ENDPOINT</value>
  </property>
  <property>
    <name>dfs.data.dir</name>
    <value>$_CONDOR_SCRATCH_DIR/data</value>
  </property>
  <property>
    <name>dfs.datanode.address</name>
    <value>0.0.0.0:0</value>
  </property>
  <property>
    <name>dfs.datanode.http.address</name>
    <value>0.0.0.0:0</value>
  </property>
  <property>
    <name>dfs.datanode.ipc.address</name>
    <value>0.0.0.0:0</value>
  </property>
</configuration>
EOF

# Try to shutdown cleanly
trap term SIGTERM

export HADOOP_CONF_DIR=$PWD/conf
export HADOOP_PID_DIR=$PWD
export HADOOP_LOG_DIR=$_CONDOR_SCRATCH_DIR/logs
./bin/hadoop-daemon.sh start datanode

# Wait for pid file
PID_FILE=$(echo hadoop-*-datanode.pid)
while [ ! -s $PID_FILE ]; do sleep 1; done
PID=$(cat $PID_FILE)

# Report back some static data about the datanode
# e.g. condor_chirp set_job_attr SomeAttr SomeData
# None at the moment.

# While the datanode is running, collect and report back stats
while kill -0 $PID; do
   # Collect stats and chirp them back into the job ad
   # None at the moment.
   sleep 30
done

The job description below uses a templating technique. The description uses a variable NameNodeAddress, which is not defined in the description. Instead, the value is provided as an argument to condor_submit. In fact, a complete job can be defined without a description file, e.g. echo queue | condor_submit -a executable=/bin/sleep -a args=1d, but more on that some other time.

hdfs_datanode.job

# Submit w/ condor_submit -a NameNodeAddress=<address>
# e.g. <address> = hdfs://$HOSTNAME:2007

cmd = hdfs_datanode.sh
args = hadoop-1.0.1-bin.tar.gz $(NameNodeAddress)

transfer_input_files = hadoop-1.0.1-bin.tar.gz

# RFE: Ability to get output files even when job is removed
#transfer_output_files = logs.tgz
#transfer_output_remaps = "logs.tgz logs.$(cluster).tgz"
output = datanode.$(cluster).out
error = datanode.$(cluster).err

log = datanode.$(cluster).log

kill_sig = SIGTERM

# Want chirp functionality
+WantIOProxy = TRUE

should_transfer_files = yes
when_to_transfer_output = on_exit

requirements = HasJava =?= TRUE

queue

hadoop-1.0.1-bin.tar.gz is available from http://archive.apache.org/dist/hadoop/core/hadoop-1.0.1/

Finally, here is a running example,

A namenode is already running -

$ ./bin/hadoop dfsadmin -conf conf/hdfs-site.xml -report
Configured Capacity: 0 (0 KB)
Present Capacity: 0 (0 KB)
DFS Remaining: 0 (0 KB)
DFS Used: 0 (0 KB)
DFS Used%: 0%
Under replicated blocks: 0
Blocks with corrupt replicas: 0
Missing blocks: 0
-------------------------------------------------
Datanodes available: 0 (0 total, 0 dead)

Submit a datanode, knowing the namenode’s IPC port is 9000 -

$ condor_submit -a NameNodeAddress=hdfs://$HOSTNAME:9000 hdfs_datanode.job
Submitting job(s).
1 job(s) submitted to cluster 169.

$ condor_q
-- Submitter: eeyore.local : <127.0.0.1:59889> : eeyore.local
 ID      OWNER            SUBMITTED     RUN_TIME ST PRI SIZE CMD 
 169.0   matt            3/26 15:17   0+00:00:16 R  0   0.0 hdfs_datanode.sh h
1 jobs; 0 idle, 1 running, 0 held

Storage is now available, though not very much as my disk is almost full -

$ ./bin/hadoop dfsadmin -conf conf/hdfs-site.xml -report
Configured Capacity: 63810015232 (59.43 GB)
Present Capacity: 4907495424 (4.57 GB)
DFS Remaining: 4907466752 (4.57 GB)
DFS Used: 28672 (28 KB)
DFS Used%: 0%
Under replicated blocks: 0
Blocks with corrupt replicas: 0
Missing blocks: 0
-------------------------------------------------
Datanodes available: 1 (1 total, 0 dead)

Submit 9 more datanodes -

$ condor_submit -a NameNodeAddress=hdfs://$HOSTNAME:9000 hdfs_datanode.job
Submitting job(s).
1 job(s) submitted to cluster 170.
...
1 job(s) submitted to cluster 178.

$ ./bin/hadoop dfsadmin -conf conf/hdfs-site.xml -report
Configured Capacity: 638100152320 (594.28 GB)
Present Capacity: 40399958016 (37.63 GB)
DFS Remaining: 40399671296 (37.63 GB)
DFS Used: 286720 (280 KB)
DFS Used%: 0%
Under replicated blocks: 0
Blocks with corrupt replicas: 0
Missing blocks: 0
-------------------------------------------------
Datanodes available: 10 (10 total, 0 dead)

At this point you can run a workload against the storage, visit the namenode at http://localhost:50070, or simply use ./bin/hadoop fs to interact with the filesystem.

Remember, all the datanodes were dispatched by a scheduler, run along side existing workload on your pool, and are completely manageable by standard policies.

Derek's Blog: OSG AHM 2012

noreply@blogger.com (Derek Weitzel) — Tue, 27 Mar 2012 17:36:34 +0000

This years all hands meeting was a great success! There where a few sessions that I really enjoyed.

AHM Pictures

Campus Caucus
There where many user engagement people there. I believe that we reached a consensus that there isn't much an engagement community.

For example, Wisconsin has a great method for distributing and running MATLAB and R application on the OSG, but there has been no knowledge transfer to other engagement folks. I know a few UNL users that have wanted to run MATLAB on our resources. If we could move a HCC MATLAB workflow to the Grid, I believe that would be a great success.

I completely agree that there is no Engagement 'community'. But I think that's true of most of the OSG. Though, there have been recently many improvements.

I think the centralized Jira has helped tremendously. It's very easy to see what other people have been working on and even the general direction of progress. Though this only works for OSG 'employees' and OSG projects.
The OSG blogs have been successful for the technology group to explain what they are working on. Though, I wish they had shorter and more often posts.

I hope that the blogs can be a way to spread the OSG Engagement activity. It's also a great way to point to code and work that is being done. Also, blog posts shouldn't be limited to things that the author is doing, but could point to what other people are doing. For example, I knew nothing about the Rosetta people at Wisconsin until my poster was setup next to theirs and was able to have a conversation. It would have been great to see some information that of what they where doing outside of the once a year meeting.

Science on the OSG

Talk

I thought this talk by Frank was great. I felt he had the same feeling that we where all feeling, that Protein processing was becoming a very large user of the OSG. We've seen this at HCC with both CPASS, CS-Rosetta, and Autodock.

Walltime usage for non-HEP

Frank also pointed out a graph of usage. At the end of the graph, there seems to be a plateau. Possibly we are hitting opportunistic resource limits?

Here's an updated usage graph (source):

Walltime usage for non-HEP updated

The thing to note is the explosive growth of GLOW VO. Their usage has increased dramatically recently.

OSG in 2017

Talk

I really liked to see what people thought the OSG would look like in 2017.

Chander's prediction that people will come to us to use the OSG. I believe this will take a critical mass of users. I think we have a good product to sell, we just need publicity.

Chander's comment on data is also important. But I believe the problem with data isn't necessarily storage, but it's access to the data. Take for example Dropbox. For free, they offer very little storage. The main advantage is that it's accessible from anywhere, laptop, desktop, iphone, web... I think a uniform data access method can get us a lot further than distributed storage.

Alain's prediction that we will be using more community software. This will take a large effort to be part of the distribution's community. I foresee us contributing packages, patches, and effort to Fedora EPEL and possibly Ubuntu. I think we are making great strides with the packaging, and would like to continue injecting us into the Fedora community.

Nebraska Campus Infrastructure

Talk

Of course, my talk is worth looking at.

This post ended up larger than I was hoping for. Oh well.

Condor Project News: Two Condor papers selected as most influential in HPDC (March 25, 2012)

Sun, 25 Mar 2012 05:00:00 +0000

Two papers by the Condor Team were selected as to represent the most influential papers in the history of The International ACM Symposium on High-Performance Parallel and Distributed Computing (HPDC). Details...

Spinning: Daylight savings, costs CPU

Tue, 20 Mar 2012 12:56:24 +0000

Erik ran into a surprising performance problem over the weekend.

If you live in an orange area of the map, mktime(3) may be killing your performance.

tm_isdst is important. Don’t guess!

Don’t construct struct tm manually. Or, even better, let mktime decide DST for you. Set tm_isdst = -1.

Are you impacted? Run your programs under callgrind, vary between Standard and Daylight, and look at mktime’s profile. Using Erik’s jig,

$ TZ=EDT valgrind --tool=callgrind ./test_mktime
...
$ TZ=EST valgrind --tool=callgrind ./test_mktime
...
$ callgrind_annotate --inclusive=yes --tree=calling ...
...

$ grep "mktime " EST.txt EDT.txt 
EST.txt:Profiled target:  ./test_mktime (PID 29595, part 1)
EST.txt:    1,765  &gt;   ???:mktime (1x) [/lib64/libc-2.5.so]
EST.txt:  546,244  *  ???:mktime [/lib64/libc-2.5.so]
EST.txt:  542,350  &gt;   ???:mktime (1x) [/lib64/libc-2.5.so]

EDT.txt:Profiled target:  ./test_mktime (PID 18878, part 1)
EDT.txt:    4,819  &gt;   ???:mktime (1x) [/lib64/libc-2.5.so]
EDT.txt:   23,119  *  ???:mktime [/lib64/libc-2.5.so]

It is currently EDT.

Derek's Blog: Burning the LiveUSBs

noreply@blogger.com (Derek Weitzel) — Fri, 16 Mar 2012 16:37:02 +0000

In my last post, I talked about the OSG LiveUSBs. Now that the conference is next week, I have started burning the USB with the image.

USB Key Piles

I burned 4 USBs at a time, using the script below. Parted didn't work, never really found out why. The symptoms where that the USB would not boot, but they where readable by Macs. So I scripted fdisk.

Condor Project News: Congratulations to Victor Ruotti (March 15, 2012)

Thu, 15 Mar 2012 05:00:00 +0000

computational biologist at the Morgridge Institute for Research, winner of the CycleCloud BigScience Challenge 2011. Read more about both the challenge and the finalists in the Cycle Computing press release.

OSG Technology Area Rumblings: Resource Isolation in Condor using cgroups

noreply@blogger.com (Brian Bockelman) — Sat, 10 Mar 2012 11:28:31 +0000

This is the last in my series on job isolation techniques. It has spanned in postings over the last month, so it may help to recap:

Part I covered process isolation, prevent processes in one job from interacting with other jobs. This has been achievable through POSIX mechanisms for awhile, but the new PID namespaces mechanisms provide improved isolation for jobs running as the same user.
Part II and Part III discussed file isolation using bind mounts and chroots. Condor uses bind mounts to remove access to "problematic" directories such as /tmp. While more complex to setup, chroots allow jobs to run in a completely separate environment as the host and further isolates the job sandbox.

This post will cover resource isolation: preventing jobs from consuming system resources promised to another job.

Condor has always had some crude form of resource isolation. For example, the worker node could be configured to detect when the processes in a job have more CPU time than walltime (a rough indication that more than one core is being used) or when the sum of each process's virtual memory size exceeds the memory requested for the job. When Condor detects too many resources are being consumed, it can take an action such as suspending or killing the job.

This traditional approach is relatively unsatisfactory for a few reasons:

Condor periodically polls to view resource consumption. Any activity between polls is unmonitored.
The metrics Condor traditionally monitors are limited to memory and CPU, where the memory metrics are poor quality for complex jobs. The sum many process's virtual memory size, on a modern Linux box, has little correlation with RAM used and is not particularly meaningful.
We can do little with the system besides detect when resource limits have been violated and kill the job.

We cannot, for example, simply instruct the kernel to reduce the job's memory or CPU usage.
Accordingly, users must ask for peak resource usage, which may be well-above average resource usage, decreasing overall throughput. If the job needs 2GB on average but 4GB for a single second, the user will ask for 4GB; the other 2GB will be un-utilized.

In Linux, the oldest form of resource isolation is processor affinity or CPU pinning: a job can be locked to a specific CPU, and all its processes will inherit the affinity. Because two jobs are locked to separate CPUs, they will never consume each others' CPU resources. CPU pinning is unsatisfactory for reasons similar to memory: jobs can't utilize otherwise-idle CPUs, decreasing potential system throughput. The granularity is also poor: you can't evenly fairshare 25 jobs on a machine with 24 cores as each job must be locked to at least one core. However, it's a step forward - you don't need to kill jobs for using too much CPU - and present in Condor since 7.3.

Newer Linux kernels support cgroups, which allow are structures for managing groups of processes, and provide controllers for managing resources in each cgroup. In Condor 7.7.0, cgroup support was added for measuring resource usage. When enabled, Condor will place each job into a dedicated cgroup for the block-I/O, memory, CPU, and "freezer" controllers. We have implemented two new limiting mechanisms based on the memory and CPU controllers.

The CPU controller provides a mechanism for fairsharing between different cgroups. CPU shares are assigned to jobs based on the "slot weight" (by default, equal to the number of cores the job requested). Thus, a job asking for 2 cores will get an average of 2 cores on a fully loaded system. If there's an idle CPU, it could utilize more than 2 cores; however, it will never get less than what it requested for a significant amount of time. CPU fairsharing provides a much finer granularity than pinning, easily allowing the jobs-to-cores ratio be non-integer.

The memory controller provides two kinds of limits: soft and hard. When soft limits are in place, the job can use an arbitrary amount of RAM until the host runs out of memory (and starts to swap); when this happens, only jobs over their limit are swapped out. With hard limits, the job immediately starts swapping once it hits its RAM limit, regardless of the amount of free memory. Both soft and hard limits default to the amount of memory requested for the job.

Both methods also have disadvantages. Soft limits can cause "well-behaved" processes to wait while the OS frees up RAM from "badly behaving" process. Hard limits can cause large amounts of swapping (for example, if there's a memory leak), decreasing the entire node's disk performance and thus adversely affecting other jobs. In fact, it may be a better use of resources to preempt a heavily-swapping process and reschedule it on another node than let it continue running. There is further room for improvement here in the future.

Regardless, cgroups and controllers provide a solid improvement in resource isolation for Condor, and finish up our series on job isolation. Thanks for reading!

Spinning: Best practices: batch applications input

Thu, 08 Mar 2012 05:16:16 +0000

For application developers and their users.

An application that is going to be run in batch mode — non-interactive, maybe scheduled to remote resources — is going to need input[0]. That input might be a few numbers or numerous data files. As an application developer, the case of data files can be difficult to get right. There are many ways to handle data files and one big pitfall: hard-coded paths.

Below are a few options for application developers and the resulting work for a Condor user of the application.

The best approach is to read input from stdin or have paths to data files passed via arguments. Doing so shows the developer has batch processing in mind, and provides the applications user with clear options.

A submit file for an application that reads from stdin -

executable = batch_app
input = input.dat
queue

A submit file for an application that takes data files as arguments -

executable = batch_app
arguments = --input=input.dat
transfer_input_files = input.dat
queue

A middle ground approach may be necessary if the set of input files is large or their relationships are complex. In such a case, a meta-data file can be used, or the input files can be laid out in a well-defined pattern in the filesystem. Note: “well-defined pattern in the filesystem” is often a myth.

Of these approaches, the meta-data file is preferred. It makes the input files and their relationships explicit. However, it can be more difficult for the application’s user from a Condor perspective. When the files are laid out in the filesystem the tendency is for the application to not have a well-defined layout, or a definition maintained independently of the application.

A submit file for an application that takes a meta-data file -

executable = batch_app
arguments = --input=input.meta
transfer_input_files = input.meta[,all the files listed in input.meta]
queue

The difficulty comes in listing all the files from the input.meta. This is often mitigated by providing URIs, or paths, in input.meta that may point into a shared filesystem. The files in a shared filesystem need not be transferred by Condor and need not be listed in transfer_input_files.

A submit file for an application that takes a hopefully-well-defined filesystem layout,

executable = batch_app
arguments = --input=data_dir
transfer_input_files = data_dir
queue

This is simpler because Condor will transfer everything under data_dir into the job’s scratch space and keep it under a directory called data_dir. Often, the data_dir will even exist on a shared filesystem and will not need to be transferred (remove transfer_input_files = data_dir and provide full path with --input).

Note: transfer_input_files = data_dir/ will not replicate the directory tree in the job’s scratch space. It will be collapsed.

These two approaches can be combined to get the best of both.

The worst approach is really a non-approach and involves hard-coding paths into the application. Arguably the application does not have a batch mode. It will fail when not run in its expected environment, which may simply mean by a user different from the developer, or on a new shared filesystem, or an old shared filesystem with new mounts. These application should be avoided or modified to provide a batch mode.

Developers beware, you can turn near success with a meta-data file into a failure by hard-coding its path.

Takeaway -

For developers, an application that has a batch processing will parametrize all its inputs[1].

For users, beware of applications that operate on data that you have not provided.

[0] Even if it is just a random seed.
[1] Database or URI connections to get data also matter.

Condor Project News: Condor in EC2 now supported by Red Hat (March 2, 2012)

Fri, 02 Mar 2012 05:00:00 +0000

Red Hat Enterprise MRG support expands into the cloud as announced in this press release, by leveraging the EC2 job universe in Condor, and by maintaining supported images of Red Hat Enterprise Linux with Condor pre-installed on Amazon storage services. Red Hat MRG can schedule local grids, remote grids, virtual machines for internal clouds, and now, rented cloud infrastructure. There is also documentation of this feature.

Condor Project News: Condor 7.7.5 released (February 28, 2012)

Tue, 28 Feb 2012 05:00:00 +0000

The Condor team is pleased to announce the newest release in our development series, 7.7.5. This release represents the feature freeze for Condor version 7.8, and we expect it to be the penultimate release in the 7.7 development series. New features in this release include better statistics for monitoring a Condor pool, better support for absent ads in the collector, fast claiming of partitionable slots, and support for some newer Linux kernel features to better support process isolation. Please see the release notes for a complete list. Condor binaries and source code are available from our Downloads page.

OSG Technology Area Rumblings: Improving File Isolation with chroot

noreply@blogger.com (Brian Bockelman) — Mon, 27 Feb 2012 11:33:56 +0000

In the last post, we examined a new Condor feature called MOUNT_UNDER_SCRATCH that will isolate jobs from each other on the file system by making world-writable directories (such as /tmp and /var/tmp) be unique and isolated per-batch-job.

That work started with the assumption that jobs from the same Unix user don't need to be isolated from each other. This isn't necessarily true on the grid: a single, shared account per-VO is still popular on the OSG. For such VOs, an attacker can gain additional credentials by reading the sandbox of each job running under the same Unix username.

To combat proxy-stealing, we use an old Linux trick called a "chroot". A sysadmin can create a complete copy of the OS inside a directory, and an appropriately-privileged process can change the root of its filesystem ("/") to that directory. In fact, the phrase "changing root" where we get the "chroot" terminology.

For example, suppose the root of the system looks like this:

[root@localhost ~]# ls /
bin     cvmfs         hadoop-data2  home        media  opt   selinux  usr
boot    dev           hadoop-data3  lib         misc   proc  srv      var
cgroup  etc           hadoop-data4  lib64       mnt    root  sys
chroot  hadoop-data1  hadoop.log    lost+found  net    sbin  tmp

The sysadmin can create a copy of the RHEL5 operating system inside a sub-directory; at our site, this is /chroot/sl5-v3/root:

[root@localhost ~]# ls /chroot/sl5-v3/root/
bin   cvmfs  etc   lib    media  opt   root  selinux  sys  usr
boot  dev    home  lib64  mnt    proc  sbin  srv      tmp  var

Note how the contents of the chroot directory are stripped down relative to the OS - we can remove dangerous binaries, sensitive configurations, or anything else unnecessary to running a job. For example, many common Linux privilege escalation exploits come from the presence of a setuid binary. Such binaries (at, cron, ping) are necessary for managing the host, but not necessary for a running job. By eliminating the setuid binaries from the chroot, a sysadmin can eliminate a common attack vector for processes running inside.

Once the directory is built, we can call chroot and isolate ourselves from the host:

[root@red-d15n6 ~]# chroot /chroot/sl5-v3/root/
bash-3.2# ls /
bin   cvmfs  etc   lib   media  opt   root  selinux  sys  usr
boot  dev    home  lib64  mnt  proc  sbin  srv      tmp  var

Condor, as of 7.7.5, now knows how to invoke the chroot syscall for user jobs. However, as the job sandbox is written outside the chroot, we must somehow transport it inside before starting the job. Bind mounts - discussed last time - come to our rescue. The entire process goes something like this:

Condor, as root, forks off a new child process.
The child uses the unshare system call to place itself in a new filesystem namespace.
The child calls mount to bind-mount the job sandbox inside the chroot. Any other bind mounts - such as /tmp or /var/tmp - are done at this time.
The child will invoke the chroot system call specifying the directory the sysadmin has configured.
The child drops privileges to the target batch system user, then calls exec to start the user process.

With this patch applied, Condor will copy only the job's sandbox forward into the filesystem namespace, meaning the job has access to no other sandbox (as all other sandboxes live outside the private namespace). This successfully isolates jobs from each other's sandboxes, even if they run under the same Unix user!

The Condor feature is referred to as NAMED_CHROOT, as sysadmins can created multiple chroot-capable directories, give them a user-friendly name (such as RHEL5, as opposed to /chroot/sl5-v3/root), and allow user jobs to ask for the directory by the friendly name in their submit file.

In addition to the security benefits, we have found the NAMED_CHROOT feature allows us to run a RHEL5 job on a RHEL6 host without using virtualization; something for the future.

Going back to our original list of directories needing isolation - system temporary directories, job sandbox, shared filesystems, and GRAM directories - we have now isolated everything except the shared filesystems. The option here is simple, if unpleasant: mount the shared file system as read-only. This is the modus operandi for $OSG_APP at many sites, and an acceptable (but not recommended) way to run $OSG_DATA (as $OSG_DATA is optional anyway). It restricts the functionality for the user, but brings us a step closer to our goal of job isolation.

After file isolation, we have one thing left: resource isolation. Again, a topic for the future.

Derek's Blog: Building an OSG-Client LiveUSB

noreply@blogger.com (Derek Weitzel) — Fri, 24 Feb 2012 15:18:41 +0000

Nebraska/OSG USB keys to be distributed at OSG-AHM 2012

Since we have started using RPM's for osg software, I've been interested if it where possible to make a LiveUSB of the client. Due to the great documentation provided on the Scientific Linux LiveCD page, along with the CentOS LiveCD page, I've created a OSG Client LiveUSB that will be put on the keys.

Desktop of OSG Client Live CD

From the picture, notice links on the desktop to OSG User Docs, OSG LiveCD Docs, and How to get a certificate.

Live image creation
The live image creation was done using the livecd-tools package. I used a Fedora 16 instance on the HCC private cloud to make SL6 images. The kickstart file used can be found on github.

What's Installed
The goal of the LiveUSB is to give an easily deliverable demo of the OSG-Client, therefore only the OSG-Client and Condor are installed. The LiveUSB has some persistant data storage area, but not much.

Tools are installed in order to install the live image, including the OSG-Client components, to the local hard drive. Researchers can then easily have a node up and running.

Also, people that know how to run virtual machines on their computers can easily create a virtual machine with the OSG-Client from this USB. Just boot from the USB, and click on the Install to Hard Drive icon on the desktop.

These keys will be distributed to attendees of the OSG All Hands meeting.

I am open for suggestions on what should be on the LiveUSB. The image is not final yet.

UPDATE:
Link to Current ISO: OSG-SL6.2-x86_64-LiveUSB.iso

OSG Technology Area Rumblings: File Isolation using bind mounts and chroots

noreply@blogger.com (Brian Bockelman) — Mon, 20 Feb 2012 08:03:49 +0000

The last post ended with a new technique for process-level isolation that unlocks our ability to safely use anonymous accounts and group accounts.

However, that's not "safe enough" for us: the jobs can still interact with each other via the file system. This post examines the directories where jobs can write into, and what can be done to remove this access.

On a typical batch system node, a user can write into the following directories:

System temporary directories: The Linux Filesystem Hierarchy Standard (FHS) provides at least two sticky, world-writable directories, /tmp and /var/tmp. These directories are traditionally unmanaged (user processes can write an uncontrolled amount of data here) and a security issue (symlink attacks and information leaks), even when user separation is in place.
Job Sandbox: This is a directory created by the batch system as a scratch location for the job. The contents of the directory will be cleaned out by the batch system after the job ends. For Condor, any user proxy, executable, or job stage-in files will be copied here prior to the job starting.
Shared Filesystems: For a non-grid site, this is typically at least $HOME, and some other site-specific directory. $HOME is owned by the user running the job. On the OSG, we also have $OSG_APP for application installation (typically read-only for worker nodes) and, optionally, $OSG_DATA for data staging (writable for worker nodes). If they exist and are writable, $OSG_APP/DATA are owned by root and marked as sticky.
GRAM directories: For non-Condor OSG sites, a few user-writable directories are needed to transfer the executable, proxy, and job stage-in files from the gatekeeper to the worker node. These default to $HOME, but can be relocated to any shared filesystem directory. For Condor-based OSG sites, this is a part of the job sandbox.

If user separation is in place and considered sufficient, filesystem isolation is taken care of for shared filesystems, GRAM directories, and the job sandbox. The systemwide temporary directories can be protected by mixing filesystem namespaces and bind mounts.

A process can be launched in its own filesystem namespace; such a process will have a copy of the system mount table. Any change made to the process's mount table will not be seen by the outside system, and will be shared with any child processes.

For example, if the user's home directory is not mounted on the host, the batch system could create a process in a new filesystem namespace and mount the home directory in that namespace. The home directory will be available to the batch job, but to no other process on the filesystem.

When the last process in the filesystem namespace exits, all mounts that are unique to that namespace will be unmounted. In our example, when the batch job exits, the kernel will unmount the home directory.

A bind mount makes a file or directory visible at another place in the filesystem - I think of it as mirroring the directory elsewhere. We can take the job sandbox directory, create a sub-directory, and bind-mount the sub-directory over /tmp. The process is mostly equivalent to the following shell commands (where $_CONDOR_SCRATCH_DIR is the location of the Condor job sandbox) in a filesystem namespace:

mkdir $_CONDOR_SCRATCH_DIR/tmp
mount --bind $_CONDOR_SCRATCH_DIR/tmp /tmp

Afterward, any files a process creates in /tmp will actually be stored in $_CONDOR_SCRATCH_DIR/tmp - and cleaned up accordingly by Condor on job exit. Any system process not in the job will not be able to see or otherwise interfere with the contents of the job's /tmp unless it can write into $_CONDOR_SCRATCH_DIR.

Condor refers to this feature as MOUNT_UNDER_SCRATCH, and will be a part of the 7.7.5 release. This will be an admin-specified list of directories on the worker node. With it, the job will have a private copy of these directories, which will be backed by $_CONDOR_SCRATCH_DIR. The contents - and size - of these will be managed by Condor, just like anything else in the scratch directory.

If user separation is unavailable or not considered sufficient (if there are, for example, group accounts), an additional layer of isolation is needed to protect the job sandbox. A topic for a future day!

Derek's Blog: Ceph on Fedora 16

noreply@blogger.com (Derek Weitzel) — Fri, 17 Feb 2012 20:53:16 +0000

I've written before how to run ceph on Fedora 15, but now I'm working on Fedora 16.

Last time I complained about how much ceph tries to do for you. For better or worse, now it attempts to do more for you!

For my setup, I had 3 nodes in the HCC private cloud. First, we need to install ceph.

$ yum install ceph

Then, create a configuration file for ceph. The RPM comes with a good example that my configuration is based on. The example script is in /usr/share/doc/ceph/sample.ceph.conf

My configuration: Derek's Configuration

The configuration has the authentication turned off. I found this useful because the ceph-authtool (yes, the renamed it since Fedora 15) is difficult to use. And because all of the nodes are on a private vlan only reachable by my openvpn key :)

Then, you need to create and distribute ssh keys to all of your nodes so that the mkcephfs can ssh to them and configure.

$ ssh-keygen

Then copy them to the nodes:

$ ssh-copy-id i-000000c2
$ ssh-copy-id i-000000c3

Be sure to make the data directories on all the nodes. In this case:

$ mkdir -p /data/osd.0
$ ssh i-000000c2 'mkdir -p /data/osd.1'
$ ssh i-000000c3 'mkdir -p /data/osd.2'

Then run the mkcephfs command:

$ mkcephfs -a -c /etc/ceph/ceph.conf

And start up the daemons:

$ service ceph start

You should have the daemons running then. If they fail for some reason, they tend to output what the problem was. Also, the logs for the services are in /var/log/ceph

To mount the filesystem, find an ip address of one of the monitors. In my case, I had a monitor on ip address 10.148.2.147. The command to mount is:

$ mkdir -p /mnt/ceph
$ mount -t ceph 10.148.2.147:/ /mnt/ceph

Since you don't have any authentication, it should work without problems.

I've had some problems with the different mds, even had a OSD die on me. It resolved itself, and I even added another OSD to take it's place, recreating the CRUSH table. Since creating this, I have even worked with the graphical interface:

And here's a presentation I did about the CEPH Paper. Note, I may not be entirely accurate in the presentation, do be kind.

An Open Science Grid Work Log: SAGA Hadoop

Fri, 17 Feb 2012 20:43:50 +0000

Introduction

Hadoop has become a fixture where big data is concerned but it has been difficult to use in HPC and HTC cluster environments. This is becoming unfortunate as an increasing number of new algorithms assume Hadoop’s an option. I tried SAGA-Hadoop first but it should be noted that, myHadoop from U. of Indiana sought to remedy this a few years ago. If someone has experience with both or would like to comment on differences, that would be appreciated.

SAGA Hadoop

SAGA-Hadoop installs, configures and executes Hadoop on clusters running batch schedulers for which SAGA has adapters. At least it’s headed in that direction.

SAGA is the Simple API for Grid Applications and an Open Grid Forum standard (GFD.90) for interfacing with diverse cluster batch scheduling systems. It is a large and complex standard so we’ll leave it at that for now. For our purposes, suffice to say that it works with PBS.

The Bliss project provides Python bindings for SAGA. Like many Python APIs, it takes a minimalist approach, not covering the entire standard and demonstrating a strong preference for simplicity and brevity.

As the helpful introductory blog post above (SAGA-Hadoop) describes, it runs Hadoop.

But Can this be Interesting on the OSG?

To make this plausible for the OSG:

We have to be able to automate the process entirely
It would be really good if there were a practical way to use command line tools as the map and reduce steps in a MapReduce computation.

So this is the story of finding those things out.

Automating Installation

The RENCI Blueridge cluster runs the PBS job manager. I scripted the installation but nothing is pretty about it yet. Except that it demonstrates automation of setting Hadoop up on an HPC cluster.

There’s Python involved so my first step was to set up a virtualenv to manage our project’s dependencies:

wget --timestamping \
   https://raw.github.com/pypa/virtualenv/master/virtualenv.py
python virtualenv.py venv
source venv/bin/activate
pip install bliss uuid

uuid is a dependency that, for whatever reason, needed to be explicitly named to pip.

the uuid module uses ifconfig. Putting it in the path changed nothing. So we force the issue by editing the module file. Again, it’s in our virtualenv so the copy’s entirely ours:

sed --in-place=.orig                       \
     s,\'ifconfig\',\'/sbin/ifconfig\',     \
     venv/lib/python2.4/site-packages/uuid.py

Make sure JAVA_HOME is set.

Then, after unzipping Hadoop, we

Get the source code for SAGA-Hadoop
Edit it to specify the login node for our cluster
Edit the bootstrap script to alter the Hadoop configs after installation. It
- Deletes an unnecessary os.makedirs() call
- Corrects the makedirs for log_dir to be for self.job_log_dir
- Uncomments the configuration of the Hadoop data node
- Alters the network itnerface to use to the locally correct eth0
- Sets JAVA_HOME
- Sets HADOOP_HEAP_SIZE

svn co https://svn.cct.lsu.edu/repos/saga-projects/applications/SAGAHadoop/saga-hadoop
sed --in-place=.orig \
    -e "s,india,br0," saga-hadoop/launcher.py
HADOOP_ENV=hadoop-1.0.0/conf/hadoop-env.sh
sed --in-place=.orig \
    -e "s,os.makedirs(job_dir),," \
    -e "s,os.makedirs(log_dir),os.makedirs(self.job_log_dir)," \
    -e "s,\<!--,," \
    -e "s,\-\-\>,," \
    -e "s,eth1,eth0," \
    -e "s,tar \-xzf hadoop.tar.gz,tar -xzf hadoop.tar.gz; echo export JAVA_HOME=$JAVA_HOME >> $HADOOP_ENV; echo export HADOOP_HEAP     _SIZE=2000 >> $HADOOP_ENV; cat $HADOOP_ENV," \
    saga-hadoop/bootstrap_hadoop.py

This is all saved as a script called build.

Running it is relatively uninteresting. It installs the virtualenv, Python dependencies and edits the config files as you’d expect.

Running Hadoop

We save these commands to a script called start:

source venv/bin/activate
./saga-hadoop/bootstrap_hadoop.py

Running it logs plenty of Hadoop information to the console and starts the server.

Command Line MapReduce with Hadoop Streaming

The command line is the lingua franca of the OSG. There’s nothing finer for figuring out a problem quickly than the command line. So how are we going to run MapReduce jobs from the command line – especially in this dynamically created environment?

Hadoop Streaming lets us run programs of our choice as the map and reduce steps. But first we need an additional Java archive to make it work. The following commands go in a file called stream. This first batch gets the JAR file and copies it to the right location

jar_url=http://repo1.maven.org/maven2/org/apache/hadoop/hadoop-streaming/1.0.0/hadoop-streaming-1.0.0.jar
lib=hadoop-streaming-1.0.0.jar work/hadoop-1.0.0/share/hadoop/contrib/streaming
mkdir -p $lib
curl $jar_url  > $lib/hadoop-streaming-1.0.0.jar

Once that’s in place, we get rid of the old output directory (not likely to be relevant for an OSG job), then

inputs=$1
output=$2
mapper=$3
reducer=$4

rm -rf out

$HADOOP_HOME/bin/hadoop jar $HADOOP_HOME/share/hadoop/contrib/streaming/hadoop-streaming-1.0.0.jar \
     -input $inputs \
     -output $output \
     -mapper $mapper \
     -reducer $reducer \
     -jobconf mapred.reduce.tasks=2

We execute the map reduce with whatever’s passed in. Then, we create a trivial program called script to be our map operator:

for x in $(seq 0 100); do
    echo pattern $x
done

Next, we run stream like this:

./stream in out script wc

Which produces the sum of word count of our 101 output lines times three

 [scox@br0:~/dev/saga-hadoop]$ cat out/part-00000
     303     606    3609

because the stream is executed once per file in our input directory (in):

 in
 |-- a
 |-- b
 `-- c

Summary

We’ve just done a completely automated install, configuration and execution of a small Hadoop cluster overlay on top of a PBS HPC cluster. We also saw automation for installnig Hadoop streaming and running a test MapReduce job using command line programs as map and reduce operators.

The overall stack looks like this:

It is probably clear that much more would be necessary to properly configure and deploy a useful cluster, particularly interfacing it properly to a PBS job context. That part will have to wait for another day. And again, if anyone reading this can comment on myHadoop, I’d appreciate the insight.

Derek's Blog: CEPH on Fedora 15

noreply@blogger.com (Derek Weitzel) — Fri, 17 Feb 2012 20:30:31 +0000

Yesterday, I read a blog post using CEPH for a backend store for virtual machine images. I've heard a lot about ceph in the last year, especially after it was integrated into the mainline kernel in 2.6.34. So I thought I'd give it a try.

Before I get into the install, I want to summarize my thoughts on Ceph. I think it has a lot of potential, but parts of it are trying too hard to do everything for you. I always think there is a careful balance between a program doing too much for you, and making you do too much. For example, the mkcephfs script that creates a ceph filesystem will ssh to all the worker nodes (defined in ceph.conf) and configure the filesystem. If I was in operations, this would scare me.

Also, the keychain configuration is overly complicated. I think the Ceph is designed to be secure over the WAN (secure, not encrypted), so maybe it's needed. But it seems overly complicated when you compare it to other distributed file systems (Hadoop, Lustre).

On the other hand, I really like the full posix compliant client, especially since it's in the mainline kernel. It is too bad that it was added in 2.6.34 rather than 2.6.32 (RHEL 6 kernel). I guess we'll have to wait 2 years for RHEL 7 to have it in something we can use in production.

Also, the distributed metadata and multiple metadata servers are interesting aspects to the system. Though, in the version I tested, the MDS crashed a few times (the system picked it up and compensated).

On Fedora 15, ceph packages are in the repos.

yum install ceph

The configuration I settled on was:

[global]
    auth supported = cephx
    keyring = /etc/ceph/keyring.admin

[mds]
    keyring = /etc/ceph/keyring.$name
[mds.i-00000072]
    host = i-00000072
[mds.i-00000073]
    host = i-00000073
[mds.i-00000074]
    host = i-00000074

[osd]
    osd data = /srv/ceph/osd$id
    osd journal = /srv/ceph/osd$id/journal
    osd journal size = 512
    osd class dir = /usr/lib64/rados-classes
    keyring = /etc/ceph/keyring.$name
[osd0]
    host = i-00000072
[osd1]
    host = i-00000073
[osd2]
    host = i-00000074

[mon]
    mon data = /srv/ceph/mon$id
[mon0]
    host = i-00000072
    mon addr = 10.148.2.147:6789
[mon1]
    host = i-00000073
    mon addr = 10.148.2.148:6789
[mon2]
    host = i-00000074
    mon addr = 10.148.2.149:6789

As you can read from the configuration file, all files are stored in /srv/ceph/... You will need to make this directory on all your worker nodes.

Next I needed to create a keyring for authentication with the client/admin/dataservers. The keyring tool is distributed with Ceph, and is called cauthtool. Even now, it's not clear to me how to use this tool, or how Ceph uses the keyring. First you need to make a caps (capabilities?) file:

osd = "allow *"
mds = "allow *"
mon = "allow *"

Here are the cauthtool commands to get it to work.

cauthtool --create-keyring /etc/ceph/keyring.bin
cauthtool -c -n i-00000072 --gen-key /etc/ceph/keyring.bin 
cauthtool -n i-00000074 --caps caps /etc/ceph/keyring.bin
cauthtool -c -n i-00000073 --gen-key /etc/ceph/keyring.bin
cauthtool -n i-00000073 --caps caps /etc/ceph/keyring.bin
cauthtool -c -n i-00000074 --gen-key /etc/ceph/keyring.bin 
cauthtool -n i-00000072 --caps caps /etc/ceph/keyring.bin
cauthtool --gen-key --name=admin /etc/ceph/keyring.admin

From the blog post linked above, I used their script to create the directories and copy the ceph.conf to the other hosts.

n=0
for host in i-00000072 i-00000073 i-00000074 ; \
   do \
       ssh root@$host mkdir -p /etc/ceph /srv/ceph/mon$n; \
       n=$(expr $n + 1); \
       scp /etc/ceph/ceph.conf root@$host:/etc/ceph/ceph.conf
   done
mkcephfs -a -c /etc/ceph/ceph.conf -k /etc/ceph/keyring.bin

Then copy the keyrings

for host in i-00000072 i-00000073 i-00000074 ; \
   do \
       scp /etc/ceph/keyring.admin root@$host:/etc/ceph/keyring.admin; \
   done

Then startup the daemons on all the nodes:

service ceph start

And to mount the system:

mount -t ceph 10.148.2.147:/ /mnt/ceph -o name=admin,secret=AQBlV5dO2TICABAA0/FP7m+ru6TJLZaPxFuQyg==

Where the secret is the output from the command:

 cauthtool --print-key /etc/ceph/keyring.bin

Inside OSG Ops.: New Home for OSG Web Site

noreply@blogger.com (Rob Q) — Wed, 15 Feb 2012 07:47:21 +0000

On Frebruary 28^th the OSG Webpages located at www.opensciencegrid.org will move to the OSG Twiki. This move corresponds with the upcoming conclusion of a contract with the Chicago based web hosting service Tilted Planet.

During the scheduled production service update on the 28^th, browsers will be redirected from the current location to the new location on the main OSG Twiki page. A mockup of the new page can be seen at twiki-itb.grid.iu.edu.

This is the first step, and likely an interim web page home, in a project that will affect the OSG Public Web Pages, the OSG Twiki, the DocDB, and possibly other OSG services with web-UI’s. Evaluation of content management systems, wikis, and documentation file database solutions has already begun and will continue over the next several months. Please contact the GOC (goc@opensciencegrid.org) if you have a suggestion for packages you think should be evaluated.

An Open Science Grid Work Log: NAMD with PBS and Infiniband on NERSC Dirac

Tue, 14 Feb 2012 15:20:50 +0000

Overview

NAMD simulates molecular motion, especially of large molecules so it’s often used to simulate molecular docking problems. One particularly interesting class of docking problem is the interaction of protein molecules with other molecules such as the cell membrane. The enormous number of atoms involved in these simulations confine the kinds of information we’re able to learn about how proteins interact with and shape their environments because more atoms require more computing power. So we’re investigating using GPU accelerated nodes in a shared memory cluster to speed up simulation time.

This describes running NAMD in a multi-node configuration at NERSC Dirac to determine if we want to build out a Pegasus workflow executing in this mode through the OSG compute element. The process is, as usual with MPI codes using cluster interconnects, highly cluster specific. The next step is to determine if it’s worth it and what our alternatives are.

Approach

If you’re having a hard time running NAMD in a PBS environment over an Infiniband interconnect, you are not alone. The NAMD release notes come right to the point:

“Writing batch job scripts to run charmrun in a queueing system can be challenging.”

These links, in addition to the release notes cited above provide useful insights:

And without further delay, here’s the approach that worked on Dirac. Mileage on your cluster may vary.

#!/bin/bash

set -x
set -e

# build a node list file based on the PBS
# environment in a form suitable for NAMD/charmrun

nodefile=$TMPDIR/$PBS_JOBID.nodelist
echo group main > $nodefile
nodes=$( cat $PBS_NODEFILE )
for node in $nodes; do
   echo host $node >> $nodefile
done

# find the cluster's mpiexec
MPIEXEC=$(which mpiexec)

# Tell charmrun to use all the available nodes, the nodelist built  above and the cluster's MPI.
CHARMARGS="+p32 ++nodelist $nodefile"

As an additional wrinkle, we want to run the GPU accelerated version. That’s why we use the +idlepoll argument to NAMD.

After setting NAMD_HOME, the command to execute NAMD is:

${NAMD_HOME}/charmrun \

${CHARMARGS} ++mpiexec ++remote-shell \

${MPIEXEC} ${NAMD_HOME}/namd2 +idlepoll <input_file>

The beginning of NAMD’s output looks like this:

Info: 1 NAMD 2.8 Linux-x86_64-ibverbs-CUDA 16 dirac48 stevecox
Info: Running on 16 processors, 16 nodes, 2 physical nodes.
Info: CPU topology information available.
Info: Charm++/Converse parallel runtime startup completed at 0.025701 s
Pe 5 sharing CUDA device 0 first 0 next 6
Pe 5 physical rank 5 binding to CUDA device 0 on dirac48: ‘Tesla C1060′ Mem: 4095MB Rev: 1.3
Pe 10 sharing CUDA device 0 first 8 next 11
Pe 10 physical rank 2 binding to CUDA device 0 on dirac47: ‘Tesla C1060′ Mem: 4095MB Rev: 1.3
Pe 8 sharing CUDA device 0 first 8 next 9
Pe 8 physical rank 0 binding to CUDA device 0 on dirac47: ‘Tesla C1060′ Mem: 4095MB Rev: 1.3
Pe 2 sharing CUDA device 0 first 0 next 3
Did not find +devices i,j,k,… argument, using all
Pe 2 physical rank 2 binding to CUDA device 0 on dirac48: ‘Tesla C1060′ Mem: 4095MB Rev: 1.3

Of particular importance, note that there is a pre-built executable specific to ibverbs-CUDA – that is, it works with infiniband connected clusters with CUDA accelerated nodes.

These are the parameters of the dirac_reg queue:

[stevecox@cvrsvc01 namd]$ qstat -Qf dirac_reg
Queue: dirac_reg
 queue_type = Execution
 Priority = 10
 max_user_queuable = 500
 total_jobs = 39
 state_count = Transit:0 Queued:4 Held:27 Waiting:0 Running:8 Exiting:0
 acl_user_enable = False
 resources_max.nodect = 12
 resources_max.walltime = 06:00:00
 resources_min.nodect = 1
 resources_default.walltime = 00:05:00
 mtime = 1323823829
 resources_assigned.nodect = 34
 max_user_run = 2
 enabled = True
 started = True

So to test jobs, I ran qsub like this:

qsub -I -q dirac_reg -l walltime=06:00:00 -l nodes=4:ppn=8

The -I parameter tells qsub to start an interactive job. The walltime parameter overrides the very low default walltime. Fnially, nodes tells PBS how many cluster nodes to use and ppn specifies the processes per node to start.

After debugging, I ran the script like this:

qsub -q dirac_reg -l walltime=06:00:00 -l nodes=4:ppn=8 ./callnamd

Results

I did three runs with 2, 4, and 8 nodes. The interesting performance number for a NAMD run is days/ns or days of computation time required per nanosecond of simulation.

[stevecox@cvrsvc01 ~]$ grep -i days dev/dukechem/osg/namd/run.* | sed -e “s,.txt,,” -e “s,.*run.,,”
2way:Info: Initial time: 16 CPUs 0.0617085 s/step 0.357109 days/ns 91.0601 MB memory
2way:Info: Initial time: 16 CPUs 0.0613538 s/step 0.355057 days/ns 94.0546 MB memory
2way:Info: Initial time: 16 CPUs 0.0619225 s/step 0.358348 days/ns 94.7324 MB memory
2way:Info: Benchmark time: 16 CPUs 0.0620334 s/step 0.35899 days/ns 94.8284 MB memory
2way:Info: Benchmark time: 16 CPUs 0.0621472 s/step 0.359648 days/ns 95.09 MB memory
2way:Info: Benchmark time: 16 CPUs 0.0620733 s/step 0.359221 days/ns 95.162 MB memory
4way:Info: Initial time: 32 CPUs 0.0472537 s/step 0.273459 days/ns 83.8981 MB memory
4way:Info: Initial time: 32 CPUs 0.0470766 s/step 0.272434 days/ns 84.8605 MB memory
8way:Info: Initial time: 64 CPUs 0.0406125 s/step 0.235026 days/ns 81.0847 MB memory
8way:Info: Initial time: 64 CPUs 0.0406405 s/step 0.235188 days/ns 82.1035 MB memory
8way:Info: Initial time: 64 CPUs 0.0407004 s/step 0.235534 days/ns 82.2474 MB memory
8way:Info: Benchmark time: 64 CPUs 0.0407453 s/step 0.235794 days/ns 82.3482 MB memory
8way:Info: Benchmark time: 64 CPUs 0.040858 s/step 0.236447 days/ns 82.3975 MB memory
8way:Info: Benchmark time: 64 CPUs 0.0406536 s/step 0.235264 days/ns 82.4038 MB memory

Here are some details of NERSC Dirac’s configuration:

Dirac is a 50 GPU node cluster connected with QDR IB. Each GPU node also contains 2 Intel 5530 2.4 GHz, 8MB cache, 5.86GT/sec QPI Quad core Nehalem processors (8 cores per node) and 24GB DDR3-1066 Reg ECC memory.

44 nodes: 1 NVIDIA Tesla C2050 (code named Fermi) GPU with 3GB of memory and 448 parallel CUDA processor cores.
4 nodes: 1 C1060 NVIDIA Tesla GPU with 4GB of memory and 240 parallel CUDA processor cores.
1 node: 4 NVIDIA Tesla C2050 (Fermi) GPU’s, each with 3GB of memory and 448 parallel CUDA processor cores.
1 node: 4 C1060 Nvidia Tesla GPU’s, each with 4GB of memory and 240 parallel CUDA processor cores.

Here are results from earlier runs on a cluster with far fewer GPUs but a configuration in which accelerated nodes contain four Nvidia Teslas (like one of the Dirac nodes):

4CPU: 0.998798 days/ns
8CPU: 0.565848 days/ns
And with the production sample at 8CPU: 0.288802

Conclusions

While these findings are preliminary, indications are that having four GPUs on a single node makes a substantial performance difference.

OSG Technology Area Rumblings: Job Isolation in Condor

noreply@blogger.com (Brian Bockelman) — Tue, 14 Feb 2012 09:46:01 +0000

I'd like to share a few exciting new features under construction for Condor 7.7.6 (or 7.9.0, as it may be).

I've been working hard to improve the job isolation techniques available in Condor. My dictionary defines the verb "to isolate" as "to be or remain alone or apart from others"; when applied to the Condor context, we'd like to isolate each job from the others. We'll define process isolation as the inability of a process running in a batch job to interfere with a process not a part of the job. Interfering with processes on Linux, loosely defined, means the sending of POSIX signals, taking control via the ptrace mechanism, or writing into the other process's memory.

Process isolation is only one aspect of job isolation. Job isolation also includes the inability to interfere with other jobs' files (file isolation) and not being able to consume others' system resources such as CPU, memory, or disk (resource isolation).

In Condor, process isolation has historically been accomplished via one of two mechanisms:

Submitting user. Jobs from Alice and Bob will be submitted as the unix users alice and bob, respectively. In this model, the jobs running on the worker node will be run as users alice and bob, respectively. The processes in the job running under user bob are protected from the processes in the job running as user alice via traditional POSIX security mechanisms.

This model makes the assumption that jobs submitted by the same user do not need isolation from each other. In other words, there shouldn't be any shared user accounts!
This model also assumes the submit host and the worker node share a common user namespace. This can be more difficult to accomplish than it sounds: if the submit host has thousands of unique users, we must make sure each functions on the worker node. If the submit host is on a remote site with a different user namespace from the worker node, this may not be easily achievable!

Per-slot users. Each "slot" (roughly corresponding to a CPU) in condor is assigned a unique unix user. The job currently running in that slot is run under the associated username.

This solves the "gotchas" noted above with the submitting user isolation model.
This is difficult to accomplish in-practice if the job wants to utilize a filesystem shared between the submit and worker nodes. The filesystem security is based on two users having distinct Unix user names; in this model, there's no way to mark your files as only readable by your own jobs.

Notice both techniques require on user isolation to accomplish process isolation. Condor has an oft-overlooked third mode:

Mapping remote users to nobody. In this mode, local users (where the site admin can define the meaning of "local") get mapped to the submit host usernames, but non-local users all get mapped to user nobody - the traditional unprivileged user on Linux.

Local users can access all their files, but remote users only get access to the batch resources - no shared file systems.

Unfortunately, this is not a very secure mode as, according to the manual, the nobody account "... may also be used by other Condor jobs running on the same machine, if it is a multi-processor machine"; not very handy advice in an age where your cell phone likely is a multi-processor machine!

This third mode is particularly attractive to us - we can avoid filesystem issues for our local users, but no longer have to create the thousands of accounts in our LDAP database for remote users. However, since jobs from remote users run under the same unix user account, the traditional security mechanism of user separation does not apply - we need a new technique!

Enter PID namespaces, a new separation technique introduced in kernel 2.6.24. By passing an additional flag when creating a new process, the kernel will assign an additional process ID (PID) to the child process. The child will believe itself to be PID 1 (that is, when the child calls getpid(), it returns 1), while the processes in the parent's namespace will see a different PID. The child will be able to spawn additional processes - all will be stuck in the same inner namespace - that similarly have an inner PID different from the outer one.

Processes within the namespace can only see and interfere (send signals, ptrace, etc) with other processes inside the namespace. By launching the new job in its own PID namespace, Condor can achieve process isolation without user isolation: the job processes are isolated from all other processes on the system.

Perhaps the best way to visualize the impact of PID namespaces in the job is to examine the output of ps:

[bbockelm@localhost condor]$ condor_run ps faux
USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
bbockelm     1  0.0  0.0 114132  1236 ?        SNs  11:42   0:00 /bin/bash /home/bbockelm/.condor_run.3672
bbockelm     2  0.0  0.0 115660  1080 ?        RN   11:42   0:00 ps faux

Only two processes can be seen from within the job - the shell executing the job script and "ps" itself.

Releasing a PID namespaces-enabled Condor is an ongoing effort: https://condor-wiki.cs.wisc.edu/index.cgi/tktview?tn=1959; I've recently re-designed the patch to be far less intrusive on the Condor internals by switching from the glibc clone() call to the clone syscall. I am hopeful it will make it in the 7.7.6 / 7.9.0 timescale.

From a process isolation point-of-view, with this patch, it now is safe to run jobs as user "nobody" or re-introduce the idea of shared "group accounts". For example, we could map all CMS users to a single "cmsuser" account without having to worry about these becoming a vector for virus infection.

However, the story of job isolation does not end with PID namespaces. Stay tuned to find out how we are tackling file and resource isolation!

An Open Science Grid Work Log: Virtual Machines for OSG

Fri, 10 Feb 2012 17:36:22 +0000

Overview

The ability to run a virtual machine with a self-contained computing environment has major advantages. Users can

Choose the operating system that’s best for the application
Execute programs that require elevated privileges
Install any software they need
Dynamically configure machine attributes like the number of cores to suit the host environment

The Engage VO is beginning to see researchers new to OSG whose default mode of operation is to spin up a VM on EC2. They quickly get used to having complete control of the computing environment.

Context

This capability has been explored for the Open Science Grid before. Clemson built Kestrel which supports KVM based virtualization with an XMPP communication architecture. Then STAR used Kestrel with great success. Clemson now also provides OneCloud based on OpenNebula.

There’s also been work by Brian Bockelman, Derek Weitzel and others to configure virtual machines running Condor to join the submit host’s pool. Infrastructure background for that work and lots of great information is available at the team’s blog.

Recently, I’ve had new Engage users who are heavy users of virtualization. As mentioned before, they tend to assume control over the environment. This background can make the need to specially prepare executables for the OSG by static compilation and other packaging seem onerous. Many Engage users, it should be added, have input and output file sizes in the low number of gigabytes and are not familiar with High Throughput Computing or a command line approach to virtualization.

They asked if it was possible to run virtual machines on the OSG so I set out to look for an approach that would allow researchers to

Create virtual machines on their desktops using simple graphical tools
Deploy virtual machines onto the OSG
Transfer input files to and from the virtual machine
Avoid complex interactions with HTC plumbing like configuring X.509 certs, Condor, etc.

Virtualization on RENCI-Blueberry

Since virtualization is not a currently supported technology on the OSG, step one is to create a small area in which we change that. RENCI’s Blueberry is a new cluster made of older machines that we’ve recently brought online. It’s a ROCKS, Centos 5.7, Torque/PBS cluster with a small number of virtualization capable nodes.

Here’s an overview of changes we made to the cluster:

First, we installed these packages on the virtualization capable nodes:

Libvirt: A library of low level capabilities supporting virtualization.
QEMU: Virtual machine emulation layer
KVM: A virtualization kernel module
XMLStarlet: A command line XSLT engine

We configured QEMU to allow the engage user to execute virsh.

Then we created a Torque/PBS queue called virt grouping the upgraded machines.

A new GlideinWMS group was added on the Engage VO frontend. Jobs deploying virtual machines are decorated with the following modifications:

Job Requirements: && (CAN_RUN_VIRTUAL_MACHINE == TRUE)
Job Attributes: +RequiresVirtualization=TRUE

A new resource was added to the GlideinWMS factory with RSL pointing at the virt queue on RENCI-Blueberry.

Creating and Running a VM

Virtual machines were created using virt-manager, the Virtual Machine Manager. It’s a graphical application providing a wizard like interface for creating and managing VMs.

We used the command line virsh tool to export an XML description of the running virtual machine. Then the XML description and the disk image (the large file containing all of the VMs data) were moved to the Engage submit host.

The OSG job was designed to

Download the virtual machine’s XML description
Determine the number of CPUs on the machine
Modify the XML description to specify
- The appropriate number of CPUs
- The correct location for the VM image file
Download the virtual machine
Execute the virtual machine

This works. Jobs configured to run in the GlideinWMS virt group on the Engage submit node map to glideins on RENCI-Blueberry. There, the jobs download the XML config and the image, make the needed edits and spawn the virtual machine.

Getting Work to the Virtual Machine

Now, if you’ve tried to do this kind of thing before, you realize this is where things get tricky.

When the virtual machine launches, it has no idea what to do. This is part of the reason that some previous approaches put Condor on the machine. That way, it can join an existing Condor pool and has all the good things that Condor brings us in terms of file transfer, matching and so on. But getting credentials into the virtual machine securely to allow it to join the Engage pool is tricky. If you know how to do that, please leave a comment.

Alternatives … and OS Versions

Now, in principle, there are two other ways to do this that would work fine. If we can get files onto and off of the machine, it would be Ok to transfer them into and out of the worker node the old fashioned way – globs-url-copy. So here are two other mechanisms for file exchange between a host and a guest:

Shared Host/Guest Filesystem: More recent versions of Libvirt/QEMU/KVM that support sharing filesystems between the host and guest. In this model, the guest’s XML description can specify a directory on the host that should be mounted within the guest. But, as I mentioned, the RENCI-Blueberry cluster runs CentOS 5.7. As such, only a significantly older version of the virtualization stack is supported. We discussed upgrading to a newer version but that would prevent this solution from being generally reproducible on OSG.

Guestfish: Next, there’s libguest and the associated interactive shell guestfish. Guestfish lets you mount a disk image in user space. That is, there’s no need to use root privileges. It also has convenient wrapper scripts for copying a file into and out of an image. But, again, it requires a version of CentOS significantly greater than 5.7.

From this angle, it looks like VMs on OSG could be an every day occurrence if it were not for very low OS version numbers.

File Sharing REST API

Before giving the approach up for dead, I decided to try something off the beaten path.

Beanstalk: I installed beanstalk on the submit node. Beanstalk is a very simple HTTP based queue. You can put messages on the queue and get them off. You can name queues – which it refers to, weirdly – as tubes. Beanstalk does not have a notion of authentication so that’s not great.

Beanstalkc: This is the Python client for Beanstalk.

Box.net: One of many file sharing sites with a REST API.

Workflow

A command line Box.net authentication script does token negotiation with Box.net mostly from the command line using wget and curl.

Then, Box.net URLs are published into the Engage event queue.

When virtual machines run, they install and run the boot script.

It installs the Beanstalk client and reads a single item from the event queue which it downloads from Box.net and processes. Queues are appropriately named so that different users and jobs never collide.

Finally, it converts the download URL to an upload URL and publishes the results of the run via the file sharing API.

So at the end of my run of 3 VM’s on RENCI-Blueberry, there were three files waiting for me at Box.net.

Conclusions

I invite comments on how others have secured communication to a VM on OSG. I’d love to hear.

In particular, as mentioned above, I’d love to hear how others have gotten X.509 credentials onto a VM in this environment.

Anyone else running VMs on the OSG?

Condor Project News: Red Hat announces the release of Red Hat Enterprise MRG 2.1, (February 10, 2012)

Fri, 10 Feb 2012 05:00:00 +0000

offering increased performance, reliability, interoperability, as presented in this news release.

Derek's Blog: Fedora 16 on OpenStack

noreply@blogger.com (Derek Weitzel) — Tue, 07 Feb 2012 13:10:18 +0000

After following Brian's guide on installing Fedora 15 on OpenStack, I thought I would try my hand at Fedora 16. There where a few differences.

Filesystem Differences
Brian's guide installed Fedora using LVM. I installed Fedora without LVM (there's a little checkbox on the partition page of Anaconda). Without LVM, I can skip the steps on listing the physical volumes and logical volumes to find the start and end of the partition.

Also, Fedora 16 uses gpt partition. fdisk command cannot read the partition table, therefore I had to install gdisk (in epel). Running it has very similar command and output:

$ /usr/sbin/gdisk -l /tmp/fedora16
GPT fdisk (gdisk) version 0.8.1

Partition table scan:
  MBR: protective
  BSD: not present
  APM: not present
  GPT: present

Found valid GPT with protective MBR; using GPT.
Disk /tmp/fedora16: 20971520 sectors, 10.0 GiB
Logical sector size: 512 bytes
Disk identifier (GUID): A351197B-8233-4811-9B28-69A1DE121AD2
Partition table holds up to 128 entries
First usable sector is 34, last usable sector is 20971486
Partitions will be aligned on 2048-sector boundaries
Total free space is 4029 sectors (2.0 MiB)

Number  Start (sector)    End (sector)  Size       Code  Name
   1            2048            4095   1024.0 KiB  EF02  
   2            4096         1028095   500.0 MiB   EF00  ext4
   3         1028096        16777215   7.5 GiB     0700  
   4        16777216        20969471   2.0 GiB     8200

Then, to extract the image:

dd if=/tmp/fedora16 of=/tmp/server-extract.img skip=1028096 count=$((16777215-1028096)) bs=512

SSH Key Differences
Brian's guide instructed you to create a /etc/rc.local. Fedora 16 sees the introduction of systemd, which no longer executes rc.local. Instead, it looks for the file /etc/rc.d/rc.local (possibly a symlink to /etc/rc.local?). This file needs to be executable and be sure to include the shebang.

Also, Fedora 16's selinux doesn't label the root file system correctly (BUG), and simply making the .ssh directory doesn't not allow sshd to read it. To solve selinux problem, I disabled selinux (bad, bad me).

Common Commands
After installing Fedora 16 into an image, and extracting the kernel and ramdisk, there where a few commands that where executed over and over as I debugged the image:

Make the changes to the image:

sudo /usr/libexec/qemu-kvm -m 2048 -drive file=/tmp/fedora16 -net nic -net user -vnc 127.0.0.1:0 -cpu qemu64 -M rhel5.6.0 -smp 2 -daemonize

Extract the partition:

dd if=/tmp/fedora16 of=/tmp/server-extract.img skip=1028096 count=$((16777215-1028096)) bs=512

Start the VM to change the label on the image:

sudo /usr/libexec/qemu-kvm -m 2048 -drive file=/tmp/fedora16 -net nic -net user -vnc 127.0.0.1:0 -cpu qemu64 -M rhel5.6.0 -smp 2 -daemonize -drive file=/tmp/server-extract.img

Rename the image to something appropriate:

mv /tmp/server-extract.img /tmp/fedora16-extracted.img

Bundle the image for OpenStack:

euca-bundle-image --kernel aki-0000002e --ramdisk ari-0000002f -i /tmp/fedora16-extracted.img -r x86_64

Upload the image to OpenStack:

euca-upload-bundle -b derek-bucket -m /tmp/fedora16-extracted.img.manifest.xm

euca-register derek-bucket/fedora16-extracted.img.manifest.xml

Now to build OSG packages for Fedora... maybe not.

Condor Project News: Upgrades recommended (February 06, 2012)

Mon, 06 Feb 2012 05:00:00 +0000

Please upgrade to the latest versions of the stable and development series. There have been many important bug fixes in both, including a fix for a possible Denial Of Service attack from trusted users. (Details here)

Spinning: Pool utilization

Tue, 31 Jan 2012 02:06:18 +0000

Here is a utilization script for a Condor pool.

$ ./utilization.sh
       Unavailable Available    Total     Used:  Avail   Total
Slots         5968      5451    11419     4179  76.66%  36.59%
Cpus          6314      5903    12217     4631  78.45%  37.90%
Memory    14277325  11776800 26054125  9908190  84.13%  38.02%

And, if you know your workload will not run on slots with less then 1GB of memory, you can filter out slots that are too small,

$ ./utilization.sh 'Memory < 1024'
       Unavailable Available    Total     Used:  Avail   Total
Slots         6292      5127    11419     4177  81.47%  36.57%
Cpus          6638      5579    12217     4629  82.97%  37.88%
Memory    14592711  11461414 26054125  9904193  86.41%  38.01%

Remember, if an attribute is not on all slots you need to use the meta-comparison operators: =?= and =!=, e.g. 'MyCustomAttr =!= True'.

OSG Technology Area Rumblings: openstack - update

noreply@blogger.com (TomW) — Fri, 27 Jan 2012 10:36:26 +0000

Last time I was able to deploy an image. Next step would be to list it and then run. But I have hit problems.

To list images I run command:

euca-describe-images

which hangs up forever and after long time exits with message "connection reset by peer".

I have disabled iptables to eliminate firewall issues. No help.

All manuals assume that euca-describe-images should simply run and do not give instruction what to do if it does not.

Following Josh's advice I did:

strace -o edi_output -f -ff euca-describe-images

and then I looked into the output files. It seems that there might be two problems:

Some euca2ools files are missing - in particular the .eucarc configuration file.
There are messages about missing python files, like for example "open("/usr/lib64/python2.6/site-packages/gtk-2.0/org.so", O_RDONLY) = -1 ENOENT (No such file or directory)" (There are manu more like that).

So it seems that the eucatools installation described in previous posts may be not complete - and it missed some key files. Or python (which we already know had to be patched) is not OK. Or both.

That's all I know for now.

Derek's Blog: Testing an Globus Free OSG-Software (From EPEL(-testing))

noreply@blogger.com (Derek Weitzel) — Tue, 24 Jan 2012 20:45:43 +0000

As you may or may not know, there is a massive globus update pending in EPEL that will update globus to the version the OSG distributes. What this means is much less work for the osg-software team since we will not have to build and support our own builds of globus.

Testing the globus from EPEL while installing some packages from osg repos is not a trivial matter.

Disable the priority of the OSG repo
Exclude globus and related packages that are already in EPEL from the osg repo.

Below is my final file /etc/yum.repos.d/osg.repo

Notice the many excludes in the file, the list may not be complete.

Installation is just:

yum install osg-client-condor --enablerepo=epel-testing

UPDATE!!!!
Testing Results
very good!

I ran 3 tests, all completely successful.
1. globus-job-run against a rpm CE.

$ globus-job-run pf-grid.unl.edu/jobmanager-fork /bin/sh -c "id"
uid=1761(hcc) gid=4001(grid) groups=4001(grid)

2. Condor-G submission
Condor-G Submission worked without problems. The submission file is below:
3. And globus-url-copy worked:

$ globus-url-copy gsiftp://pf-grid.unl.edu/etc/hosts ./hosts

Spinning: EC2, VNC and Fedora

Tue, 24 Jan 2012 15:27:00 +0000

If you have ever wondered about running a desktop session in EC2, here is one way to set it up and some pointers.

First, start an instance, my preferred way is via Condor. I used ami-60bd4609 on an m1.small, providing a basic Fedora 15 server. Make sure the instance’s security group has port 22 (ssh) open.

Second, install a desktop environment, e.g. yum groupinstall 'GNOME Desktop Environment'. This is 467 packages and will take about 18 minutes.

Third, install and setup a VNC server. yum install vnc-server ; vncpasswd ; vncserver :1. This produces a running desktop that can be contacted by a vncviewer.

Finally, connect via an SSH secured VNC session.

VNC_VIA_CMD='/usr/bin/ssh -i KEYPAIR.pem -l ec2-user -f -L "$L":"$H":"$R" "$G" sleep 20' vncviewer localhost:1 -via INSTANCE_ADDRESS

What’s going on here? vncviewer allows for a proxy host when connecting to the vncserver. That is the -via argument. The VNC_VIA_CMD is an environment variable that specifies the command used to connect to the proxy. Here it is modified to provide the keypair needed to access the instance, and the user ec2-user, which is the default user on Fedora AMIs. The INSTANCE_ADDRESS is the Hostname from condor_ec2_q.

Alternatively, ssh-add KEYPAIR.pem followed by vncviewer localhost:1 -via ec2-user@INSTANCE_ADDRESS. However, be careful if you have many keys stored in your ssh-agent. They will all be tried and the remote sshd may reject your connection before the proper keypair is found.

Tips:

It takes about 20 minutes from start to vncviewer. Once the instance is setup consider creating your own AMI.
Set a password for ec2-user, otherwise the screensaver will lock you out. Use sudo passwd ec2-user.
Remember AWS charges for data transmitted out of the instance, as well as the uptime of the instance, see EC2 Pricing. You will want to figure out how much bandwidth your workflow takes on average to figure out total cost. For me, a half hour of browsing Planet Fedora, editing with emacs, and compiling some code, transmitted about 60MB of data. That measurement is the difference in eth0′s “TX bytes” as reported by ifconfig. This is not a perfect estimate because there is may have been data transferred within EC2, which is not charged.
For transmit rates, consider running bmw-ng to see what actions use the most bandwidth.
Generally, make the screen update as little as possible. Constantly changing graphics on web pages can run 60-120KB/s. Compare that to a text console and emacs producing a TX rate closer to 5-25KB/s.
Cover consoles with compilations, or compile in a low verbosity mode.

Derek's Blog: Initial EL6 Packages for OSG

noreply@blogger.com (Derek Weitzel) — Fri, 20 Jan 2012 14:58:38 +0000

Last night I completed initial packages for EL6 support. Just like for EL5, the first OSG component I created is the osg-wn-client.

The osg-wn-client has a complicated dependency tree. Easily some of the most difficult packages where form glite.

Just some quick tidbits that made the transition easier:

UUID Differences
uuid.h and the associated library is used by many applications. In el5, uuid is provided by the e2fsprogs package. In el6, it has it's own package, libuuid. It was common for me to copy this tidbit into a few packages:

gsoap Differences
glite-fts-client and glite-data-delegation-api-c both use gsoap. In the past, it was common to copy stdsoap2.c from the gsoap distribution and compile that into your program. Now that gsoap is a regular library though, it should be linked into the system's version. In order to do this, I had to add patches to the Makefiles for both packages to link against the system's gsoap.

What's next?
The next step is the osg-client. Since there are no more glite packages for the osg-client, this step should be easier.

Spinning: Manage inventory with Wallaby

Sun, 15 Jan 2012 17:49:53 +0000

Wallaby will manage your configuration, as well as an inventory of your machines. It can differentiate between machines that are expected to be present and those that opportunistically appear.

Build the roster with wallaby add-node -

$ wallaby add-node node0.local node1.local node2.local
Adding the following node: node0.local
Console Connection Established...
Adding the following node: node1.local
Adding the following node: node2.local
$ for i in $(seq 3 10); do wallaby add-node node$i.local; done
Adding the following node: node3.local
Console Connection Established...
Adding the following node: node4.local
Console Connection Established...
...

List expected nodes (provisioned) -

$ wallaby inventory
Console Connection Established...
P        Node name                 Last checkin
-        ---------                 ------------
+      node0.local Wed Jan 11 07:32:33 -0500 20
+      node1.local Thu Jan 05 12:15:00 -0500 20
+     node10.local Wed Jan 11 07:31:56 -0500 20
+      node2.local Wed Jan 11 07:31:56 -0500 20
+      node3.local Wed Jan 11 07:15:21 -0500 20
+      node4.local Wed Jan 11 07:31:42 -0500 20
+      node5.local Wed Jan 11 07:16:47 -0500 20
+      node6.local                        never
+      node7.local Wed Jan 11 07:32:33 -0500 20
+      node8.local Wed Jan 11 07:32:33 -0500 20
+      node9.local Wed Jan 11 07:30:47 -0500 20
-      robin.local Thu Dec 15 14:11:35 -0500 20
-      woods.local Tue Jan 10 20:33:47 -0500 20

List opportunistic, bonus nodes (unprovisioned) -

$ wallaby inventory -o unprovisioned
Console Connection Established...
P        Node name                 Last checkin
-        ---------                 ------------
-      robin.local Thu Dec 15 14:11:35 -0500 20
-      woods.local Tue Jan 10 20:33:47 -0500 20

Provisioned nodes that have never checked in, maybe setup failed -

$ wallaby inventory -c 'last_checkin == 0 && provisioned'
Console Connection Established...
P        Node name                 Last checkin
-        ---------                 ------------
+      node6.local                        never

Provisioned node that have not checked in for the past 4 hours, maybe machine is down -

$ wallaby inventory -c 'last_checkin > 0 && last_checkin < 4.hours_ago && provisioned'
Console Connection Established...
P        Node name                 Last checkin
-        ---------                 ------------
+      node1.local Thu Jan 05 12:15:00 -0500 20

Unprovisioned nodes that have not checked in for 48 hours, candidates for wallaby remove-node -

$ wallaby inventory -c 'last_checkin < 48.hours_ago && !provisioned'
Console Connection Established...
P        Node name                 Last checkin
-        ---------                 ------------
-      robin.local Thu Dec 15 14:11:35 -0500 20

Enjoy.

OSG Technology Area Rumblings: How to register an image in openstack

noreply@blogger.com (TomW) — Tue, 10 Jan 2012 08:53:04 +0000

After having installed and configured the worker and controller nodes of the openstack testbed we would like to upload images into it.

First I downloaded some images to /root/images on controller node. One is from Xin and another one is a minimal image for testing I got from the net. I have no idea what are they worth.

Then I tried to follow the instructions

http://docs.openstack.org/cactus/openstack-compute/admin/content/part-ii-getting-virtual-machines.html

which go like this:

image="ubuntu1010-UEC-localuser-image.tar.gz"
wget http://c0179148.cdn1.cloudfiles.rackspacecloud.com/ubuntu1010-UEC-localuser-image.tar.gz
uec-publish-tarball $image [bucket-name] [hardware-arch]

and I could not find where does the

uec-publish-tarball

command comes from. Finally I realized that it comes from Ubuntu and the manual became Ubuntu specific without saying it explicitly.

So I tried different approach.

cd /root/images

glance add name="My Image" < sl61-kvm.tar.bz2 # the image I got from Xin

The command responded that the image got Id=1, which is a good sign.

Then I did:

glance show 1

and got:

URI: http://0.0.0.0/images/1
Id: 1
Public: No
Name: My Image
Size: 199737477
Location: file:///var/lib/glance/images/1
Disk format: raw
Container format: ovf

Which suggests that the file is in the system. But when I tried:

glance index

it said:

no public images found

So I tried to register it again:

glance add name="My Image" is_public=true < sl61-kvm.tar.bz2
Added new image with ID: 2

I tried to list:

glance index
Found 1 public images...
ID Name Disk Format Container Format Size
---------------- ------------------------------ -------------------- -------------------- --------------
2 My Image raw ovf 199737477

So it seems we have uploaded an image to the system.

Now I have to figure out how to run it.

OSG Technology Area Rumblings: How to configure worker node - part 2

noreply@blogger.com (TomW) — Fri, 06 Jan 2012 09:37:49 +0000

Compute node configuration - continued

We execute the following commands:

This command is supposed to synchronize the database:

/usr/bin/nova-manage db sync 
Now we have to create users and projects. We call both users and projects "nova"

/usr/bin/nova-manage user admin nova
/usr/bin/nova-manage project create nova nova 
/usr/bin/nova-manage network create 192.168.0.0/24 1 256

We check that users and projects were created correctly:

/usr/bin/nova-manage project list
nova

/usr/bin/nova-manage user list
nova

Create Certifications

On the controller node execute

mkdir –p /root/creds

/usr/bin/python /usr/bin/nova-manage project zipfile nova nova /root/creds/novacreds.zip

If you encounter a python error, then apply the python patch described few posts earlier.

Create /root/creds on the compute node and copy the
novacreds.zip file there. Then unpack it

unzip /root/creds/novacreds.zip -d /root/creds/

A few files will appear, among them
/root/creds/novarc . This file needs to be appended to .bashrc, but there is a catch:
first line of the file has an error and has to be replaced:


Original line:

NOVA_KEY_DIR=$(pushd $(dirname $BASH_SOURCE)>/dev/null; pwd; popd>/dev/null)

has to be replaced with

NOVA_KEY_DIR=~/creds

The content of novarc file now is

NOVA_KEY_DIR=~/creds

export EC2_ACCESS_KEY="XXXXXXXXXXXXXXXXXXXXXXXX:nova"
export EC2_SECRET_KEY="XXXXXXXXXXXXXXXXXXXXXXXX"
export EC2_URL="http://130.199.148.53:8773/services/Cloud"
export S3_URL="http://130.199.148.53:3333"
export EC2_USER_ID=42 # nova does not use user id, but bundling requires it
export EC2_PRIVATE_KEY=${NOVA_KEY_DIR}/pk.pem
export EC2_CERT=${NOVA_KEY_DIR}/cert.pem
export NOVA_CERT=${NOVA_KEY_DIR}/cacert.pem
export EUCALYPTUS_CERT=${NOVA_CERT} # euca-bundle-image seems to require this set
alias ec2-bundle-image="ec2-bundle-image --cert ${EC2_CERT} --privatekey ${EC2_PRIVATE_KEY} --user 42 --ec2cert ${NOVA_CERT}"
alias ec2-upload-bundle="ec2-upload-bundle -a ${EC2_ACCESS_KEY} -s ${EC2_SECRET_KEY} --url ${S3_URL} --ec2cert ${NOVA_CERT}"
export NOVA_API_KEY="XXXXXXXXXXXXXXXXXXXXXXXXXXX"
export NOVA_USERNAME="nova"
export NOVA_URL="http://130.199.148.53:8774/v1.0/"


Where "XXXX.." strings denote keys which I do not post here, for security.

The content of novarc file should now be added to bashrc:

cat /root/creds/novarc >> ~/.bashrc source ~/.bashrc

This should be done both on compute and controller nodes.

Enable access to worker node

First unset a proxy and then do:

euca-authorize -P icmp -t -1:-1 default euca-authorize -P tcp -p 22 default

An Open Science Grid Work Log: Protected: Grayson Science Workflow on the Hybrid Grid

Thu, 05 Jan 2012 16:49:51 +0000

This post is password protected. You must visit the website and enter the password to continue reading.

OSG Technology Area Rumblings: How to configure worker node

noreply@blogger.com (TomW) — Thu, 05 Jan 2012 13:09:38 +0000

In the following I will describe how to configure the worker node. I assume that the worker node has been already installed following the instructions posted on this blog.

Firs of all, before we start, we still need to add nova-network (it has not been installed so far).

Do:

yum install openstack-nova-network

Once this is done, we can go on and edit the /etc/nova/nova.conf file.

First, add to the file the option

--daemonize=1

The relevant switches are:

--sql_connection
--s3_host
--rabbit_host
--ec2_api
--ec2_url
--fixed_range
--network_size

In the end the configuration file should look like:

--auth_driver=nova.auth.dbdriver.DbDriver
--buckets_path=/var/lib/nova/buckets
--ca_path=/var/lib/nova/CA
--cc_host=
--credentials_template=/usr/share/nova/novarc.template
--daemonize=1
--dhcpbridge_flagfile=/etc/nova/nova.conf
--dhcpbridge=/usr/bin/nova-dhcpbridge
--ec2_api=130.199.148.53
--ec2_url=http://130.199.148.53:8773/services/Cloud
--fixed_range=192.168.0.0/16
--glance_host=
--glance_port=9292
--image_service=nova.image.glance.GlanceImageService
--images_path=/var/lib/nova/images
--injected_network_template=/usr/share/nova/interfaces.rhel.template
--instances_path=/var/lib/nova/instances
--keys_path=/var/lib/nova/keys
--libvirt_type=kvm
--libvirt_xml_template=/usr/share/nova/libvirt.xml.template
--lock_path=/var/lib/nova/tmp
--logdir=/var/log/nova
--logging_context_format_string=%(asctime)s %(name)s: %(levelname)s [%(request_id)s %(user)s %(project)s] %(message)s
--logging_debug_format_suffix=
--logging_default_format_string=%(asctime)s %(name)s: %(message)s
--network_manager=nova.network.manager.VlanManager
--networks_path=/var/lib/nova/networks
--network_size=8
--node_availability_zone=nova
--rabbit_host=130.199.148.53
--routing_source_ip=130.199.148.53
--s3_host=130.199.148.53
--scheduler_driver=nova.scheduler.zone.ZoneScheduler
--sql_connection=mysql://{USER}:{PWD}@130.199.148.53/{DATABASE}
--state_path=/var/lib/nova
--use_cow_images=true
--use_ipv6=false
--use_s3=true
--use_syslog=false
--verbose=false
--vpn_client_template=/usr/share/nova/client.ovpn.template

where {USER},{PWD} and {DATABASE} denote nova database user, pasword and database name.

Now go to the controller node and open the following ports for incoming connections: 3333,3306,5672,8773,8000.

Go back to worker node and prepare /root/bin/openstack-init.sh script with the following content:

#!/bin/bash
for n in ajax-console-proxy compute vncproxy network; do
service openstack-nova-$n $@;
done

Then run

/root/bin/openstack-init.sh stop
Stopping OpenStack Nova Web-based serial console proxy: [ OK ]
Stopping OpenStack Nova Compute Worker: [ OK ]
Stopping OpenStack Nova VNC Proxy: [ OK ]
Stopping OpenStack Nova Network Controller: [ OK ]
[root@gridreserve30 compute]# /root/bin/openstack-init.sh start
Starting OpenStack Nova Web-based serial console proxy: [ OK ]
Starting OpenStack Nova Compute Worker: [ OK ]
Starting OpenStack Nova VNC Proxy: [ OK ]
Starting OpenStack Nova Network Controller: [ OK ]

to be continued...

OSG Technology Area Rumblings: What's the hold-up?

noreply@blogger.com (Brian Bockelman) — Thu, 29 Dec 2011 17:12:19 +0000

Do you have the following diagram memorized?

If your site runs Condor, you probably should. It shows the states of the condor_startd, the activities within the state, and the transitions between them. If you want to have jobs reliably pre-empted (or is that killed? Or vacated?) from the worker node for something like memory usage, a clear understanding is required.

However, the 30 state transitions might be a bit much for some site admins who just want to kill jobs that go over a memory limit. In such a case, admins can utilize the SYSTEM_PERIODIC_REMOVE or the SYSTEM_PERIODIC_HOLD configuration parameters on the condor_schedd to respectively remove or hold jobs.

These expressions periodically evaluate the schedd's copy of the job ClassAd (by default, once every 60s); if they evaluate to true for a given job, they will remove or hold it. This will almost immediately preempt execution on the worker node.

[Note: While effective and simple, these are not the best way to accomplish these sort of policies! As the worker node may talk to multiple schedd's (via flocking, or just through a complex pool with many schedd's), it's best to express the node's preferences locally.]

At HCC, the periodic hold and release policy looks like this:

# hold jobs using absurd amounts of disk (100+ GB)
SYSTEM_PERIODIC_HOLD = \
   (JobStatus == 1 || JobStatus == 2) && ((DiskUsage > 100000000 || ResidentSetSize > 1600000))

# forceful removal of running after 2 days, held jobs after 6 hours,
# and anything trying to run more than 10 times
SYSTEM_PERIODIC_REMOVE = \
   (JobStatus == 5 && CurrentTime - EnteredCurrentStatus > 3600*6) || \
   (JobStatus == 2 && CurrentTime - EnteredCurrentStatus > 3600*24*2) || \
   (JobStatus == 5 && JobRunCount >= 10) || \
   (JobStatus == 5 && HoldReasonCode =?= 14 && HoldReasonSubCode =?= 2)

We place anything on hold that goes over some pre-defined resource limit (disk usage or memory usage). Jobs are removed if they have been on hold for a long time, have run for too long, have restarted too many times, or are missing their input files.

Note that this is a flat policy for the cluster - heterogeneous nodes with larges amounts of RAM per core would not be well-utilized. We could tweak this by having users utilize the RequestMemory attribute to their job's ad (defaulting to 1.6GB), place into the Requirements that the slot have sufficient memory, and have the node only accept jobs that request memory below a certain threshold. The expression above could then be tweaked to hold jobs where (ResidentSetSize > RequestMemory). Perhaps more on that in the future if we go this route.

While the SYSTEM_PERIODIC_* expressions are useful, Dan Bradley recently introduce me to the SYSTEM_PERIODIC_*_REASON parameter. This allows you to build a custom hold message for the user whose jobs you're about to interrupt. The expression is evaluated within the context of the job's ad, and the resulting string is placed in the job's HOLD_REASON. As an example, previously, the hold message was something bland and generic:

The SYSTEM_PERIODIC_HOLD expression evaluated to true.

Why did it evaluate to true? Was it memory or disk usage? When it was held, how bad was the disk/memory usage? These things can get lost in the system. Oops. We added the following to our schedd's configuration:

# Report why the job went on hold.
SYSTEM_PERIODIC_HOLD_REASON = \
   strcat("Job in status ", JobStatus, \
   " put on hold by SYSTEM_PERIODIC_HOLD due to ", \
   ifThenElse(isUndefined(DiskUsage) || DiskUsage < 100000000, \
      strcat("memory usage ", ResidentSetSize), \
      strcat("disk usage ", DiskUsage)), ".")

Now, we have beautiful error messages in the user's logs explaining the issue:

Job in status 2 put on hold by SYSTEM_PERIODIC_HOLD due to memory usage 1620340."

One less thing to get confused about!

Spinning: Amazon S3 Object Expiration, what about Instance Expiration

Wed, 28 Dec 2011 12:46:06 +0000

AWS is providing APIs that take distributed computing concerns into account. One could call them cloud concerns these days. Unfortunately, not all cloud providers are doing the same.

Idempotent instance creation showed up in Sept 2010, providing the ability to simplify interactions with EC2. Idempotent resource allocation is critical for distributed systems.

S3 object expiration appeared in Dec 2011, allowing for service-side managed deallocation of S3 resources.

Next up? It would be great to have an EC2 instance expiration feature. One that could be (0) assigned per instance and (1) adjusted while the instance exists. Bonus if can also be (2) adjusted from within the instance without credentials. Think leases.

OSG Technology Area Rumblings: A simple iRODS Micro-Service

noreply@blogger.com (Ashu Guru) — Fri, 23 Dec 2011 06:14:49 +0000

Introduction

The goal I had for this task was to identify and understand the steps and configurations involved in writing a micro-service and seeing it in action - for details regarding iRODS please refer to documentation at https://www.iRODS.org/. The micro-service that I wrote is very simplistic (it writes a hello world message to the system log), however it serves its purpose by providing an overview of steps that will be involved in writing a useful micro-service.

Before I document the configurations and codes involved in creating and registering the new micro-service let’s look at figure 1.

Figure 1 shows a high level view of invocation of a micro-service by the iRODS rules engine. One way of looking at the micro-service and the iRODS rule engine is to think of it as an event based triggering system that can perform ‘operations’ on the data objects, and/or external resources. The micro-services are registered in iRODS rule definitions and the rule engine invokes them based on the condition specified for that rule. For a list of places in the iRODS workflow where a micro-service may be triggered please visit: https://www.irods.org/index.php/Default_iRODS_Rules.

Also you may refer to https://www.iRODS.org/index.php/Rule_Engine for a detailed diagram of a micro-service invocation.

Figure 2 above shows the communication between the iRODS rule engine and a micro-service. A simplistic view of the communication layers is that the rule engine calls a defined C procedure, which exposes its functionality through an interface (commonly prefixed with msi). The arguments to the procedure are passed through a structure named msParam_t that is defined below:

typedef struct MsParam {
  char *label;
  char *type;         /* this is the name of the packing instruction in
                       * rodsPackTable.h */
  void *inOutStruct;
  bytesBuf_t *inpOutBuf;
} msParam_t;

Writing the micro-service

Figure 3 shows the steps involved in creating a new micro-service:

Write the C procedure

The C code below (lets call it test.c) has a function writemessage that writes a message to the system log. There is an interface to the function named msiWritemessage which exposes the writemessage function. The msi function takes a list of arguments of type msParam_t and a last argument of type ruleExecInfo_t for the result of the operation.

#include <stdio.h>
#include <unistd.h>
#include <syslog.h>
#include <string.h>
#include "apiHeaderAll.h"


void writemessage(char arg1[], char arg2[]);
int msiWritemessage(msParam_t *mParg1, msParam_t *mParg2,  ruleExecInfo_t *rei);


void writemessage(char arg1[], char arg2[]) {
    openlog("slog", LOG_PID|LOG_CONS, LOG_USER);
    syslog(LOG_INFO, "%s %s from micro-service", arg1, arg2);
    closelog();
}

int msiWritemessage(msParam_t *mParg1, msParam_t *mParg2,  ruleExecInfo_t *rei)
{
 char *in1;
 int *in2;
 RE_TEST_MACRO ("    Calling Procedure");
 // the above line is needed for loop back testing using irule -i option
 if ( strcmp( mParg1->type, STR_MS_T ) == 0 )
 {
    in1 = (char*) mParg1->inOutStruct;
 }
 if ( strcmp( mParg2->type, INT_MS_T ) == 0 )
 {
    in2 = (int*) mParg2->inOutStruct;
 }
 writemessage(in1, in1);
 return rei->status;
}

Next I will make a folder structure in the module folder of iRODS home for placing this micro-service and copy a few files from an example properties module and modify them to fit the test.c micro-service

cd ~irods
mkdir modules/HCC
cd modules/HCC

mkdir microservices
mkdir rules
mkdir lib
mkdir clients
mkdir servers

mkdir microservices/src
mkdir microservices/include
mkdir microservices/obj
cp ../properties/Makefile .
cp ../properties/info.txt .

Listed below is my working copy of Makefile and the info.txt

#Makefile
ifndef buildDir
buildDir = $(CURDIR)/../..
endif

include $(buildDir)/config/config.mk
include $(buildDir)/config/platform.mk
include $(buildDir)/config/directories.mk
include $(buildDir)/config/common.mk

#
# Directories
#
MSObjDir =    $(modulesDir)/HCC/microservices/obj
MSSrcDir =    $(modulesDir)/HCC/microservices/src
MSIncDir =    $(modulesDir)/HCC/microservices/include

# Source files

OBJECTS =    $(MSObjDir)/test.o


# Compile and link flags
#
INCLUDES +=    $(INCLUDE_FLAGS) $(LIB_INCLUDES) $(SVR_INCLUDES)
CFLAGS_OPTIONS := $(CFLAGS) $(MY_CFLAG)
CFLAGS =    $(CFLAGS_OPTIONS) $(INCLUDES) $(MODULE_CFLAGS)

.PHONY: all server client microservices clean
.PHONY: server_ldflags client_ldflags server_cflags client_cflags
.PHONY: print_cflags

# Build everytying
all:    microservices
    @true

# List module's objects and needed libs for inclusion in clients
client_ldflags:
    @true

# List module's includes for inclusion in the clients
client_cflags:
    @true

# List module's objects and needed libs for inclusion in the server
server_ldflags:
    @echo $(OBJECTS) $(LIBS)

# List module's includes for inclusion in the server
server_cflags:
    @echo $(INCLUDE_FLAGS)

# Build microservices
microservices:    print_cflags $(OBJECTS)

# Build client additions
client:
    @true

# Build server additions
server:
    @true

# Build rules
rules:
    @true

# Clean
clean:
    @echo "Clean image module..."
    rm -rf $(MSObjDir)/*.o


# Show compile flags
print_cflags:
    @echo "Compile flags:"
    @echo "    $(CFLAGS_OPTIONS)"

# Compile targets
#
$(OBJECTS): $(MSObjDir)/%.o: $(MSSrcDir)/%.c $(DEPEND)
    @echo "Compile image module `basename $@`..."
    @$(CC) -c $(CFLAGS) -o $@ $<

info.txt

Name:        HCC
Brief:        HCC Test microservice
Description:    HCC Test microservice.
Dependencies:
Enabled:    yes
Creator:    Ashu Guru
Created:    December 2011
License:    BSD

In the next step I will define the micro-service header and micro-service table files so that the iRODS can be configured with the new micro-service. This is done in the folder microservices/include. In this example there is no header for this code so I have left the header file blank; in the micro-service table file I have the entry for the table definition. The specifics to note below are that the first argument is the label of the micro-service, the second argument is the count of input arguments (do not count the ruleExecInfo _t argument) of the msi interface and the third argument is the name of the msi interface function.

File microservices/include/microservices.table

{ "msiWritemessage",2,(funcPtr) msiWritemessage },

Following is the directory tree structure for the HCC module that I have so far:

bash-4.1$ pwd

/opt/iRODS/modules
bash-4.1$ tree HCC
HCC
├── clients
├── info.txt
├── lib
├── Makefile
├── microservices
│   ├── include
│   │   ├── microservices.header
│   │   ├── microservices.table
│   ├── obj
│   └── src
│       ├── test.c
├── rules
└── servers

Next I will make an entry for enabling the new module (this micro-service), this is done in the file ~irods/config/config.mk so that the iRODS Makefile can include the new micro-service for build. To do this simply add the module folder name (in my case HCC) to the variable MODULES.

Compile and test

cd ~irods/modules/<YOURMODULENAME>
make

The above commands should result in creation of an object file in the micro-service/obj folder. I am going to test the micro-service manually first, to accomplish this I will create a client side rule file in the folder ~irods/ clients/icommands/test/rules. I have named the file aguru.ir and following are the contents of the file:

aguruTest||msiWritemessage(*A,*B)|nop
*A=helloworld%*B=testing

The first line in file is the rules definition and the second line are the input parameters. To test the micro-service I will invoke the micro-service which will then write a message to the system log (see figure below).

Recompile iRODS

Before this step I must make the entries for the headers and the msi table in the iRODS main micro-service action table (i.e. file ~irods/server/re/include/reAction.h). This should be done using the following commands:

rm server/re/include/reAction.h
make reaction

However, I had to manually add the code segment below to the file server/re/include/reAction.h file to accomplish that:

int msiWritemessage(msParam_t *mParg1, msParam_t *mParg2,  ruleExecInfo_t *rei);

Finally, recompile iRODS

cd ~irods
make test_flags
make modules
./irodsctl stop
make clean
make
./irodsctl start
./irodsctl status

Register Micro-service and Test

In this step we define a rule that will trigger the micro-service when a new data object is uploaded to iRODS. Open the file ~irods/server/config/reConfigs/core.re and add the following line the Test Rules section.

acPostProcForPut {msiWritemessage("HelloWorld","String 2"); }

That is it… if now I put (iput) any file into iRODS a message is added to the /var/log/messages file on the iRODS server. Please note that the above rule is not filtering a particular occurrence but is a catchall rule that applies to all put events.

References:
https://www.irods.org/
http://www.wrg.york.ac.uk/iread/compiling-and-running-irods-with-micros-services
http://technical.bestgrid.org/index.php/IRODS_deployment_plan

An Open Science Grid Work Log: CAMx on OSG with UNC Environmental Science and Engineering

Fri, 16 Dec 2011 16:22:36 +0000

Dr. William Vizuete’s team at the UNC department of Environmental Science and Engineering evaluates and uses a number of air quality models which are, in turn used by regulatory agencies to understand air quality issues. From Dr. Vizuete’s research page:

“Using high performance computers and three dimensional models to simulate the atmosphere, I am working to improve our understanding of the formation of atmospheric air pollution. These computer models improve our understanding of the extremely complex chemical and physical processes that occur in the atmosphere. A better understanding of the atmosphere gives us the knowledge to improve the tools and methods that policy makers use to make effective control strategies to clean the air above our dirtiest cities.”

Two models, CAMx by ENVIRON and CMAQ are heavily used.

We chose to start with CAMx because word on the street is that it’s easier to build than CMAQ. Well, it’s all relative.

Here’s the new automated build process for CAMx. It uses the Intel fortran compiler version 11.1 and builds an x86_64 binary that’s statically linked.

Also built in that process is a program called avgdif from the ENVIRON website. It is a standalone fortran program that compares the average difference between two runs. This allows us to validate execution of the CAMx test cases. It is slightly modified in order to have enough memory for this model (maxx and maxy are set to 120 near the top of the file). Finally, there’s a custom makefile for it to work with the Intel compiler.

CAMx has for several versions been OpenMP enabled. This is well suited for the OSG’s HTPC node concurrency model since the job can be configured to take advantage of all the CPUs on a node but does not require the additional complexity of a statically linked MPI launcher.

The new job is being tested now. A GlideinWMS submit script has been set up to launch the job to various HTPC compute venues. The job itself will:

Fetch binaries from the RENCI continuous integration environment
Fetch test cases from ENVIRON’s website
Execute the test
Execute avgdif on the outputs
Return the results of avgdif

Once this is done, the environmental engineering community will be able to give it a try once they’ve joined the Engage VO.

An Open Science Grid Work Log: Pegasus WMS and NERSC Dirac

Fri, 16 Dec 2011 16:19:51 +0000

The Engage VO’s recently begun working with Duke Chemistry on a protein folding simulation that’s taking us to interesting places.

In particular, deep inside the National Energy Research Scientific Center – NERSC and specifically, to their 50 node General Purpose Graphics Processor (GPGPU) cluster, Dirac. You know you’re having a good time when the cluster you’re using is at a cutting edge DOE research lab and they consider it experimental.

As noted in previous posts, performance for many molecular dynamics applications is greatly improved on GPUs. We’re using NAMD which provides pre-built binaries for GPUs. The binaries ship with dynamic libraries for CUDA, its only external dependency. As such, it’s practical for use on the Open Science Grid since it can be relocated to new clusters trivially.

NAMD, like Amber and many scientific codes, especially ones that simulate natural phenomena, supports restart. That is, output from one run can be used as input to another to continue the calculation. This is important since ultimately we’re modeling a biological system that doesn’t have a particular endpoint – it just keeps going. It also has implications for the shape of our workflow. In this case we have one computation activity – execute NAMD – which we’d like to repeat, feeding the input from one run into the next.

We used Pegasus WMS as the wrokflow execution engine. There are lots of reasons for this but here are some of the basics:

It’s state of the art for creating a useful layer of workflow abstraction over the OSG’s runtime infrastructure.
The support list is incredibly helpful.
It allows me to use my personal OSG/DOE X.509 user credential which is required by NERSC policy.

Each compute job stages in

The GPU accelerated version of NAMD
The input data files
A statically linked version of MPICH2
The ouptut file from the previous run if one exists

Output is written from this job, archived and transferred back to the submit host.

One of the interesting aspects of this is how to generate workflows with these characteristics. Each compute job is somewhat logically self contained. We’d like to be able to repeat them in chains and otherwise organize them hierarchically – for example – to do parallel simulations, each of which consists of a chain of compute jobs. More on this in a later post.

Another challenge, from a workflow maintenance and configuration perspective is how to get very hardware specific in our job specifications while keeping the workflow abstract. Here’s the job specification for running NAMD on Dirac’s Tesla GPUs.

<!– part 3: Definition of all jobs/dags/daxes (at least one) –>
<job id=”1n3″ namespace=”namd-flow.0″ name=”namd”>
<argument>–model=ternarycomplex119819 –config=ternarycomplex_popcwimineq-05 –slice=0 –namdType=CUDA –runLength=100</argument>
<profile namespace=”globus” key=”queue”>dirac_reg</profile>
<profile namespace=”globus” key=”xcount”>8:tesla</profile>
<profile namespace=”globus” key=”maxWallTime”>240</profile>
<profile namespace=”globus” key=”jobType”>mpi</profile>
<profile namespace=”globus” key=”host_xcount”>1</profile>
<profile namespace=”condor” key=”x509userproxy”>/home/scox/dev/grayson/var/proxy/x509_proxy_scox</profile>
<uses name=”mpich2-static-1.1.1p1.tar.gz” link=”input”/>
<uses name=”cpuinfo” link=”input”/>
<uses name=”beratan-0.tar.gz” link=”input”/>
<uses name=”namd.tar.gz” link=”input”/>
<uses name=”out-0.tar.gz” link=”output”/>
</job>

Again, more on how we’re managing this in a later post.

OSG Technology Area Rumblings: How to create openstack controller

noreply@blogger.com (TomW) — Thu, 15 Dec 2011 12:06:20 +0000

As before, the "official" instructions on which our procedure is based are here:

http://docs.openstack.org/cactus/openstack-compute/admin/content/installing-openstack-compute-on-rhel6.html

First setup the repository:

wget http://yum.griddynamics.net/yum/cactus/openstack/openstack-repo-2011.2-1.el6.noarch.rpm
rpm -ivh openstack-repo-2011.2-1.el6.noarch.rpm

Then install openstack and dependencies

yum install libvirt
chkconfig libvirtd on
/etc/init.d/libvirtd start
yum install euca2ools openstack
nova-{api,compute,network,objectstore,scheduler,volume} openstack-nova-cc-config openstack-glance

Start services:

service mysqld start
chkconfig mysqld on
service rabbitmq-server start
chkconfig rabbitmq-server on

Setup database authorisations. First set up root password:

mysqladmin -uroot password

Now, to automate the procedure create an executable shell script

openstack-db-setup.sh

with the following content (fill the relevant user name and password fields as well as the IP's):

#!/bin/bash

DB_NAME=nova
DB_USER=
DB_PASS=
PWD=

#CC_HOST="A.B.C.D" # IPv4 address
CC_HOST="130.199.148.53" # IPv4 address, fill your own
#HOSTS='node1 node2 node3' # compute nodes list
HOSTS='130.199.148.54' # compute nodes list, fill your own

mysqladmin -uroot -p$PWD -f drop nova
mysqladmin -uroot -p$PWD create nova

for h in $HOSTS localhost; do
echo "GRANT ALL PRIVILEGES ON $DB_NAME.* TO '$DB_USER'@'$h' IDENTIFIED BY '$DB_PASS';" | mysql -uroot -p$DB_PASS mysql
done
echo "GRANT ALL PRIVILEGES ON $DB_NAME.* TO $DB_USER IDENTIFIED BY '$DB_PASS';" | mysql -uroot -p$DB_PASS mysql
echo "GRANT ALL PRIVILEGES ON $DB_NAME.* TO root IDENTIFIED BY '$DB_PASS';" | mysql -uroot -p$DB_PASS mysql

And now execute this script:

./openstack-db-setup.sh

Create db schema

nova-manage db sync

Now comes point which is not in the "official" instructions. The installation will not work unless you patch your python:

patch -p0 < rhel6-nova-network-patch.diff

Create logical volumes:

lvcreate -L 1G --name test nova-volumes

For your convenience create an openstack startup shell script openstack-init.sh

Here is its content:

#!/bin/bash
for n in api compute network objectstore scheduler volume; do
service openstack-nova-$n $@;
done
service openstack-glance-api $@

And finally we are ready to start openstack:

openstack-init.sh start

With fingers crossed you should get

Starting OpenStack Nova API Server: [ OK ]
Starting OpenStack Nova Compute Worker: [ OK ]
Starting OpenStack Nova Network Controller: [ OK ]
Starting OpenStack Nova Object Storage: [ OK ]
Starting OpenStack Nova Scheduler: [ OK ]
Starting OpenStack Nova Volume Worker: [ OK ]
Starting OpenStack Glance API Server: [ OK ]

Now we need to configure and customize the installation which is another story for another day...

./openstack-init.sh start

If everything goes fine

Starting OpenStack Nova API Server: [ OK ]
Starting OpenStack Nova Compute Worker: [ OK ]
Starting OpenStack Nova Network Controller: [ OK ]
Starting OpenStack Nova Object Storage: [ OK ]
Starting OpenStack Nova Scheduler: [ OK ]
Starting OpenStack Nova Volume Worker: [ OK ]
Starting OpenStack Glance API Server: [ OK ]

OSG Technology Area Rumblings: How to create openstack worker node

noreply@blogger.com (TomW) — Thu, 15 Dec 2011 11:46:58 +0000

The "official" instructions how to install openstack components are located here:

http://docs.openstack.org/cactus/openstack-compute/admin/content/installing-openstack-compute-on-rhel6.html

Unfortunately they are not very clear and miss some key points. Below is summary of our installation procedure.

First of all, let us install worker node.

wget http://yum.griddynamics.net/yum/cactus/openstack/openstack-repo-2011.2-1.el6.noarch.rpm
rpm -ivh openstack-repo-2011.2-1.el6.noarch.rpm
yum install libvirt
chkconfig libvirtd on
/etc/init.d/libvirtd start
yum install openstack-nova-compute openstack-nova-compute-config
service openstack-nova-compute start

If everything goes fine you should see

Starting OpenStack Nova Compute Worker: [ OK ]

Spinning: New toy: newpgid

Tue, 13 Dec 2011 11:14:41 +0000

Useful with cpusoak and memsoak,

newpgid.c

#include <unistd.h>

int
main(int argc, char *argv[])
{
  setpgid(0, 0);

  execvp(argv[1], &(argv[1]));

  return 1;
}

When you want to start a new process in its own process group for easy killing.

If you have coreutils 7.0+, you can take advantage of timeout, which happens to setpgid.

OSG Technology Area Rumblings: Network Accounting for Condor

noreply@blogger.com (Brian Bockelman) — Thu, 08 Dec 2011 17:03:39 +0000

It's been a long time since the August post describing how to set up manual network accounting for a process. We now have a solution integrated into Condor and available on github. It requires a bit to understand how it works, so I've put together a series of diagrams to illustrate it.

First, we start off with the lowly condor_starter on any worker node with an network connection (to simplify things, I didn't draw the other condor processes involved):

By default, all processes on the node are in the same network namespace (labelled the "System Network Namespace" in this diagram). We denote the network interface with a box, and assume it has address 192.168.0.1.

Next, the starter will create a pair of virtual ethernet devices. We will refer to them as pipe devices, because any byte written into one will come out of the other - just how a venerable Unix pipe works:

By default, the network pipes are in a down state and have no IP address associated with them. Not very useful! At this point, we have some decisions to make: how should the network pipe device be presented to the network? Should it be networked at layer 3, using NAT to route packets? Or should we bridge it at layer 2, allowing the device to have a public IP address?

Really, it's up to the site, but we assume most sites will want to take the NAT approach: the public IP address might seem useful, but would require a public IP for each job. To allow customization, all the routing is done by a helper script, but provide a default implementation for NAT. The script:

Takes two arguments, a unique "job identifier" and the name of the network pipe device.
Is responsible for setting up any routing required for the device.
Must create an iptables chain using the same name of the "job identifier".

Each rule in the chain will record the number of bytes matched; at the end of the job, these will be reported in the job ClassAd using an attribute name identical to the comment on the rule.

On stdout, returns the IP address the internal network pipe should use.

Additionally, the Condor provides a cleanup script does the inverse of the setup script. The result looks something like this:

Next, the starter forks a separate process in a new network namespace using the clone() call with the CLONE_NEWNET flag. Notice that, by default, no network devices are accessible in the new namespace:

Next, the external starter will pass one side of the pipe to the other namespace; the internal stater will do some minimal configuration of the device (default route, IP address, set the device to the "up" status):

Finally, the starter exec's to the job. Whenever the job does any network operations, the bytes are routed via the internal network pipe, come out the external network pipe, and then are NAT'd to the physical network device before exiting the machine.

As mentioned, the whole point of the exercise is to do network accounting. Since all packets go through one device, Condor can read out all the activity via iptables. The "helper script" above will create a unique chain per job. This allows some level of flexibility; for example, the chain below allows us to distinguish between on-campus and off-campus packets:

Chain JOB_12345 (2 references)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 ACCEPT     all  --  veth0  em1     anywhere             129.93.0.0/16        /* OutgoingInternal */
    0     0 ACCEPT     all  --  veth0  em1     anywhere            !129.93.0.0/16        /* OutgoingExternal */
    0     0 ACCEPT     all  --  em1    veth0   129.93.0.0/16        anywhere             state RELATED,ESTABLISHED /* IncomingInternal */
    0     0 ACCEPT     all  --  em1    veth0  !129.93.0.0/16        anywhere             state RELATED,ESTABLISHED /* IncomingExternal */
    0     0 REJECT     all  --  any    any     anywhere             anywhere             reject-with icmp-port-unreachable

Thus, the resulting ClassAd history from this job will have an attribute for NetworkOutgoingInternal, NetworkOutgoingExternal, NetworkIncomingInternal, and NetworkIncomingInternal. We have an updated Condor Gratia probe that looks for Network* attributes and reports them appropriately to the accounting database.

Thus, we have byte-level network, allowing us to answer the age-old question of "how much would a CMS T2 cost on Amazon EC2?". Or perhaps we could answer "how much is a currently running job going to cost me?" Matt has pointed out the network setup callout could be used to implement security zones, isolating (or QoS'ing) jobs of certain users at the network level. There are quite a few possibilities!

We'll definitely be returning to this work mid-2012 when the local T2 is based on SL6, and this patch can be put into production. There will be some further engagement with the Condor team to see if they're interested in taking the patch. The Gratia probe work to manage network information will be interesting upstream too. Finally, I encourage interested readers to take a look at the github branch. The patch itself is a tour-de-force of several dark corners of Linux systems programming (involves using clone, synchronization between processes with pipes, sending messages to the kernel via netlink to configure the routing, and reading out iptables configurations using C). It was very rewarding to implement!

Spinning: Service as a Job: Memcached

Mon, 05 Dec 2011 12:45:59 +0000

Running services such as Tomcat or Qpidd show how to schedule and manage a service’s life-cycle via Condor. It is also possible to gather and centralize statistics about a service as it runs. Here is an example of how with memcached.

As with tomcat and qpidd, there is a control script and a job description.

New in the control script for memcached will be a loop to monitor and chirp back statistic information.

memcached.sh

#!/bin/sh

# condor_chirp in /usr/libexec/condor
export PATH=$PATH:/usr/libexec/condor

PORT_FILE=$TMP/.ports

# When we get SIGTERM, which Condor will send when
# we are kicked, kill off memcached.
function term {
   rm -f $PORT_FILE
   kill %1
}

# Spawn memcached, and make sure we can shut it down cleanly.
trap term SIGTERM
# memcached will write port information to env(MEMCACHED_PORT_FILENAME)
env MEMCACHED_PORT_FILENAME=$PORT_FILE memcached -p -1 "$@" &

# We might have to wait for the port
while [ ! -s $PORT_FILE ]; do sleep 1; done

# The port file's format is:
#  TCP INET: 56697
#  TCP INET6: 47318
#  UDP INET: 34453
#  UDP INET6: 54891
sed -i -e 's/ /_/' -e 's/\(.*\): \(.*\)/\1=\2/' $PORT_FILE
source $PORT_FILE
rm -f $PORT_FILE

# Record the port number where everyone can see it
condor_chirp set_job_attr MemcachedEndpoint \"$HOSTNAME:$TCP_INET\"
condor_chirp set_job_attr TCP_INET $TCP_INET
condor_chirp set_job_attr TCP_INET6 $TCP_INET6
condor_chirp set_job_attr UDP_INET $UDP_INET
condor_chirp set_job_attr UDP_INET6 $UDP_INET6

# While memcached is running, collect and report back stats
while kill -0 %1; do
   # Collect stats and chirp them back into the job ad
   echo stats | nc localhost $TCP_INET | \
    grep -v -e END -e version | tr '\r' '\0' | \
     awk '{print "stat_"$2,$3}' | \
      while read -r stat; do
         condor_chirp set_job_attr $stat
      done
   sleep 30
done

A refresher about chirp. Jobs are stored in condor_schedd processes. They are described using the ClassAd language, extensible name value pairs. chirp is a tool a job can use while it runs to modify its classad stored in the schedd.

The job description, passed to condor_submit, is vanilla except for how arguments are passed to memcached.sh. The dollardollar use, see man condor_submit, allows memcached to use as much memory as is available on the slot where it gets scheduled. Slots may have different amounts of Memory available.

memcached.job

cmd = memcached.sh
args = -m $$(Memory)

log = memcached.log

kill_sig = SIGTERM

# Want chirp functionality
+WantIOProxy = TRUE

should_transfer_files = if_needed
when_to_transfer_output = on_exit

queue

An example, note that the set of memcached servers to use is generated from condor_q,

$ condor_submit -a "queue 4" memcached.job
Submitting job(s)....
4 job(s) submitted to cluster 80.

$ condor_q -format "%s\t" MemcachedEndpoint -format "total_items: %d\t" stat_total_items -format "memory: %d/" stat_bytes -format "%d\n" stat_limit_maxbytes
eeyore.local:50608	total_items: 0	memory: 0/985661440
eeyore.local:47766	total_items: 0	memory: 0/985661440
eeyore.local:39130	total_items: 0	memory: 0/985661440
eeyore.local:57410	total_items: 0	memory: 0/985661440

$ SERVERS=$(condor_q -format "%s," MemcachedEndpoint); for word in $(cat words); do echo $word > $word; memcp --servers=$SERVERS $word; \rm $word; done &
[1] 959

$ condor_q -format "%s\t" MemcachedEndpoint -format "total_items: %d\t" stat_total_items -format "memory: %d/" stat_bytes -format "%d\n" stat_limit_maxbytes
eeyore.local:50608	total_items: 480	memory: 47740/985661440
eeyore.local:47766	total_items: 446	memory: 44284/985661440
eeyore.local:39130	total_items: 504	memory: 50140/985661440
eeyore.local:57410	total_items: 490	memory: 48632/985661440

$ condor_q -format "%s\t" MemcachedEndpoint -format "total_items: %d\t" stat_total_items -format "memory: %d/" stat_bytes -format "%d\n" stat_limit_maxbytes
eeyore.local:50608	total_items: 1926	memory: 191264/985661440
eeyore.local:47766	total_items: 1980	memory: 196624/985661440
eeyore.local:39130	total_items: 2059	memory: 204847/985661440
eeyore.local:57410	total_items: 2053	memory: 203885/985661440

$ condor_q -format "%s\t" MemcachedEndpoint -format "total_items: %d\t" stat_total_items -format "memory: %d/" stat_bytes -format "%d\n" stat_limit_maxbytes
eeyore.local:50608	total_items: 3408	memory: 338522/985661440
eeyore.local:47766	total_items: 3542	memory: 351784/985661440
eeyore.local:39130	total_items: 3666	memory: 364552/985661440
eeyore.local:57410	total_items: 3600	memory: 357546/985661440

[1]  + done       for word in $(cat words); do; echo $word > $word; memcp --servers=$SERVERS ;

Enjoy.

OSG Technology Area Rumblings: Details on glexec improvements

noreply@blogger.com (Brian Bockelman) — Thu, 01 Dec 2011 17:46:58 +0000

My last blog post gave a quick overview of why glexec exists, what issues folks run into, and what we did to improve it. Let's go into some details.

How Condor Update Works
The lcmaps-plugin-condor-update package contains the modules necessary to advertise the payload certificate of the last glexec invocation in the pilot's ClassAd. The concept is simple - the implementation is a bit tricky.

For a long time, Condor has had a command-line tool called condor_advertise for awhile; it allows an admin to hand-advertise updates to ads in the collector. Unfortunately, that's not quite what we need here: we want to update the job ad in the schedd, while condor_advertise typically updates the machine ad in the collector. Close, but no cigar.

There's a lesser-known utility called condor_chirp that we can use. Typically, condor_chirp is used to do I/O between the schedd and the starter (for example, you can pull/push files on demand in the middle of the job), but it can also update the job's ad in the schedd. The syntax is simple:

condor_chirp ATTR_NAME ATTR_VAL

(look at the clever things Matt does with condor_chirp). As condor_chirp allows additional access to the schedd, the user must explicitly request it in the job ad. If you want to try it out, you must add the following line into your submit file:

+WantIOProxy=TRUE

To work, chirp must know how to contact the starter and have access to the "magic cookie"; these are located inside the $_CONDOR_SCRATCH_DIR, as set by Condor in the initial batch process. As the glexec plugin runs as root (glexec must be setuid root to launch a process as a different UID), we must guard against being fooled by the invoking user.
Accordingly, the plugin uses /proc to read the parentage of the process tree until it finds a process owned by root. If this is not init, it is assumed the process is the condor_starter, and the job's $_CONDOR_SCRATCH_DIR can be deduced from the $CWD and the PID of the starter. Since we only rely on information from root-owned processes, we can be fairly sure this is the correct scratch directory. As a further safeguard, before invoking condor_chirp, the plugin drops privilege to that of the invoking user. Along with the other security guarantees provided by glexec, we have confidence that we are reading the correct chirp configuration and are not allowing the invoker to increase its privileges.

Once we know how to invoke condor_chirp, the rest of the process is all downhill. glexec internally knows the payload's DN, the payload Unix user, and does the equivalent of the following:

condor_chirp set_job_attr glexec_user "hcc"
condor_chirp set_job_attr glexec_x509userproxysubject "/DC=org/DC=cilogon/C=US/O=University of Nebraska-Lincoln/CN=Brian Bockelman A621"
condor_chirp set_job_attr glexec_time 1322761868

condor_chirp writes the data into the starter, which then updates the shadow, then the schedd (some of the gory details are covered in the Condor wiki).

The diagram below illustrates the data flow:

Putting this into Play
If you really want to get messy, you can check out the source code from Subversion at:

svn://t2.unl.edu/brian/lcmaps-plugins-condor-update

(web view)

The current version of the plugin is 0.0.2. It's available in Koji, or via yum in the osg-development repository:

yum install --enablerepo=osg-development lcmaps-plugins-condor-update

(you must already have the osg-release RPM installed and glexec otherwise configured).

After installing it, you need to update the /etc/lcmaps.db configuration file on the worker node to invoke the condor-update module. In the top half, I add:

condor_updates = "lcmaps_condor_update.mod"

Then, I add condor-update to the glexec policy:

glexec:

verifyproxy -> gumsclient
gumsclient -> condor_updates
condor_updates -> tracking

Note we use the "tracking" module locally; most sites will use the "glexec-tracking" module. Pick the appropriate one.

Finally, you need to turn on the I/O proxy in the Condor submit file. We do this by editing condor.pm (for RPMs, located in /usr/lib/perl5/vendor_perl/5.8.8/Globus/GRAM/JobManager/condor.pm). We add the following line into the submit routine, right before queue is added to the script file:

print SCRIPT_FILE "+WantIOProxy=TRUE\n";

All new incoming jobs will get this attribute; any glexec invocations they do will be reflected at the CE!

GUMS and Worker Node Certificates

To map a certificate to a Unix user, glexec calls out to the GUMS server using XACML with a grid-interoperable profile. In the XACML callout, GUMS is given the payload's DN and VOMS attributes. The same library (LCMAPS/SCAS-client) and protocol can also make callouts directly to SCAS, more commonly used in Europe.

GUMS is a powerful and flexible authorization tool; one feature is that it allows different mappings based on the originating hostname. For example, if desired, my certificate could map to user hcc at red.unl.edu but map to cmsprod at ff-grid.unl.edu. To prevent "just anyone" from probing the GUMS server, GUMS requires the client to present X509 a certificate (in this case, the hostcert); it takes the hostname from the client's certificate.

This has the unfortunate side-effect of requiring a host certificate on every node that invokes GUMS; OK for the CE (100 in the OSG), but not for glexec on the worker nodes (thousands on the OSG).

When glexec is invoked in EGI, SCAS is invoked using the pilot certificate for HTTPS and information about the payload certificate in the XACML callout; this requires no worker node host certificate.

To replicate how glexec works in EGI, we had to develop a small patch to GUMS. When the pilot certificate is used for authentication, the pilot's DN is recorded to the logs (so we know who is invoking GUMS), but the host name is self-reported in the XACML callout. As the authentication is still performed, we believe this relaxing of the security model is acceptable.

A patched, working version of GUMS can be found in Koji and is available in the osg-development repository. It will still be a few months before the RPM-based GUMS install is fully documented and released, however.

Once installed, two changes need to be made at the server:

Do all hostname mappings based on "DN" in the web interface, not the "CN".
Any group of users (for example, /cms/Role=pilot) that want to invoke GUMS must have "read all" access, not just "read self".

Further, /etc/lcmaps.db needs to be changed to remove the following lines from the gumsclient module:

"-cert   /etc/grid-security/hostcert.pem"
"-key    /etc/grid-security/hostkey.pem"
"--cert-owner root"

This will be all automated going forward - but all should help remove some of the pain in deploying glexec!

Spinning: Custom resource attributes: Facter

Tue, 29 Nov 2011 11:54:36 +0000

Condor provides a large set of attributes, facts, about resources for scheduling and querying, but it does not provide everything possible. Instead, there is a mechanism to extend the set. Previously, we added FreeMemoryMB. The set can also be extend with information from Facter.

Facter provides an extensible set of facts about a system. To include facter facts we need a means to translate them into attributes and add to Startd configuration.

$ facter
...
architecture => x86_64
domain => local
facterversion => 1.5.9
hardwareisa => x86_64
hardwaremodel => x86_64
physicalprocessorcount => 1
processor0 => Intel(R) Core(TM) i7 CPU       M 620  @ 2.67GHz
selinux => true
selinux_config_mode => enforcing
swapfree => 3.98 GB
swapsize => 4.00 GB
...

The facts are of the form name => value, not very far off from ClassAd attributes. A simple script to convert all the facts into attribute with string values is,

/usr/libexec/condor/facter.sh

#!/bin/sh
type facter &> /dev/null || exit 1
facter | sed 's/\([^ ]*\) => \(.*\)/facter_\1 = "\2"/'

$ facter.sh
...
facter_architecture = "x86_64"
facter_domain = "local"
facter_facterversion = "1.5.9"
facter_hardwareisa = "x86_64"
facter_hardwaremodel = "x86_64"
facter_physicalprocessorcount = "1"
facter_processor0 = "Intel(R) Core(TM) i7 CPU       M 620  @ 2.67GHz"
facter_selinux = "true"
facter_selinux_config_mode = "enforcing"
facter_swapfree = "3.98 GB"
facter_swapsize = "4.00 GB"
...

And the configuration, simply dropped into /etc/condor/config.d,

/etc/condor/config.d/49facter.config

FACTER = /usr/libexec/condor/facter.sh
STARTD_CRON_JOBLIST = $(STARTD_CRON_JOBLIST) FACTER
STARTD_CRON_FACTER_EXECUTABLE = $(FACTER)
STARTD_CRON_FACTER_PERIOD = 300

A condor_reconfig and the facter facts will be available,

$ condor_status -long | grep ^facter
...
facter_architecture = "x86_64"
facter_facterversion = "1.5.9"
facter_domain = "local"
facter_swapfree = "3.98 GB"
facter_selinux = "true"
facter_hardwaremodel = "x86_64"
facter_selinux_config_mode = "enforcing"
facter_processor0 = "Intel(R) Core(TM) i7 CPU       M 620  @ 2.67GHz"
facter_selinux_mode = "targeted"
facter_hardwareisa = "x86_64"
facter_swapsize = "4.00 GB"
facter_physicalprocessorcount = "1"
...

For scheduling, just use the facter information in job requierments, e.g. requirements = facter_selinux == "true".

Or, query your pool to see what resources are not running selinux,

$ condor_status -const 'facter_selinux == "false"'
Name               OpSys      Arch   State     Activity LoadAv Mem   ActvtyTime
eeyore.local       LINUX      X86_64 Unclaimed Idle     0.030  3760  0+00:12:31
                     Machines Owner Claimed Unclaimed Matched Preempting
        X86_64/LINUX        1     0       0         1       0          0
               Total        1     0       0         1       0          0

Oops.

Inside OSG Ops.: GOC holiday schedule

noreply@blogger.com (scott) — Mon, 21 Nov 2011 06:06:21 +0000

From 24/Nov through 27/Nov the GOC will be operating on a Holiday
schedule. Staff will be available to respond to emergencies but
routine operations will resume at start of business Monday 28/Nov.

The GOC wishes its users and OSG staff a happy and satisfying
Thanksgiving Holiday.

OSG Technology Area Rumblings: Improving the glexec-enabled life

noreply@blogger.com (Brian Bockelman) — Fri, 11 Nov 2011 08:32:28 +0000

Pilot-based workflow management systems have had a dramatic transformation of how we view the grid today. Instead of queueing a job (the "payload") in a workflow onto a site on a grid, these systems send an "empty" job that starts up, then downloads and starts the payload from from a central endpoint. In CS terms, it switches from a model of "work delegation" to "resource allocation". By allocating the resource (i.e., starting the pilot job) prior to delegating work, users no longer have to know the vagaries/failure modes of direct grid submission and don't have to pay the price of sending their payloads to a busy site!

In short, pilot jobs make the grid much better.

However, like most concepts, pilot jobs are a trade-off: they make life easier for users, but harder for security folks and sysadmins. Pilots are sent using one certificate, but payloads are run under a different identity. If the payload job wants to act on behalf of the user, it needs to bring the user's grid credentials to the worker node. [Side note: this is actually an interesting assumption. The PanDA pilot system, heavily utilized by ATLAS, does not bring credentials to the worker node. This simplifies this problem, but opens up a different set of concerns.] If both pilot and payload are run as the same Unix user, the payload user can easily access the credentials (including the pilot credentials), executables, and output data of other running payloads.

The program glexec is a "simple" idea to solve this problem: given a set of grid credentials, launch a process under corresponding the Unix account at the site. For example, with credentials from the HCC VO:

[bbockelm@brian-test ~]$ whoami
bbockelm
[bbockelm@brian-test ~]$ GLEXEC_CLIENT_CERT=/tmp/x509up_u1221 /usr/sbin/glexec 
/usr/bin/whoami
hcc

(You'll notice the invocation is not as simple as typing "glexec whoami"; it's not exactly designed for end-user invocation). To achieve the user switching, glexec has to be setuid root. Setuid binaries must be examined under a security microscope, which have unfortunately led to a slow adoption of glexec.

The idea is that pilot jobs would wrap the payload with a call to "glexec", separating the payload from the pilot and other payloads. From there, it goes horribly wrong. Not wrong really - but rather things get sticky.

Since the pilot and payload are both low-privileged users, the pilot doesn't have permission to clean up or kill the payload. It must again use glexec to send signals and delete sandboxes. The several invocations are easy to screw up (and place load on the authorization system!). There are tricky error conditions - if authorization breaks in the middle of the job, how does the pilot clean up the payload?

As the payload is a full-fledged Linux process, it can create other processes, daemonize, escape from the batch system, etc. As previously discussed, the batch system - with root access - typically does a poor job tracking processes. The pilot will be hopeless unless we provide some assistance.

Glexec imposes an integration difficulty at some sites. There are popular cron scripts that kill process belonging to users on a node that aren't currently running batch system jobs. So, if the pilot maps to "cms" and the payload maps to "cmsuser", the batch system only knows about "cms", and the cronjob will kill all processes belonging to "cmsuser". We lost quite a few jobs at some sites before we figured this out!

Site admins manage the cluster via the batch system. Since the payload is invisible to the batch system, we're unable to kill jobs from a user with batch system tools (condor_rm, qdel). In fact, if we get an email from a user asking for help understanding their jobs, we can't even easily find where the job is running! Site admins have to ssh into each worker node and examine the running jobs; a process that is simply medieval.

Finally, on the OSG, invoking the authorization system requires host certificate credentials. This is not a problem when host certs are needed for a handful of CEs at the site, but explodes when glexec is run on each worker node. This is a piece of unique state on the worker nodes for sites to manage, adding to the glexec headache.

We're the Government. We're here to help.

The OSG Technology group has decided to tackle the three biggest site-admin usability issues in glexec:

Batch system integration: The Condor batch system provides the ability for running jobs to update the submit node with arbitrary status. We have developed a plugin that updates the job's ClassAd with the payload's DN whenever glexec is invoked.
Process tracking: There is an existing glexec plugin to do process tracking. However, this requires a admin to set up secondary GID ranges (an administration headache) and suffers the previously-documented process tracking issues. We will port the ProcPolice daemon over to the glexec plugin framework.
Worker node certificates: We propose to fix this via improvements to GUMS, allowing the mappings to be performed based on the presence of "Role=pilot" VOMS extension in the pilot certificate.

The plugins in (1) and (2) have been prototyped, and are available in the osg-development repository as "lcmaps-plugins-condor-update" and "lcmaps-plugins-process-tracking", respectively. The third item is currently cooking.

The "lcmaps-plugins-condor-update" is especially useful, as it's a brand-new capability as opposed to an improvement. It advertises three attributes in the job's ClassAd:

glexec_x509userproxysubject: The DN of the payload user.
glexec_user: The Unix username for the payload.
glexec_time: The Unix time when glexec was invoked.

We can then use it to filter and locate jobs. For example, if a user named Ian complains his jobs are running slowly, we could locate a few with the following command:

[bbockelm@t3-sl5 ~]$ condor_q -g -const 'regexp("Ian", glexec_x509userproxysubject)' -format '%s ' ClusterId -format '%s\n' RemoteHost | head
868341 slot6@red-d11n10.red.hcc.unl.edu
868343 slot7@node238.red.hcc.unl.edu
868358 slot6@red-d11n9.red.hcc.unl.edu
868366 slot2@node239.red.hcc.unl.edu
868373 slot3@node119.red.hcc.unl.edu
868741 slot8@red-d9n6.red.hcc.unl.edu
868770 slot3@red-d9n8.red.hcc.unl.edu
868819 slot5@node109.red.hcc.unl.edu
868820 slot4@node246.red.hcc.unl.edu
868849 slot2@red-d11n6.red.hcc.unl.edu

Slick!

Spinning: Getting started: Condor and EC2 EC2 execute node

Thu, 10 Nov 2011 05:36:39 +0000

We have been over starting and managing instances from Condor, using condor_ec2_q to help, and importing existing instances. Here we will cover extending an existing pool using execute nodes run from EC2 instances. We will start with an existing pool, create an EC2 instance, configure the instance to run condor, authorize the instance to join the existing pool, and run a job.

Let us pretend that the node running your existing pool’s condor_collector and condor_schedd is called condor.condorproject.org.

These instructions will require bi-directional connectivity between condor.condorproject.org and your EC2 instance. condor.condorproject.org must be connected to the internet with a publically routable address. Also, ports must be open in its firewall for the Collector and Schedd. The EC2 execute nodes have to be able to connect to condor.condorproject.org to talk to the condor_collector and condor_schedd. It cannot be behind a NAT or firewall. Okay, let’s start.

I am going to use ami-60bd4609, a publically available Fedora 15 AMI. You can either start the instance via the AWS console, or submit it by following previous instructions.

Once the instance is up and running, login and sudo yum install condor. Note, until BZ656562 is resolved, you will have to sudo mkdir /var/run/condor; sudo chown condor.condor /var/run/condor before starting condor. Start condor with sudo service condor start to get a personal condor.

Configuring condor on the instance is very similar to creating a multiple node pool. You will need to set the CONDOR_HOST, ALLOW_WRITE, and DAEMON_LIST,

# cat > /etc/condor/config.d/40execute_node.config
CONDOR_HOST = condor.condorproject.org
DAEMON_LIST = MASTER, STARTD
ALLOW_WRITE = $(ALLOW_WRITE), $(CONDOR_HOST)
^D

If you do not give condor.condorproject.org WRITE permissions, the Schedd will fail to start jobs. StartLog will report,

PERMISSION DENIED to unauthenticated@unmapped from host 128.105.291.82 for command 442 (REQUEST_CLAIM), access level DAEMON: reason: DAEMON authorization policy contains no matching ALLOW entry for this request; identifiers used for this host: 128.105.291.82,condor.condorproject.org, hostname size = 1, original ip address = 128.105.291.82

Now remember, we need bi-directional connectivity. So condor.condorproject.org must be able to connect to the EC2 instance’s Startd. The condor_start will listen on an ephemeral port by default. You could restrict it to a port range or use condor_shared_port. For simplicity, just force a non-ephemeral port of 3131,

# echo "STARTD_ARGS = -p 3131" >> /etc/condor/config.d/40execute_node.config

You can now open TCP port 3131 in the instance’s iptables firewall. If you are using the Fedora 15 AMI, the firewall is off by default and needs no adjustment. Additionally, the security group on the instance needs to have TCP port 3131 authorized. Use the AWS Console or ec2-authorize GROUP -p 3131.

If you miss either of these steps, the Schedd will fail to start jobs on the instance, likely with a message similar to,

Failed to send REQUEST_CLAIM to startd ec2-174-129-47-20.compute-1.amazonaws.com <174.129.47.20:3131>#1220911452#1#... for matt: SECMAN:2003:TCP connection to startd ec2-174-129-47-20.compute-1.amazonaws.com <174.129.47.20:3131>#1220911452#1#... for matt failed.

A quick service condor restart on the instance, and a condor_status on condor.condorproject.org would hopefully show the instance joined the pool. Except the instance has not been authorized yet. In fact, the CollectorLog will probably report,

PERMISSION DENIED to unauthenticated@unmapped from host 174.129.47.20 for command 0 (UPDATE_STARTD_AD), access level ADVERTISE_STARTD: reason: ADVERTISE_STARTD authorization policy contains no matching ALLOW entry for this request; identifiers used for this host: 174.129.47.20,ec2-174-129-47-20.compute-1.amazonaws.com
PERMISSION DENIED to unauthenticated@unmapped from host 174.129.47.20 for command 2 (UPDATE_MASTER_AD), access level ADVERTISE_MASTER: reason: ADVERTISE_MASTER authorization policy contains no matching ALLOW entry for this request; identifiers used for this host: 174.129.47.20,ec2-174-129-47-20.compute-1.amazonaws.com

The instance needs to be authorized to advertise itself into the Collector. A good way to do that is to add,

ALLOW_ADVERTISE_MASTER = $(ALLOW_WRITE), ec2-174-129-47-20.compute-1.amazonaws.com
ALLOW_ADVERTISE_STARTD = $(ALLOW_WRITE), ec2-174-129-47-20.compute-1.amazonaws.com

to condor.condorproject.org’s configuration and reconfig with condor_reconfig. A note here, ALLOW_WRITE is added in because I am assuming you are following previous instructions. If you have ALLOW_ADVERTISE_MASTER/STARTD already configured, you should append to them instead. Also, appending for each new instance will get tedious. You could be very trusting and allow *.amazonaws.com, but it is better to use SSL or PASSWORD authentication. I will describe that some other time.

After the reconfig, the instance will eventually show up in a condor_status listing.

$ condor_status
Name               OpSys      Arch   State     Activity LoadAv Mem   ActvtyTime
localhost.localdom LINUX      INTEL  Unclaimed Benchmar 0.430  1666  0+00:00:04

The name is not very helpful, but also not a problem.

It is time to submit a job.

$ condor_submit
Submitting job(s)
cmd = /bin/sleep
args = 1d
should_transfer_files = IF_NEEDED
when_to_transfer_output = ON_EXIT
queue
.^D
1 job(s) submitted to cluster 14.

$ condor_q
-- Submitter: condor.condorproject.org : <128.105.291.82:36900> : condor.condorproject.org
 ID      OWNER            SUBMITTED     RUN_TIME ST PRI SIZE CMD
  31.0   matt           11/11 11:11   0+00:00:00 I  0   0.0  sleep 1d
1 jobs; 1 idle, 0 running, 0 held

The job will stay idle forever, which is no good. The problem can be found in the SchedLog,

Enqueued contactStartd startd=<10.72.55.105:3131>
In checkContactQueue(), args = 0x9705798, host=<10.72.55.105:3131>
Requesting claim localhost.localdomain <10.72.55.105:3131>#1220743035#1#... for matt 31.0
attempt to connect to <10.72.55.105:3131> failed: Connection timed out (connect errno = 110).  Will keep trying for 45 total seconds (24 to go).

The root cause is that the instance has two internet addresses. A private one, which is not routable from condor.condorproject.org, that it is advertising,

$ condor_status -format "%s, " Name -format "%s\n" MyAddress
localhost.localdomain, <10.72.55.105:3131>

And a public one, which can be found from within the instance,

$ curl -f http://169.254.169.254/latest/meta-data/public-ipv4
174.129.47.20

Condor has a way to handle this. The TCP_FORWARDING_HOST configuration parameter can be set to the public address for the instance.

# echo "TCP_FORWARDING_HOST = $(curl -f http://169.254.169.254/latest/meta-data/public-ipv4)" >> /etc/condor/config.d/40execute_node.config

A condor_reconfig will apply the change, but a restart will clear out the old entry first. Oops. Note, you cannot set TCP_FORWARDING_HOST to the public-hostname of the instance, because the public hostname will be revolved within the instance and will resolve to the instance’s internal, private address.

When setting TCP_FORWARDING_HOST, also set PRIVATE_NETWORK_INTERFACE to let the host talk to itself over its private address.

# echo "PRIVATE_NETWORK_INTERFACE = $(curl -f http://169.254.169.254/latest/meta-data/local-ipv4)" >> /etc/condor/config.d/40execute_node.config

Doing so will prevent the condor_startd from using its public address to send DC_CHILDALIVE messages to the condor_master, which might fail because of a firewall or security group setting,

attempt to connect to <174.129.47.20:34550> failed: Connection timed out (connect errno = 110).  Will keep trying for 390 total seconds (200 to go).
attempt to connect to <174.129.47.20:34550> failed: Connection timed out (connect errno = 110).
ChildAliveMsg: failed to send DC_CHILDALIVE to parent daemon at <174.129.47.20:34550> (try 1 of 3): CEDAR:6001:Failed to connect to <174.129.47.20:34550>

Or if simply because the master does not trust the public address,

PERMISSION DENIED to unauthenticated@unmapped from host 174.129.47.20 for command 60008 (DC_CHILDALIVE), access level DAEMON: reason: DAEMON authorization policy contains no matching ALLOW entry for this request; identifiers used for this host: 174.129.47.20,ec2-174-129-47-20.compute-1.amazonaws.com, hostname size = 1, original ip address = 174.129.47.20

Now run that service condor restart and the public, routable address will be advertised,

$ condor_status -format "%s, " Name -format "%s\n" MyAddress
localhost.localdomain, <174.129.47.20:3131?noUDP>

The job will be started on the instance automatically,

$ condor_q -run
-- Submitter: condor.condorproject.org : <128.105.291.82:36900> : condor.condorproject.org
 ID      OWNER            SUBMITTED     RUN_TIME HOST(S)
  31.0   matt           11/11 11:11   0+00:00:11 localhost.localdomain

If you want to clean up the localhost.localdomain, set the instance’s hostname and restart condor,

$ sudo hostname $(curl -f http://169.254.169.254/latest/meta-data/public-hostname)
$ sudo service condor restart
(wait for the start to advertise)
$ condor_status -format "%s, " Name -format "%s\n" MyAddress
ec2-174-129-47-20.compute-1.amazonaws.com, <174.129.47.20:3131?noUDP>

In summary,

Configuration changes on condor.condorproject.org,

ALLOW_ADVERTISE_MASTER = $(ALLOW_WRITE), ec2-174-129-47-20.compute-1.amazonaws.com
ALLOW_ADVERTISE_STARTD = $(ALLOW_WRITE), ec2-174-129-47-20.compute-1.amazonaws.com

Setup on the instance,

# cat > /etc/condor/config.d/40execute_node.config
CONDOR_HOST = condor.condorproject.org
DAEMON_LIST = MASTER, STARTD
ALLOW_WRITE = $(ALLOW_WRITE), $(CONDOR_HOST)
STARTD_ARGS = -p 3131
^D
# echo "TCP_FORWARDING_HOST = $(curl -f http://169.254.169.254/latest/meta-data/public-ipv4)" >> /etc/condor/config.d/40execute_node.config
# echo "PRIVATE_NETWORK_INTERFACE = $(curl -f http://169.254.169.254/latest/meta-data/local-ipv4)" >> /etc/condor/config.d/40execute_node.config
# hostname $(curl -f http://169.254.169.254/latest/meta-data/public-hostname)

Spinning: Getting started: Condor and EC2 Importing instances with condor_ec2_link

Sat, 05 Nov 2011 16:02:58 +0000

Starting and managing instances describes the powerful feature of Condor to start and manage EC2 instances, but what if you are already using something other than Condor to start your instance, such as the AWS Management Console.

Importing instances turns out to be straightforward, if you know how instances are started. In a nutshell, the condor_gridmanager executes a state machine and records its current state in an attribute named GridJobId. To import an instance, submit a job that is already in the state where an instance id has been assigned. You can take a submit file and add +GridJobId = “ec2 https://ec2.amazonaws.com/ BOGUS INSTANCE-ID“. The INSTANCE-ID needs to be the actual identifier of the instance you want to import. For instance,

...
ec2_access_key_id = ...
ec2_secret_access_key = ...
...
+GridJobId = "ec2 https://ec2.amazonaws.com/ BOGUS i-319c3652"
queue

It is important to get the ec2_access_key_id and ec2_secret_access_key correct. Without them Condor will not be able to communicate with EC2 and EC2_GAHP_LOG will report,

$ tail -n2 $(condor_config_val EC2_GAHP_LOG)
11/11/11 11:11:11 Failure response text was '
AuthFailureAWS was not able to validate the provided access credentialsab50f005-6d77-4653-9cec-298b2d475f6e'.

This error will not be reported back into the job, putting it on hold, instead the gridmanager will think the EC2 is down for the job. Oops.

$ grep down $(condor_config_val GRIDMANAGER_LOG)
11/11/11 11:11:11 [10697] resource https://ec2.amazonaws.com is now down
11/11/11 11:14:22 [10697] resource https://ec2.amazonaws.com is still down

To simplify the import, here is a script that will use ec2-describe-instances to get useful metadata about the instance and populate a submit file for you,

condor_ec2_link

#!/bin/sh

# Provide three arguments:
#  . instance id to link
#  . path to file with access key id
#  . path to file with secret access key

# TODO:
#  . Get EC2UserData (ec2-describe-instance-attribute --user-data)

ec2-describe-instances --show-empty-fields $1 | \
   awk '/^INSTANCE/ {id=$2; ami=$3; keypair=$7; type=$10; zone=$12; ip=$17; group=$29}
        /^TAG/ {name=$5}
        END {print "universe = grid\n",
                   "grid_resource = ec2 https://ec2.amazonaws.com\n",
                   "executable =", ami"-"name, "\n",
                   "log = $(executable).$(cluster).log\n",
                   "ec2_ami_id =", ami, "\n",
                   "ec2_instance_type =", type, "\n",
                   "ec2_keypair_file = name-"keypair, "\n",
                   "ec2_security_groups =", group, "\n",
                   "ec2_availability_zone =", zone, "\n",
                   "ec2_elastic_ip =", ip, "\n",
                   "+EC2InstanceName = \""id"\"\n",
                   "+GridJobId = \"$(grid_resource) BOGUS", id, "\"\n",
                   "queue\n"}' | \
      condor_submit -a "ec2_access_key_id = $2" \
                    -a "ec2_secret_access_key = $3"

In action,

$ ./condor_ec2_link i-319c3652 /home/matt/Documents/AWS/Cert/AccessKeyID /home/matt/Documents/AWS/Cert/SecretAccessKey
Submitting job(s).
1 job(s) submitted to cluster 1739.

$ ./condor_ec2_q 1739
 ID      OWNER            SUBMITTED     RUN_TIME ST PRI SIZE CMD
1739.0   matt           11/11 11:11   0+00:00:00 I  0   0.0 ami-e1f53a88-TheNa
  Instance name: i-319c3652
  Groups: sg-4f706226
  Keypair file: /home/matt/Documents/AWS/name-TheKeyPair
  AMI id: ami-e1f53a88
  Instance type: t1.micro
1 jobs; 1 idle, 0 running, 0 held

(20 seconds later)

$ ./condor_ec2_q 1739
 ID      OWNER            SUBMITTED     RUN_TIME ST PRI SIZE CMD
1739.0   matt           11/11 11:11   0+00:00:01 R  0   0.0 ami-e1f53a88-TheNa
  Instance name: i-319c3652
  Hostname: ec2-50-17-104-50.compute-1.amazonaws.com
  Groups: sg-4f706226
  Keypair file: /home/matt/Documents/AWS/name-TheKeyPair
  AMI id: ami-e1f53a88
  Instance type: t1.micro
1 jobs; 0 idle, 1 running, 0 held

There are a few things that can be improved here, the most notable of which is the RUN_TIME. The Gridmanager gets status data from EC2 periodically. This is how the EC2RemoteVirtualMachineName (Hostname) gets populated on the job. The instance’s launch time is also available. Oops.

Inside OSG Ops.: Moving Services to Bloomington

noreply@blogger.com (scott) — Thu, 03 Nov 2011 08:08:33 +0000

As you know, the GOC updates services on the second and fourth Tuesday of each month.
The update scheduled for November 8th marks a milestone for the infrastructure team.
After this date all GOC services (with one exception) be be hosted exclusively in
the Bloomington, Indiana data center.

Previously, most services had two instances, one physically hosted in Indianapolis
the other in Bloomington. These instances are in DNS round robin allowing users
of these services transparent use of either instance. The GOC will continue to
operate (at least) two instances and keep them in round robin, but both instances
will be in Bloomington.

So why the change? Originally, the Bloomington machine room was extremely unreliable.
Problems included a leaky roof, insufficient cooling and power and a lack
of space. In short, the systems hosted there had outgrown the facility. The machine
room in Indianapolis was larger, newer and considered more reliable. The old Bloomington
machine room went down during a thunderstorm when it was discovered that both electrical
feeds were, at one point, hung from the same utility pole. (Care to guess where the
lightning struck?) Two weeks were required to restore power during which many of the
university enterprise services were unavailable. This situation was clearly unacceptable
so the university decided to invest $37.2M in a new, state-of-the-art data center.

The 92,000 sq. ft. Bloomington data center is designed to withstand category 5 tornadoes.
The facility is secured with card-key access and 7 x 24 x 365 video surveillance.
Only staff with systems or network administration privileges have access to the machine room
requiring biometric identity verification. Fire suppression is provided by a double interlock
system accompanied by a Very Early Smoke Detection Apparatus (VESDA). Three circuits feed
the Data Center, traveling redundant physical paths from two different substations.
Any two circuits can fully power the building. A flywheel motor/generator set conditions
the power and provides protection against transient events and uninterruptible power
supplies protect against failures of moderate (~1 hour) duration. Dual diesel generators
can provide power for 24 hours in the event of a longer term power failure. In house
chillers provide cooling. Externally supplied chilled water plus city water can be used
in the event of a failure of this system.

Several advantages are realized by hosting all instances in one location. Service failures
associated with the network between Indianapolis and Bloomington are avoided. By using the
same LAN, DNS round robin can be replaced with Linux Virtual Server (LVS) giving control
of round robin to the GOC rather than the DNS administrators at Indiana University. Also
avoided are failures associated with the loss of one of two data centers. It is trivial to
move virtual machines from host to host since the IP address of the VM does not change, a
property allowing detailed load balancing on all VM hosts.

The GOC looks forward to continuing providing services with the availability OSG users
have come to expect.