News and Announcements from OSG Operations > RESOLVED: OSG PKI Currently Experiencing Difficulties Issuing Certificates

Colleagues,

The OSG PKI (OIM to CILogon) was experiencing difficulties issuing both personal and host certificates. Certificates were able to be requested and approved, but once the requester clicked the issue button, the operation failed.  This issue has been fixed. Any certificates that failed during the issuing stage will need to be re-requested.  We apologize for any convenience this has caused and if you have any questions, please don't hesitate to contact us using the information below.

News and Announcements from OSG Operations > OSG PKI Currently Experiencing Difficulties Issuing Certificates

Colleagues,

The OSG PKI (OIM to CILogon) is currently experiencing difficulties issuing both personal and host certificates. Certificates are able to be requested and approved, but once the requester clicks the issue button, the operation fails.  Contact has been made with CILogon and we are looking into the situation.  This outage could last until tomorrow, but we will update you as we learn more. We apologize for any inconvenience.

Condor Project News > HTCondor 8.7.1 released! ( April 24, 2017 )

The HTCondor team is pleased to announce the release of HTCondor 8.7.1. This development series release contains new features that are under development. This release contains all of the bug fixes from the 8.6.2 stable release. Enhancements in the release include: Several performance enhancements in the collector; Further refinement and initial documentation of the HTCondor Annex; Enable chirp for Docker jobs; Job Router uses first match rather than round-robin matching; The schedd tracks jobs counts by status for each owner; Technology preview of late job materialization in the schedd. Further details can be found in the Development Version History and the Stable Version History. HTCondor 8.7.1 binaries and source code are available from our Downloads page.

Condor Project News > HTCondor 8.6.2 released! ( April 24, 2017 )

The HTCondor team is pleased to announce the release of HTCondor 8.6.2. A stable series release contains significant bug fixes. Highlights of this release are: New metaknobs for mapping users to groups; Now case-insensitive with Windows user names when storing credentials; Signal handling in the OpenMPI script; Report RemoteSysCpu for Docker jobs; Allow SUBMIT_REQUIREMENT to refer to X509 secure attributes; Linux kernel tuning script takes into account the machine's role. More details about the fixes can be found in the Version History. HTCondor 8.6.2 binaries and source code are available from our Downloads page.

News and Announcements from OSG Operations > GOC Service Update - Tuesday, April 25th

The GOC will upgrade the following services beginning Tuesday, 25/Apr at 14:00 UTC. The GOC reserves 8 hours in the unlikely event unexpected problems are encountered.

MyOSG
Modify some links associated with BDII and GratiaWeb. Adding GRACC graphs.

Ticket Exchange
FNAL configuration update

OIM
Modify Gratia references to appropriate Gracc equivalent

Display
Update to CentOS7, replace Gratia references with appropriate GRACC equivalent

Web services
Apply standard software updates

All services
Operating system updates; reboots will be required. The usual HA mechanisms will be used, but some services will experience brief outages.

Condor Project News > Preliminary HTCondor Week schedule available (April 15, 2017 )

A preliminary verson of the full HTCondor Week schedule is now available. You can access the schedules for each day from the HTCondor Week page. Please remember that the registration deadline for HTCondor Week is Tuesday, April 25 (register here).

News and Announcements from OSG Operations > Announcing OSG CA Certificate Update

We are pleased to announce a data release for the OSG Software Stack.
Data releases do not contain any software changes.

This release contains updated CA Certificates based on IGTF 1.82:
- Added new G2 UGrid trust anchor (UA)
- Extended validity for AEGIS CA (RS)
- Withdrawn discontinued FNAL KCA (US)
- Extended validity for REUNA CA (CL)

Release notes and pointers to more documentation can be found at:

https://twiki.opensciencegrid.org/bin/view/Documentation/Release3/Release33223

Need help? Let us know:

https://twiki.opensciencegrid.org/bin/view/Documentation/Release3/HelpProcedure

We welcome feedback on this release!

News and Announcements from OSG Operations > Announcing OSG Software version 3.3.23

We are pleased to announce OSG Software version 3.3.23.

Changes to OSG 3.3.23 include:
- LCMAPS VOMS plugin: Use VOMS attributes to map users
- HTCondor-CE 2.1.5: LCMAPS VOMS integration, package Slurm configuration
- CVMFS 2.3.5: Fixes, including automount fix when autofs restarts on EL7
- Pegasus 4.7.4: Updated from version 4.6.1
- OSG CE 3.3-12: Removed gip and osg-info-services, see release notes

Changes to the Upcoming Repository include:
- LCMAPS 1.6.6-1.3: Enable VOMS attribute checking by default
- Frontier Squid 3.5.24-3.1: Fix for crashes under heavy load

Release notes and pointers to more documentation can be found at:

https://www.opensciencegrid.org/bin/view/Documentation/Release3/Release3323

Need help? Let us know:

https://www.opensciencegrid.org/bin/view/Documentation/Release3/HelpProcedure

We welcome feedback on this release!

News and Announcements from OSG Operations > GratiaWeb to GRACC transition

OSG has been building a modernized accounting system, GRÅCC, to better meet OSG and stakeholder accounting needs in the years to come. In addition to new ways to collect and store accounting data, there is a new web interface:

    https://gracc.opensciencegrid.org/

   OSG staff have started using this new URL over the past two months, and now we believe that it is ready for everyone to use. At this time, we strongly encourage users to switch to it. If you find issues, please contact OSG Operations <goc@opensciencegrid.org>.

   Due to the shutdown of the Gratia database at FNAL on May 1st, we plan to retire the older accounting reporting service (GratiaWeb) on 18 April 2017.

    http://gratiaweb.grid.iu.edu/

   This schedule leaves a bit of room to fix any problems that crop up due to the transition from one reporting service to the other. Please contact OSG Operations with any concerns or if you will need assistance migrating from GratiaWeb to GRÅCC.

News and Announcements from OSG Operations > Applications for OSG User School 2017 due April 14!

NOTE: The application deadline for the OSG User School 2017 is soon!
Submit applications and recommendations by Friday, April 14th.

------------------------------------------------------------------------

If you could access hundreds or even thousands of computers for your
scholarly work, what could you do? How could it transform your work?
What discoveries might you make?

We are seeking applicants for the Open Science Grid (OSG) User School
2017, which takes place July 17-21 at the beautiful University of
Wisconsin in Madison. Participants will learn to use high throughput
computing (HTC) to harness vast amounts of computing power for research,
applicable to nearly any field of study (e.g., physics, chemistry,
engineering, life sciences, earth sciences, agricultural and animal
sciences, economics, social sciences, medicine, and more).

http://research.cs.wisc.edu/htcondor/htc.html
https://www.opensciencegrid.org/news/research-highlights-list/

Using lectures, discussions, roleplays, and lots of hands-on work with
OSG experts in HTC, participants will learn how HTC systems work, how to
run and manage many jobs and huge datasets, how to implement a realistic
scientific computing workflow, and where to turn for help and more info.

Worried about costs? We pay all basic travel, hotel, and food costs for
applicants who are selected to attend. This is a valuable offer!

Ideal candidates are graduate students whose research involves or could
involve large-scale computing - work that cannot be done on one laptop
or a handful of computers. Also, we accept some post-doctoral students,
faculty, staff, and advanced undergraduates, so make a strong case for
yourself regardless of your current role!

IMPORTANT DATES
  Application Period (OPEN NOW): 1 March - 14 April 2017
  OSG User School: 17-21 July 2017

MORE INFORMATION AND APPLICATIONS
  Web: http://www.opensciencegrid.org/UserSchool
  Email: user-school@opensciencegrid.org
  Facebook: https://www.facebook.com/OSGUserSchool
  Twitter: https://twitter.com/OSGUserSchool

Please forward this announcement to anyone who might benefit from the
OSG User School. And consider posting our flyer where appropriate:

https://www.opensciencegrid.org/userschoolflyer/
https://twiki.opensciencegrid.org/twiki/pub/Education/OSGUserSchool2017/osg-user-school-2017-flyer.pdf

News and Announcements from OSG Operations > Re: XRootD 4.6.0 removed from OSG 3.3.22

On March 11th, OSG Software removed XRootD 4.6.0 from the Yum (RPM) repositories, due to defects in the software. However, the OSG worker node tarballs had already been created with some elements of the XRootD 4.6.0 release. We discovered issues related to the tarball contents last week, and on March 29th, OSG Software rebuilt and re-released the OSG 3.3.22 worker node tarballs to use XRootD 4.5.0 instead.

If you installed OSG worker node tarballs (directly or from OASIS) between March 11th and 29th, you should consider reinstalling them using the latest builds (labeled as version 3.3.22-2).

News and Announcements from OSG Operations > GOC Service Update - Tuesday, April 11th at 14:00 UTC

The GOC will upgrade the following services beginning Tuesday, April 11th at 14:00 UTC. The GOC reserves 8 hours in the unlikely event unexpected problems are encountered.

MyOSG
- Modify some links associated with BDII and GratiaWeb

Jira Update
- Upgrade Jira to 7.3.1, and underlying OS to RHEL 7

Ticket Security Notification changes
- Update certain options on the security ticket creation form per https://jira.opensciencegrid.org/browse/TICKET-129

News and Announcements from OSG Operations > Travel Support Available for PEARC17 Student Contributors

PEARC17, Practice & Experience in Advanced Research Computing 2017, is now offering financial support opportunities for students with submissions to the main conference program—thanks to support from XSEDE, the San Diego Supercomputer Center, and through the fundraising efforts of STEM-Trek and Virginia Tech. The deadline to apply is May 25, 2017.

Funding is available to cover costs of airfare, shared lodging, and registration fees. Due to funding constraints, participation in the PEARC17 Student Program is limited to students at U.S. and Canadian institutions, and partial support is requested from the student’s institution.

To receive travel support for the PEARC17 Student Program, students need to  APPLY HERE no later than May 25, and are required to participate in all student activities, including the volunteer program. Student authors will be notified of accepted papers by mid-April and accepted posters by mid-May.

Learn more about the PEARC17 Student Program and participation opportunities here  http://pearc17.pearc.org/student-program.

ABOUT PEARC17

PEARC17—Practice & Experience in Advanced Research Computing 2017 - unites the high-performance computing and advanced digital research communities, addressing the challenges of using and operating advanced research computing within academic and open science communities. Being held in New Orleans July 9-13, PEARC17 offers a robust technical program, as well as networking, professional growth and multiple student participation opportunities. See pearc17.pearc.org for more information.

News and Announcements from OSG Operations > Announcing VO Package Update

We are pleased to announce a data release for the OSG Software Stack.
Data releases do not contain any software changes.

This release contains VO Package v72:
- Added VOMS server to OSG VO: voms.grid.iu.edu
- Removed CSIU, NYSGRID, and OSGEDU VOs

Release notes and pointers to more documentation can be found at:

https://www.opensciencegrid.org/bin/view/Documentation/Release3/Release33222

Need help? Let us know:

https://www.opensciencegrid.org/bin/view/Documentation/Release3/HelpProcedure

We welcome feedback on this release!

Marco's Blog! > A cell counting demo using BOSCO and OSG Connect

This post describes a demo that I ran few months ago on Open Science Grid using BOSCO and OSG-Connect.
Let's assume I have a lot of scientific data to analyze, e.g. I want to count the cells in many images stored on my computer, and I don't have the computing power to complete in a timely manner. Opportunistic resources on Open Science Grid and some tools available to its community can help me get the task done. In this tutorial I will show:
  • how to register on OSG Connect to get access to OSG resources 
  • how to easily move your data so that it can be available on the Web and used by your jobs 
  • how to install and use BOSCO to submit the python jobs and retrieve the results on your laptop or workstation 

Here is a recording and below is an extended version of the original tutorial. Although OSG is still active, the original tutorial page and some of the resources described below may not be available any more.




The task

I have several pictures like this:
Figure 1.  Sample image
These pictures look a bit like biological samples where cells (clusters of different luminosity) are visible on top of a background. And I'd like to count the "cells" into the picture. This is done by a clusterization process illustrated in Figure 2 and in this tutorial. The image on the left is just adding colors but has the same information of the black and white one. Using the intensity of all pixels the algorithm evaluates a threshold and builds a mask (second image) that allows to separate the important elements from the background. Finally the distinct elements are separated and counted.

Figure 2.  Example of clusterization

Register on OSG Connect

Before using OSG you have to get access to the resources. This page explains how to join OSG. Here are the practical steps:
  1. Go to the OSG Connect web site and register by clicking on "Log In or Register" (upper right corner) and following the instructions. Here a document explaining the registration
  2. Join a project or create a new project (e.g. one describing your research activity Test-CellCounting).
  3. Login on the submit host login.osgconnect.net and start submitting your jobs. Here is a quickstart guide but you'll feel right at home if you are familiar with HTCondor. Just remember to specify a ProjectName in your jobs or to add a $HOME/.osg_default_project file with your default project name. And there is much more: ConnectBook is a collection of guides about OSG Connect.

Transfer the files

In the submit host the directory ~/data/public is exported on the Web by stash, the OSG Connect data server, as https://stash.osgconnect.net/+USER_NAME/ (where USER_NAME is your user name on OSG Connect)and you can use it to move your data. See Access Stash remotely from your job using HTTP for an example.
I will use this feature and transfer my files there using Globus Online.

Install and setup BOSCO

Download & Install Bosco

This is a very abbreviated install document for BOSCO. For the full install document, view Bosco Quick Start Installer.
Bosco Download Visit the Bosco download page. Choose the Quick Start Installer. After downloading the installer, from the terminal, untar it and run the installer as a regular user:
[user@hostname ~]$ tar xvzf ./bosco_quickstart.tar.gz
[user@hostname ~]$ ./bosco_quickstart

Starting Bosco & adding your first cluster using the quick start

login.osgconnect.net is an HTCondor resource I can connect to BOSCO answering the questions:
  • login.osgconnect.net (FQDN of the cluster) 
  • myuser (my user name) 
  • condor (queue manager in the cluster) 

Run the code 

You can test the job locally on your laptop/workstation or login.osgconnect.net but then all the jobs will run on different resources across OSG, all with different software stacks. So I don't know if the software I'm using will be available.
Anyway there are software packages like R and Anaconda (Python, NumPy, ...) available across almost all of OSG thanks to PALMS and OASIS. And I can make sure that my job runs on a node where PALMS and OASIS are available. See Software access using OASIS for detailed instructions on how to use PALMS and OASIS and for a list of the available software.
Python 2.7.5 with all the scientific libraries (the ones included in Anaconda) is all that I will need for my jobs.
Here is the python script counting the "cells" in the image. This is a simplified version. On a real analysys the treshold is more complex than a simple average and some pre-filtering of the image or pattern recognition are used to improve the results:
#!/bin/env python
# count_cells.py - script counting the cells in the image

import numpy as np
from scipy import ndimage
import Image
from scipy.misc import fromimage

# Import image
pic = Image.open("./cellimage.tiff")
im = fromimage(pic)

# Evaluate treshold and segment the image
T = im.mean()
mask = im > T
label_im, nb_labels = ndimage.label(mask)

# print result
print nb_labels
Here is a wrapper script that downloads the image file, sets up the environment and launches the python script. Replace USER_NAME with your username on login.osgconnect.net:
#!/bin/sh
# job_wrapper.sh - script guiding the job execution
IMAGE_FILE=cellimage.tiff
IMAGE_URL=http://stash.osgconnect.net/+USER_NAME/images/cellimage$1.tiff

# download the file
wget -O $IMAGE_FILE "$IMAGE_URL" >&2
if [ ! $? -eq 0 ]; then
echo "Wget failed. Trying curl." >&2
curl -o $IMAGE_FILE "$IMAGE_URL" >&2
if [ ! $? -eq 0 ]; then
echo "Also curl failed. Exiting." >&2
ls -l >&2
exit
fi
fi
if [ ! -f ./$IMAGE_FILE ]; then
echo "Missing image file ./$IMAGE_FILE. Exiting." >&2
ls -l >&2
exit
fi

# setup the environment
source /cvmfs/oasis.opensciencegrid.org/osg/palms/setup 1>&2
palmsdosetup anaconda 1>&2

python ./count_cells.py
Here is the HTCondor submit file:
# job.sub - HTCondor submit file
# If submitting from login.osgconnect.net use: universe = vanilla
# When submitting via BOSCO use:
universe = grid

# remember to create the batch1 directory
Error = batch1/job.err.$(Cluster)-$(Process)
Output = batch1/job.out.$(Cluster)-$(Process)
Log = batch1/job.log.$(Cluster)

# Remember to set the job_wrapper.sh es executable (chmod +x):
Executable = job_wrapper.sh
Arguments = $(Process)
Transfer_input_files = count_cells.py
ShouldTransferFiles = YES
When_to_transfer_output = ON_EXIT

Requirements = (HAS_CVMFS_oasis_opensciencegrid_org =?= TRUE) && (CVMFS_oasis_opensciencegrid_org_REVISION >= 729)
+ProjectName = "ConnectTrain"

# Submitting 100 jobs to analyze 100 different images
queue 100
To submit the jobs I normal HTCondor commands as documented in the BOSCO quickstart. condor_submit returns a job ID that you will then use for the following commands (referred as JOB_ID below). To be more precise the number returned is the job cluster ID, then within my cluster there are 100 job processes (0 to 99), each analyzing a different image:
  • to submit: condor_submit job.sub 
  • then to monitor the jobs running: watch -n 2 condor_q JOB_ID 

Once the jobs completed, the result (cells counted in the image) was returned in the jobs standard output that is copied locally. To sum all the cells I evaluated the total. Replace jour job ID (returned by condor_submit) in the command below:
cat batch1/job.out.JOB_ID-* | paste -sd+ - | bc

References and additional material

Here the links to some more proper cell counting articles. You can take any code and adapt your workflow to run in parallel on OSG:



News and Announcements from OSG Operations > Be a mentor at PEARC17 and make a difference!

Make a difference in the lives of next-generation scientists and science leaders at PEARC17—Practice & Experience in Advanced Research Computing 2017 in New Orleans, July 9-13. As a mentor, you will give students the opportunity to learn from your expertise as they explore a variety of related issues, such as differing specialties, postgraduate training, and work/life balance; plus enjoy Monday’s complimentary student-mentor dinner. Many mentorships turn into lifetime friendships and have countless benefits.  

Check "I want to be a mentor" when you register for the PEARC17 conference, which will take place in New Orleans, July 9-13. The conference is open to professionals and students in advanced research computing. Registration for the conference and hotel are now open.


ABOUT PEARC17

Practice & Experience in Advanced Research Computing 2017—unites the high-performance computing and advanced digital research communities, addressing the challenges of using and operating advanced research computing within academic and open science communities. Being held in New Orleans July 9-13, PEARC17 offers a robust technical program, as well as networking, professional growth and multiple student participation opportunities. See pearc17.pearc.org for more information, and follow us PEARC on Twitter at (PEARC_17) and on Facebook (PEARChpc).

News and Announcements from OSG Operations > Center for Network and Storage-Enabled Collaborative Computational Science Symposium

Dear Colleagues,

Please join us May 18-19 in Ann Arbor for the first symposium of the Center for Network and Storage-Enabled Collaborative Computational Science (CNSECCS).

The symposium is intended to bring together scientists, researchers and infrastructure specialists to share experiences and best practices, and to discuss both challenges and possible solutions that enable scientists to work together on “big, distributed or diverse data”.

Speakers will come from a wide range of research domains, as well as federal funding agencies:

  * Amy Friedlander, Deputy Division Director, Division of Advanced Cyberinfrastructure (CISE/ACI), National Science Foundation.
  * Richard Carlson, Program Officer, Advanced Scientific Computing Research (ASCR), Department of Energy
  * Nina Silverberg, Program Director, Alzheimer's Disease Centers program, Division of Neuroscience, National Institutes of Health

We are also accepting abstracts from potential presenters through the beginning of April.

For more information and to register, visit:
http://micde.umich.edu/centers/cnseccs/2017-symposium/

For more about the center, see:
http://micde.umich.edu/centers/cnseccs/

And for any questions, please email
micde-contact@umich.edu.

We hope to see you there!

Best regards,

Shawn McKee, Director, CNSECCS

News and Announcements from OSG Operations > XRootD 4.6.0 removed from OSG 3.3.22

XRootD 4.6.0 was pulled from the OSG repositories. The OSG repositories still contain XRootD 4.5.0.
XRootD 4.6.0 was in the OSG repositories from Tuesday, March 11th until Friday, March 17th.

End users encountered XRootD failures when using GSI authentication.
The problems have been traced to a defect in the CRL verification code.
XRootD running in a server mode will likely experience problems,
since reading the served file usually requires authentication.
XRootD running in the manager mode may not experience any difficulties,
since the redirector typically does not do authentication.

To check if you have the affected components installed, run
`rpm -qa | grep xrootd` to display the versions of your xrootd packages.
If any of them are version 4.6.0, run `yum downgrade <package>` on those
packages.

We are awaiting the 4.6.1 XRootD release which corrects these issues.
We will test and release it next available OSG release.

We apologize for any inconvenience. Our release testing did not catch
this problem. We are augmenting our release processes to catch this
specific problem as well as similar issues that may come up in other
packages.

News and Announcements from OSG Operations > GOC Service Update - Tuesday, March 28th at 14:00 UTC

The GOC will upgrade the following services beginning Tuesday, March 28th at 14:00 UTC. The GOC reserves 8 hours in the unlikely event unexpected problems are encountered.

Condor Collectors
- Update HTCondor View

Data
- Update internal database servers data1 and data 2

Repo
- Rebuild production instances at RHEL6

Web Services
- Apply standard software updates

All services
- Operating system updates; reboots will be required. The usual HA mechanisms will be used, but some services will experience brief outages.

News and Announcements from OSG Operations > OSG BDII Decommissioning

OSG Collaborators,

As has been previously announced, OSG Operations and Technology are planning the retirement of the BDII information service located at is.grid.iu.edu on March 31st, 2017. As of that date, the SLA for the BDII will expire. In addition to this, on April 4th, at 9:00 US Eastern time, Operations will power off both instances of the BDII.

This is the final announcement regarding the decommissioning of this service.

Pegasus news feed > Pegasus Featured in Blue Waters Webinar Series

Blue Waters provides a series of free webinars throughout the year. The Scientific Workflows webinar track provides an overview of common scientific workflows and tools that enable them. In the first presentation of series of webinars, Scott Callaghan presented an overview of scientific workflows, where he discussed what the community means by “workflows” and what elements make up a workflow.

The presentation features Pegasus WMS as the solution adopted by the Southern California Earthquake Center (SCEC) to address challenges of large scale scientific computations in the Cybershake workflow, which include workflow automation, data management, resource provisioning (mover 500+K jobs through the cluster efficiently), and error handling.

The video for the presentation is shown below, and the full description of the webinar is available here. This presentation has also been featured in insideHPC.

 

The Pegasus team will also present a webinar in the Blue Waters Scientific Workflow webinar track:

Blue Waters Webinar: Pegasus Workflow Management System
Link: https://bluewaters.ncsa.illinois.edu/webinars/workflows/pegasus
Date: May 24, 2017
Presenter: Mats Rynge

 

 

 

 

215 views


News and Announcements from OSG Operations > Announcing OSG Software version 3.3.22

We are pleased to announce OSG Software version 3.3.22.

Changes to OSG 3.3.22 include:
- HTCondor: Patched to avoid Job Router crash when submit requirements fail
- New default configuration: GUMS uses proper TLS version(s) with VOMS admin
- BLAHP 1.18.29: Better Slurm integration, fixed problem with proxy refresh
- HTCondor-CE 2.1.4: Respect 'RequestCpus', Added JSON attributes for AGIS
- Gratia probe 1.17.4: Now picks up 'RequestCpus' with HTCondor-CE
- Update to CVMFS 2.3.3
- Update to GlideinWMS 3.2.18
- Update to XRootD 4.6.0
- HDFS: GridFTP prints proper error message when HDFS quota is exhausted
- VOMS 2.0.14-1.3: Now validates top-level group of proxy

Changes to the Upcoming Repository include:
- HTCondor 8.6.1: New stable series of HTCondor in Upcoming
- Update to frontier-squid 3.5.24

Release notes and pointers to more documentation can be found at:

https://www.opensciencegrid.org/bin/view/Documentation/Release3/Release3322

Need help? Let us know:

https://www.opensciencegrid.org/bin/view/Documentation/Release3/HelpProcedure

We welcome feedback on this release!


Subscribe