News and Announcements from OSG Operations > Announcing OSG Software version 3.3.30

We are pleased to announce OSG Software version 3.3.30.

Changes to OSG 3.3.30 include:
- OSG PKI: Certificate requests are authenticated by default
- BLAHP: Fixed bug in Slurm memory-use parsing that caused jobs to be held
- HTCondor: Fixed issue validating VOMS proxies
- XRootD 4.7.1: Fixed occasional crash when LCMAPS callout to GUMS fails
- CVMFS 2.4.2: Server side bug fixes
- GridFTP-HDFS: Added support for CMVFS checksums
- Globus GridFTP server: Fixed IPv6 redirection and IPv4 EPSV response
- LCMAPS VOMS Plugin: Documented how to map using all FQANs
- RSV: Fixed CRL freshness probe, removed unused probes

Note: This is the last release where the target release date is the second
Tuesday of the month. In the future, we will release whenever we have
significant or pressing changes.

Release notes and pointers to more documentation can be found at:

http://opensciencegrid.github.io/docs/release/3.3/release-3-3-30/

Need help? Let us know:

http://opensciencegrid.github.io/docs/common/help/

We welcome feedback on this release!

News and Announcements from OSG Operations > Announcing OSG Software version 3.4.5

We are pleased to announce OSG Software version 3.4.5.

Changes to OSG 3.4.5 include:
- OSG PKI: Certificate requests are authenticated by default
- BLAHP: Fixed bug in Slurm memory-use parsing that caused jobs to be held
- HTCondor: Updated to 8.6.8 and fixed issue validating VOMS proxies
- XRootD 4.7.1: Fixed occasional crash when LCMAPS callout to GUMS fails
- CVMFS 2.4.2: Server side bug fixes
- Globus GridFTP server: Fixed IPv6 redirection and IPv4 EPSV response
- LCMAPS VOMS Plugin: Documented how to map using all FQANs
- RSV: Fixed CRL freshness probe, removed unused probes
- Upcoming: Updated to HTCondor 8.7.5 and BLAHP 1.18.34

Note: This is the last release where the target release date is the second
Tuesday of the month. In the future, we will release whenever we have
significant or pressing changes.

Release notes and pointers to more documentation can be found at:

http://opensciencegrid.github.io/docs/release/3.4/release-3-4-5/

Need help? Let us know:

http://opensciencegrid.github.io/docs/common/help/

We welcome feedback on this release!

News and Announcements from OSG Operations > Operations Service Update - Tuesday, November 20th beginning 14:00 UTC

The GOC will upgrade the following services on November 20th beginning 14:00 UTC. The GOC reserves 8 hours in the unlikely event unexpected problems are encountered.

MyOSG
  * Installing nodejs and associated libraries.

Event
  * Configuration Change, memory and data usage threshold. Increasing available resources to survive 4 day outages on consumer side.

Condor Project News > HTCondor Security Release: 8.6.8 and 8.7.5 ( November 14, 2017 )

The HTCondor team is pleased to announce the release of HTCondor 8.6.8 and HTCondor 8.7.5. These releases contain an important fix for a security issue. Affected users should update as soon as possible. More details on the security issue are in the Vulnerability Report HTCondor binaries and source code are available from our Downloads page.

Derek's Blog > Cleaning Up GRACC

The GRid ACcounting Collector (GRACC) is the OSG’s new version of accounting software, replacing Gratia. It has been running in production since March 2017. Last week, on Friday November 3rd, we held a GRACC Focus Day. Our goal was to clean up data that is presented in GRACC. My changes where:

  • Update the GRACC-Collector to version 1.1.8. The primary change in this release is setting the messages sent to RabbitMQ to be “persistent”. The persistent messages are then saved to disk in order to survive a RabbitMQ reboot.
  • Use case-insenstive comparisons to determine the Open Science Grid Information Management system (OIM) information. This was an issue with GPGrid (Fermilab), which was registered as GPGRID.
  • Set the OIM_Site equal to the Host_description attribute if the OIM logic is unable to determine the registered OIM site. This is especially useful for the LIGO collaboration, which uses sites in Europe that are not registered in OIM. Now, instead of a lot of Unknown sites listed on the LIGO site listing, it shows the somewhat reported site name of where the job ran.
GRACC Projects Page GRACC Projects Page for LIGO

Regular Expression Corrections

One of the common problems we have in GRACC is poor data coming from the various probes installed at hundreds of sites. We don’t control the data coming into GRACC, so occasionally we must make corrections to the data for clarity or correctness. One of these corrections is misreporting the “site” that the jobs ran on.

In many instances, the probe is unable to determine the site and simply lists the hostname of the worker node where the job ran. This can cause the cardinality of sites listed in GRACC to increase dramatically as we get new hostnames inserted into the sites listing. If the hostnames are predictable, a regular expression matching algorithm can match a worker node hostname to a proper site name.

The largest change for GRACC was the regular expression corrections. With this new feature, GRACC administrators can set corrections to match on attributes using regular expression patterns. For example, consider the following correction configuration.

[[Corrections]]
index = 'gracc.corrections'
doc_type = 'host_description_regex'
match_fields = ['Host_description']
source_field = 'Corrected_OIM_Site'
dest_field = 'OIM_Site'
regex = true

This configuration means:

Match the Host_description field in the incoming job record with the regular expression Host_description field in the corrections table. If they are a match, take the value in the Corrected_OIM_Site field in the corrections table and place it into the OIM_Site field in the job record.

And the correction document would look like:

{
  "_index": "gracc.corrections-0",
  "_type": "host_description_regex",
  "_id": "asldkfj;alksjdf",
  "_score": 1,
  "_source": {
    "Host_description": ".*\.bridges\.psc\.edu",
    "Corrected_OIM_Site": "PSC Bridges",
  }
}

The regular expression is in the Host_description FIELD.

So, if the incoming job record is similar to :

{
...
"Host_description": "l006.pvt.bridges.psc.edu"
...
}

Then the correction would modify or create values such that the final record would approximate:

{
...
"Host_description": "l006.pvt.bridges.psc.edu",
"OIM_Site": "PSC Bridges",
"RawOIM_Site": ""
...
}

Note that the Host_description field stays the same. We must keep it the same because it is used in record duplicate detection. If we modified the field and resummarized previous records, then it would cause multiple records to represent the same job.

The GRid ACcounting Collector (GRACC) is the OSG’s new version of accounting software, replacing Gratia. It has been running in production since March 2017. Last week, on Friday November 3rd, we held a GRACC Focus Day. Our goal was to clean up data that is presented in GRACC. My changes where:

Pegasus news feed > Pegasus at SC 2017

Are you going to attend the SC17 conference in Denver, Colorado on November 12-17, 2017?

We will be giving a presentation “Software Integrity with Pegasus:Securing Scientific Workflow Data”  about our work as part of the  NSF SWIP project on

  • Tuesday, Nov 14th at 2:00pm at the University of Southern California booth #1731.
  • Wednesday, Nov 15th at 10:30am at the Indiana University booth #601.
  • Wednesday, Nov 15th at 11:30am at the RENCI booth #437.

There will be a Pegasus Introduction and Upcoming Features Talk on

  • Wednesday, Nov 15th at 2:00pm at the University of Southern California booth #1731.

We will be holding Pegasus office hours on

  • Tuesday, Nov 14th at 3:00 pm – 4:00 pm at the University of Southern California booth #1731.
  • Wednesday, Nov 15th at 3:00 pm – 4:00 pm at the University of Southern California booth  #1731.

If the above times don’t work for you and you would like to meet with us, you can reach us at pegasus aT isi dot edu to schedule a meeting at SC.

There will also be presentation Panorama 360: Performance Data Capture and Analysis for End-to-end Scientific Workflows about our work as part of DOE Panorama Project on

  • Tuesday, Nov 14th at 12:30 pm – 1:30 pm at the RENCI booth #437.
  • Thursday, Nov 16th at 11:30 am – 12:30 pm at the RENCI booth #437.

Two research papers will be presented in the WORKS workshop. Please, join us in the workshops and let’s have some coffee and very interesting discussions.

WORKS 2017: 12th Workflows in Support of Large-Scale Science (Monday, November 13, 2017):

  • S. Callaghan, G. Juve, K. Vahi, P. J. Maechling, T. H. Jordan, and E. Deelman, “rvGAHP – Push-based Job Submission using Reverse SSH Connections,” in 12th Workshop on Workflows in Support of Large-Scale Science (WORKS’17), 2017.
    [Bibtex]
    @inproceedings{callaghan-works-2017,
    title = {{rvGAHP} - Push-based Job Submission using Reverse SSH Connections},
    author = {Callaghan, Scott and Juve, Gideon and Vahi, Karan and Maechling, Philip J. and Jordan, Thomas H. and Deelman, Ewa},
    booktitle = {12th Workshop on Workflows in Support of Large-Scale Science (WORKS'17)},
    year = {2017},
    pages = {},
    doi = {},
    note = {Funding Acknowledgments: NSF 1664162 and 1443047}
    }
  • [PDF] [DOI] R. Ferreira da Silva, S. Callaghan, and E. Deelman, “On the Use of Burst Buffers for Accelerating Data-Intensive Scientific Workflows,” in 12th Workshop on Workflows in Support of Large-Scale Science (WORKS’17), 2017.
    [Bibtex]
    @inproceedings{ferreiradasilva-works-2017,
    title = {On the Use of Burst Buffers for Accelerating Data-Intensive Scientific Workflows},
    author = {Ferreira da Silva, Rafael and Callaghan, Scott and Deelman, Ewa},
    booktitle = {12th Workshop on Workflows in Support of Large-Scale Science (WORKS'17)},
    year = {2017},
    pages = {},
    doi = {10.1145/3150994.3151000},
    note = {Funding Acknowledgments: DOE DE-SC0012636, NSF 1664162}
    }

SC Floor Plan with Booth Locations highlighted can be found here.

145 views


News and Announcements from OSG Operations > Announcing changes to the OSG software release model

After the November 14 release, the software and release team will be moving to a flexible release model [1]. Under this model, we'll be free to do releases on days other than the second Tuesday of every month.

We are making this change to deliver upstream updates more quickly without having to designate them as high-priority software releases. As a user, you shouldn't notice much of a difference since we are aiming for approximately monthly software releases. We will still distribute data releases as soon as they are ready.

Let us know if you have any questions or concerns at help@opensciencegrid.org!

[1] https://opensciencegrid.github.io/technology/policy/flexible-release-model/

News and Announcements from OSG Operations > Announcing OSG CA Certificate Update

We are pleased to announce a data release for the OSG Software Stack.
Data releases do not contain any software changes.

This release contains updated CA Certificates based on IGTF 1.87:
- added new accredited classic DarkMatter Private Root G4 and ICA (AE)
- updated PK-Grid-2007 trust anchor with extended validity period (PK)
- extended validity period for UNAMgrid-ca trust anchor (MX)

Release notes and pointers to more documentation can be found at:

http://opensciencegrid.github.io/docs/release/3.4/release-3-4-4-3/
http://opensciencegrid.github.io/docs/release/3.3/release-3-3-29-3/

Need help? Let us know:

http://opensciencegrid.github.io/docs/common/help/

We welcome feedback on this release!

News and Announcements from OSG Operations > GOC Service Update, Tuesday, November 7th at 14:00 UTC

The GOC will upgrade the following services on November 7th beginning 14:00 UTC. The GOC reserves 8 hours in the unlikely event unexpected problems are encountered.

Glidein
  * Update filebeat on the GOC factory. This will not require a downtime.

Ticket
  * Security notification, CCs email notification adjustment


Condor Project News > HTCondor 8.7.4 released! ( October 31, 2017 )

The HTCondor team is pleased to announce the release of HTCondor 8.7.4. This development series release contains new features that are under development. This release contains all of the bug fixes from the 8.6.7 stable release. Enhancements in the release include: Improvements to DAGMan including support for late job materialization; Updates to condor_annex including improved status reporting; When submitting jobs, HTCondor can now warn about job requirements; Fixed a bug where remote CPU time was not recorded in the history; Improved support for OpenMPI jobs; The high availability daemon now works with IPV6 and shared_port; The HTCondor Python bindings are now available for Python 2 and 3 in pip. Further details can be found in the Development Version History and the Stable Version History. HTCondor 8.7.4 binaries and source code are available from our Downloads page.

Condor Project News > HTCondor 8.6.7 released! ( October 31, 2017 )

The HTCondor team is pleased to announce the release of HTCondor 8.6.7. A stable series release contains significant bug fixes. Highlights of this release are: Fixed a bug where memory limits might not be updated in cgroups; Add SELinux type enforcement rules to allow condor_ssh_to_job to work; Updated systemd configuration to shutdown HTCondor in an orderly fashion; The curl_plugin utility can now do HTTPS transfers; Specifying environment variables now works with the Python Submit class. More details about the fixes can be found in the Version History. HTCondor 8.6.7 binaries and source code are available from our Downloads page.

Condor Project News > NCSA calls on HTCondor partnership to process data for the Dark Energy Survey ( October 30, 2017 )

HPCWire recently published an aritcle on how HTCondor enables NCSA to take raw data from the Dark Energy camera telescope and process and disseminate the results within hours of observations occurring.

Pegasus news feed > Pegasus Contributed to New Gravitational-Wave Detector Discovery

Visualization of a neutron star merger seen in gravity and matter. Photo/Karan Jani-Georgia Tech.

 

A collaboration that began 16 years ago between computer scientists at the USC Information Sciences Institute (ISI) and members of the Laser Interferometer Gravitational-Wave Observatory (LIGO) and Virgo projects is opening up a new window onto the nature of the universe.

Pegasus, a specialized computer program developed by a team of ISI computer scientists led by distributed computing expert Ewa Deelman in collaboration with HTCondor, played an important role in a new astronomy discovery announced by LIGO, Oct. 16 in Washington, D.C.

For the first time, scientists have directly detected gravitational waves — ripples in space-time — in addition to light produced by colliding neutron stars. Detected by two identical LIGO detectors on Aug. 17, this marks the first time that a cosmic event has been viewed in both gravitational waves and light.

The software developed at ISI allowed the LIGO scientists to confirm the signal’s significance by conducting rigorous offline analyses of massive amounts of data. As part of this latest discovery, in August 2017 Pegasus managed almost 4,000 workflows with more than nine million tasks.

Professor Duncan Brown of Syracuse University, a member of the LIGO Scientific Collaboration, said: “Thanks to our collaboration with the Pegasus, OSG, and Condor teams, we can now turn around our offline analyses in days not weeks. This is essential for getting confirmations of low- latency alerts and getting our results out to the world.”

Other benefits of Pegasus include automatic recovery after job failures, the ability to automatically manage data flow, and a dashboard to track progress and identify the sources of job failures.

This is not the first time that Pegasus, a workflow management tool initially designed for astronomers, has helped to propel a new scientific discovery. In 2016, LIGO scientists first discovered gravitational waves, confirming Einstein’s Theory of Relativity. The breakthrough, which received the 2017 Nobel Prize for physics, was made possible in part by Pegasus.

“It is extremely exciting to see years of computer science research and software development have impact on cutting- edge science,” said Deelman.

“Thanks to the funding that Pegasus continues to receive from National Science Foundation and Department of Energy, we hope to improve our solutions and make them available to new science communities.”

 

Source: ISI News

 

171 views


News and Announcements from OSG Operations > GOC Service Update, Tuesday, October 24th at 14:00 UTC

The GOC will upgrade the following services October 24 beginning 14:00 UTC. The GOC reserves 8 hours in the unlikely event unexpected problems are encountered.

Glidein
  * install filebeat on the GOC factory. This will not require a downtime.

All Services
  * Operating system updates, reboots will be required. The usual HA mechanisms will be used, but some services will experience brief outages.

News and Announcements from OSG Operations > Announcing OSG CA Certificate and VO Package Updates

We are pleased to announce a data release for the OSG Software Stack.
Data releases do not contain any software changes.

This release contains updated CA Certificates based on IGTF 1.86:
- updated MaGrid CA with extended validity period (MA)
- removed discontinued pkIRISGrid CA (ES)

This release also contains VO Package v75:
- Add CMS wildcard to default map file

Release notes and pointers to more documentation can be found at:

http://opensciencegrid.github.io/docs/release/3.4/release-3-4-4-2/
http://opensciencegrid.github.io/docs/release/3.3/release-3-3-29-2/

Need help? Let us know:

http://opensciencegrid.github.io/docs/common/help/

We welcome feedback on this release!

News and Announcements from OSG Operations > RESOLVED: Ticket web interface experiencing problems for some users

Colleagues,

The OSG Ticketing interface was experiencing problems for some users.  Users were able to authenticate and view tickets, but unable to update tickets.  Operations identified this problem and patched the ticketing interface.  The interface seems to be operating as expected for all users now.

We apologize for any inconvenience this caused.  Please alert us if you see any more unexpected behavior.

News and Announcements from OSG Operations > Ticket web interface experiencing problems for some users

Colleagues,

The OSG Ticketing interface  is currently experiencing problems for some users.  Users are able to authenticate and view tickets, but unable to update tickets.  Operations has identified this problem and is currently working on the solution. If you must update a ticket immediately, there is a button on the ticket page that says, “Update w/Email.” Clicking this button will set up a message with your mail client so that you can update the ticket.

We apologize for any inconvenience this is causing and will update you when we have more information.

News and Announcements from OSG Operations > Announcing OSG Software version 3.4.4

We are pleased to announce OSG Software version 3.4.4.

Changes to OSG 3.4.4 include:
- Updated gsi-openssh-server to interoperate with clients using OpenSSL 1.1
- Singularity 2.3.2: Now works with Docker's updated registry RESTful API
- HTCondor 8.6.6: Bug fix release
- globus-gridftp-server-control 5.2: Allow 400 responses to stat failures
- osg-ca-scripts now properly declares its dependency on the wget package
- osg-configure works with HTCondor 8.7.2+ and when fetch-crl is missing
- HTCondor 8.7.3 in Upcoming

Release notes and pointers to more documentation can be found at:

http://opensciencegrid.github.io/docs/release/3.4/release-3-4-4/

Need help? Let us know:

http://opensciencegrid.github.io/docs/common/help/

We welcome feedback on this release!

News and Announcements from OSG Operations > Announcing OSG Software version 3.3.29

We are pleased to announce OSG Software version 3.3.29.

Changes to OSG 3.3.29 include:
- voms-admin-server: Updated to fix Apache struts vulnerability
- Updated gsi-openssh-server to interoperate with clients using OpenSSL 1.1
- globus-gridftp-server-control 5.2: Allow 400 responses to stat failures
- osg-ca-scripts now properly requires wget to be installed
- osg-configure works when fetch-crl is missing

Release notes and pointers to more documentation can be found at:

http://opensciencegrid.github.io/docs/release/3.3/release-3-3-29/

Need help? Let us know:

http://opensciencegrid.github.io/docs/common/help/

We welcome feedback on this release!

Pegasus news feed > Nobel Prize-winning discovery on Gravitational Waves came about with contributions from Pegasus

An artist’s impression shows gravitational waves generated by binary neutron stars. (Photo/R. Hurt, Caltech-JPL)

 

By Emily Gersema, USC News

The Nobel Prize-winning discovery that gravitational waves exist in the universe, which in turn further confirmed Albert Einstein’s General Theory of Relativity, was made possible in part by a collaboration with USC computer scientists.

By developing a specialized computer program called Pegasus, a team of USC Information Sciences Institute researchers facilitated the work of scientists who this week won the Nobel Prize in Physics for their discovery of gravitational waves that are powerful enough to ripple throughout space and distort the shape of the cosmos.

While the prize is shared by Rainer Weiss of the Massachusetts Institute of Technology, and Caltech scientists Kip Thorne and Barry Barish, the joy over their achievement is shared by colleagues.

“It’s very exciting to see 16 years of collaboration come to this point,” said Ewa Deelman, who spearheaded Pegasus’ development as a research professor in computer science and research director for the ISI at the USC Viterbi School of Engineering. “Gravitational waves, like the ones emitted by merging black holes, give a unique insight into how the universe is structured and how it developed. It’s a basic question about nature.”

Experimental data

The software developed at ISI supported computational research, such as the type conducted to detect cosmic gravitational waves at the Laser Interferometer Gravitational- wave Observatory (LIGO). It automated the execution of hundreds of thousands of computational tasks on high-end systems, while managing the data flow between tasks and accessing experimental data across wide area networks.

Deelman said the Pegasus program, so named because it was initially designed for astronomers, was first conceived as a virtual data grid that would deliver information based on whether it was available in existing data sets (such as information collected by NASA) or by computing new data on demand.

One challenge that Deelman and her team encountered when designing the program was that the scientists had an idea of their workflow stored in their minds and wanted it enacted exactly that way, not necessarily in the manner suggested by a computer. The computer scientists had to create something that could quickly link the researchers’ ideas to existing data, software and computing resources.

“Pegasus allows you to describe these computational steps [workflow tasks] in an abstract way, providing the logic and data flow between computational steps, and then ties in those abstractions to relevant data sets, software and computational resources that can enact these steps,” Deelman said. “It maps the abstract tasks descriptions onto the available resources and manages their execution in an automated, robust and efficient way.”

Pegasus eliminated the need for developing manual commands or scripts. It was designed to quickly recover from failures and ensure the data are delivered securely and efficiently.

Relatively speaking

LIGO’s discovery marked nearly 100 years since Einstein wrote his General Theory of Relativity, when he suggested that space and time are a single continuum, “space-time,” and that matter and energy could warp the shape of space-time and produce gravity.

Deelman led a team for developing Pegasus that over the years included USC computer scientists Karan Vahi, Mats Rynge, Rajiv Mayani, Rafael Ferreira da Silva, Gideon Juve, Gaurang Mehta and a cohort of PhD and master’s degree candidates to write the software in a way that functioned intuitively and reliably for the scientists. Early on, the team included ISI scientists such as Carl Kesselman, director of the Center for Discovery Informatics at the USC Michelson Center for Convergent Bioscience, which will open next month.

Other researchers from the University of Wisconsin- Madison’s HTCondor team led by Miron Livny also had a hand in the creation of Pegasus and have collaborated on the project since 2003.

“Over the years, Pegasus greatly benefited from collaborations with domain scientists from various projects, including LIGO’s Duncan Brown, Scott Koranda, Kent Blackburn, Peter Couvares and Stuart Anderson, among others. They provided us with real-life computational challenges that make our software relevant and inspiration that keeps us motivated to do better,” Deelman said.

Since Pegasus is an open-source software, it is available for other researchers to download without any licensing fees. Deelman said it has been used for other scientific initiatives, including those at the Southern California Earthquake Center, which uses the software to model seismic hazards in SoCal and predict the flow of an earthquake.

Pegasus also has been used for modeling climate change, to track the extinction of monkeys, create better soybeans and to understand genetic patterns in disease and conditions such as schizophrenia.

“With Pegasus, I think there will be many more new exciting discoveries,” Deelman said.

 

View this story on USC News

 

233 views


Condor Project News > HTCondor role in Nobel-winning gravitational wave discovery ( October 5, 2017 )

The Laser Interferometer Gravitational-Wave Observatory (LIGO) unlocked the final door to Albert Einstein's Theory of Relativity, winning the Nobel prize this week and proving that gravitational waves produce ripples through space and time. Since 2004, HTCondor has been a core part of the data analysis effort. See this recent article , as well as this one, for more details.

News and Announcements from OSG Operations > GOC Service Update - Monday, October 9th at 14:00 UTC

The GOC will upgrade the following services beginning Monday, October 9 at 14:00 UTC. The GOC reserves 8 hours in the unlikely event unexpected problems are encountered.

MyOSG
  * Configure backup scripts

Event
  * Enable stomp endpoint
  * Add several user accounts

Oasis-login
  * Complete gsissh configuration
  * Update to Centos7

Ticket
  * Ticket status update (close/resolve) addition


Subscribe