Entity

Time filter

Source Type

Texas, Texas, United States

News Article
Site: http://www.scientificcomputing.com/rss-feeds/all/rss.xml/all

When the National Science Foundation (NSF) announced the first-ever direct observation of gravitational waves, they confirmed a prediction Albert Einstein made when he published his General Theory of Relativity 100 years ago. Einstein never expected gravitational waves to be observed. These ripples in the fabric of spacetime are triggered by events occurring so far away and so long ago that, by the time they reached earth, he figured their signals would be too weak to measure. But Einstein didn’t foresee the development of the Laser Interferometer Gravity-Wave Observatory (LIGO) detectors — instruments so sensitive they can measure signals that change a fraction of the width of an atom’s nucleus over a distance of more than two miles. He didn’t anticipate Moore’s Law and the enormous computational resources that would enable scientists to extract and analyze these signals. With LIGO’s success, scientists have yet again confirmed Einstein’s genius. They’ve also opened a window on a whole new field of study — gravitational astrophysics — that will provide exciting new ways to light up the dark aspects of the universe. The LIGO team’s discovery and its ongoing work will deepen our understanding of phenomena such as black holes, neutron stars and, potentially, even the big bang. And it will lead to new insights we can’t yet imagine. “This discovery is the first time we’ve been able to hear the cosmos communicating to us using gravitational waves,” Dr. David Reitze, Executive Director of LIGO Laboratory told the U.S. House Committee on Science, Space and Technology.1 “I am quite confident that it won’t be the last time. Even more exciting, we will see, or rather hear, something completely unexpected from the universe that will revolutionize our understanding of the universe. We cannot wait to get there.” The LIGO experiment is awe-inspiring at both ends of the size spectrum. The gravitational waves captured by the LIGO detectors resulted from two black holes orbiting each other, colliding and merging approximately 1.3 billion years ago and 1.3 billion light-years away. The black holes, which were about 29 and 36 times the mass of our sun, collided at nearly half the speed of light. Their collision converted mass to the energy equivalent of three suns, producing the gravitational wave that LIGO detected. In the fraction of a second before the merger, they radiated with an energy 50 times greater than all the other stars in the visible universe. The LIGO equipment that observed the waves are the most sensitive measuring devices ever made, according to David Shoemaker, MIT LIGO Laboratory Director and Leader of the Advanced LIGO upgrade. Located at LIGO observatories in Louisiana and Washington State, each detector has an L-shaped interferometer that splits laser light, beams it in a vacuum down the 2.5 mile sides of the L, bounces it off mirrors at the ends of the L, measures how long each beam takes to travel and return, recombines it, and analyzes the “interference pattern.” Distances are measured over the 2.5 miles at a precision of 1/1019 of a meter, or one ten-thousandth the diameter of a proton. “The LIGO experiment gets to very fundamental physics,” says Stuart B. Anderson, Senior Research Scientist at Caltech and head of computing for LIGO. “When this gravitational wave passed through our solar system, it effectively changed the distance between the earth and the sun by the width of one atom. To be able to make this exquisitely precise measurement about something that’s so fundamental to our universe, in such a high-risk, high-impact area — it’s incredibly exciting.” The LIGO success is a lesson in visionary leadership, commitment and collaboration. “NSF showed foresight and determination in pursuing the goal of detecting gravitational waves and learning what these waves will be able to tell us,” Shoemaker says. “As with a lot of transformational science, you’re dealing with large-scale challenges. You have to develop new technologies and new workplace skills. You have to think in terms of longer time scales for development. NSF was far-sighted, and it made those commitments, including supporting non-teaching academic research scientists like Stuart and me.” International collaboration has been a central element. MIT and Caltech have been the pathfinders, beginning with NSF-funded projects since the 1970s to explore interferometer research. A broader, self-governing organization, the LIGO Scientific Collaboration (LSC), formed in 1997 to exploit the scientific potential of the LIGO instruments that were then under construction. Caltech and MIT play leading roles in the LSC, with responsibility for designing the equipment, creating the experiment, and operating the hardware. The LSC today includes more than 1,000 scientists at 90 universities and research institutions in 15 countries. “LSC is a big happy laboratory family where people work in collaboration and synergy across multiple labs and multiple universities,” says Shoemaker. “It’s a wonderfully distributed system with a great richness to the collaboration.” Massive computing to validate the signal and advance the science Recording a gravitational signal is one thing. LIGO teams also use sophisticated algorithms and massive computing resources to identify and validate the signal and explore its scientific significance. “Large-scale computing is absolutely key,” Anderson says. “We have very complicated workflows to execute, including one that has anywhere from 100,000 to 1 million compute tasks that need to be organized and executed. We ran something like 50 million core hours total to extract the signal and pull the science out of the data stream. We use a highly distributed computing environment with approximately 20,000 cores, and 90 percent are Intel processors.” The gravitational wave reading came on September 14, shortly after a major upgrade to the next-generation interferometer known as Advanced LIGO or ALIGO. “We had ramped up to take three months of readings and, almost immediately, we saw this very loud, very bright and highly statistically significant signal,” Anderson recalls. “It was a loud, unambiguous signal, but because it came so early in the run, it seemed almost too good to be true. For an extraordinary claim, there’s an extraordinary burden of proof, so we worked very hard and went through the data extremely carefully to try to prove ourselves wrong.” Between the September signal capture and NSF’s February 11, 2016, announcement, LSC researchers performed three major computational tasks. The first involves sifting the data from 100,000 other channels that are recorded simultaneously with gravitational wave channel. Scrutinizing these channels helps eliminate the effects of something like a minor earthquake or even a truck driving into a parking lot contaminating the data. In the second task, researchers conduct computationally intensive pseudo-experiments to evaluate the statistical significance of their signal capture. These experiments build confidence in the signal’s validity by allowing scientists to see how much louder it is compared to false alarms and noise fluctuations. With the signal validated, the third task focuses on determining what events produced the signal and how far it traveled. This work extracts the detailed parameters of the signal and matches them against models that have been developed based on relativity theory. The end result was the NSF announcement, a peer-reviewed paper in Physical Review Letters, celebrations by scientists and citizens around the world, and even appearances on popular television shows. Many observers say the team’s science leaders can also clear space on their shelves for a Nobel Prize. Reflecting the LSC’s broad network of participants, much of LIGO’s computing infrastructure is distributed around the world and performed by LSC members. Low-latency computing, which must process incoming data and keep pace as it is received, is performed on dedicated systems at the observatories, Caltech, and redundant sites in the US and Europe. Other workflows are less time-sensitive, and many are embarrassingly parallel and loosely coupled. This makes it possible to distribute the data, matching the workload to the most suitable platform and having different systems examine different portions of the parameter space. In addition to dedicated resources at Caltech and other LSC institutions, LIGO codes run on national multi-petaflops systems funded through NSF’s Extreme Science and Engineering Discovery Environment (XSEDE) program, such as the Stampede supercomputer at Texas Advanced Computing Center — a Dell PowerEdge cluster equipped with Intel Xeon Phi coprocessors — and the Comet system at the San Diego Supercomputer Center — a Dell-integrated cluster using the Intel Xeon processor E5-2600 v3 family  as well as 36 NVIDIA GPU nodes. LIGO also captures unused cycles from campus grids using NSF and the Department of Energy’s Open Science Grid (OSG), as well as from volunteer computing from the Einstein at Home project. Compute clusters access cached copies of the data over a variety of file systems. The central data archive, housed at Caltech, holds 5.3 PB of observational data and 1.1 PB of analysis results. The archive uses Oracle Hierarchical Storage Management (HSM) to manage storage tiering between disk and tape. The software infrastructure includes the HTCondor workload management system, Pegasus Workflow Management System, and Open Science Grid Software. The Python open source programming language, which Intel software experts have contributed to developing, is in heavy use. Even with such extensive resources, the LIGO workloads are compute-constrained — and rising. “Our science is computation-bound,” Anderson says. “If I had access to all the Intel computers around the world right now, I would use them. There are codes that would benefit from being run with greater sensitivity. As it is, we make compromises to get the most science we can out of the available computing, and we try to get as much computing as we can.” Much of the work involves single-precision, floating-point operations, with heavy use of numerical libraries including the Intel Math Kernel Library (Intel MKL) and the FFTW open source C subroutine library for computing discrete Fourier transforms. LIGO scientists gained a tenfold speedup for parameter extraction by optimizing LIGO codes for Intel MKL. They used Stampede for the optimization work, and Anderson says the improvements transferred well to other systems. “Because of those improvements, we could search all the parameter space we wanted to,” he comments. “Without that, we would not have been able to analyze all the data and keep up with it as it was coming in.” Moving to the Advanced LIGO instrument increased the project’s computational load, in both the number of tasks and total number of floating-point operations, by more than an order of magnitude. Among the contributing factors, the new instrument is more sensitive at lower frequencies, bringing potential events into the observable band for longer and providing a larger and more data-rich parameter space to search. Caltech upgraded its supercomputer to keep pace, adding Intel Xeon processor E3 and E5 v3 families and Intel Solid-State Drives (Intel SSDs). Workflows that need local temporary databases to manage their results got a particular boost from using the Intel SSDs as a local cache, according to Anderson. Caltech also uses some Intel Xeon Phi coprocessors, and Anderson is exploring using more of them going forward. “I’m excited that Intel Xeon Phi processors code name Knight’s Landing will be a cost-effective way to get access to lots of floating-point engines and run the same code as the Intel Xeon processor, so we don’t have to spend a lot of human resources to port it over and get it working,” Anderson says. “We’ll be well-positioned to take advantage of the Intel AVX-512 instruction set.” As work goes forward, LIGO’s need for compute cycles will climb even more sharply.  The instrumentation team will continue to tune the ALIGO detector for increased sensitivity, expanding the parameter space further. Additional gravitational wave detectors are coming on line around the world and in space, offering greater opportunities for collaboration and further insights into the workings of the universe. Astrophysicists have a long list of questions they want to answer and ways they want to tie the study of gravitational waves to other observational techniques and other fields of astronomy. Then there are the surprises. “When you open a brand new window, you see things you didn’t expect,” says Anderson. “That’s the really exciting aspect to this. We have to be prepared to discover things we don’t know about, and we really don’t know what their computational costs will be. From the perspective of software and middleware, we have to be agile enough to follow up on new things. We need to do it as efficiently as possible and at reasonable costs.” Programs such as the National Strategic Computing Initiative (NSCI), which aims to accelerate the development of exascale computers, may prove key to building on LIGO’s discoveries. “As the Advanced LIGO instruments are tuned up over the next few years to probe an order of magnitude larger volume of the universe, the corresponding computational challenge to search those data will also grow by more than an order of magnitude,” Anderson says. “In addition to the large high-throughput computing needs to search LIGO data for gravitational waves, high-performance computing is critical to solving Einstein’s equations numerically to provide information on both what LIGO ought to look for and in extracting the detailed information available in the waveforms recorded by LIGO. This HPC task will become significantly more challenging when LIGO detects other types of systems than the initial discovery of two merging black holes. For example, it will require world-class HPC resources to model the complicated physics of matter under extreme conditions, such as relativistically accurate simulations of merging neutron stars as they are ripped apart by gravitational forces, or to accurately simulate a three-dimensional supernova explosion.” At a total investment of $1.1 billion over the last three decades, LIGO is delivering benefits beyond the scientific impact. A major area of impact is in technology innovation and technology transfer to the private sector. “To make LIGO work, we had to develop the world’s most stable lasers, the world’s best mirrors, some of the world’s largest vacuum systems, as well as push the frontiers of quantum science and high performance computing,” Reitze told the Science, Space and Technology Committee. “LIGO advances the state-of-the-art in every technology it uses, and it uses lots of technology. We have partnered with many commercial technology firms in the U.S. and abroad to produce the incredible technology that LIGO uses.” LIGO is also an engine of workforce development, educating scientists who have either stayed with the program or carried their advanced skills to national laboratories, Silicon Valley, and companies in sectors such as biotechnology, aerospace and telecommunications. The excitement generated by the LIGO discoveries is helping to inspire a new generation of students, raise appreciation of science, and offer each of us a deeper sense of awe at the complexity of the universe. “Einstein was a fascinating figure,” Shoemaker says in conclusion. “The LIGO announcement once again shows that he was visionary in an almost unimaginable way. I hope taxpayers and organizations who helped support this process feel a sense of satisfaction from their participation in the endeavor. They’ve been part of something important.”


News Article
Site: http://www.scientificcomputing.com/rss-feeds/all/rss.xml/all

Last year, President Obama announced the National Strategic Computing Initiative (NSCI), an executive order to increase research, development and deployment of high performance computing (HPC) in the United States, with the National Science Foundation, the Department of Energy and the Department of Defense as the lead agencies. One of NSCI's objectives is to accelerate research and development that can lead to future exascale computing systems — computers capable of performing one billion billion calculations per second (also known as an exaflop). Exascale computers will advance research, enhance national security and give the U.S. a competitive economic advantage. Experts believe simply improving existing technologies and architectures will not get us to exascale levels. Instead, researchers will need to rethink the entire computing paradigm — from power, to memory, to system software — to make exascale systems a reality. The Argo Project is a three-year collaborative effort, funded by the Department of Energy, to develop a new approach for extreme-scale system software. The project involves the efforts of 40 researchers from three national laboratories and four universities working to design and prototype an exascale operating system and the software to make it useful. To test their new ideas, the research team is using Chameleon, an experimental environment for large-scale cloud computing research supported by the National Science Foundation and hosted by the University of Chicago and the Texas Advanced Computing Center (TACC). Chameleon — funded by a $10 million award from the NSFFutureCloud program — is a re-configurable testbed that lets the research community experiment with novel cloud computing architectures and pursue new, architecturally-enabled applications of cloud computing. "Cloud computing has become a dominant method of providing computing infrastructure for Internet services,” said Jack Brassil, a program officer in NSF's division of Computer and Network Systems. "But to design new and innovative compute clouds and the applications they will run, academic researchers need much greater control, diversity and visibility into the hardware and software infrastructure than is available with commercial cloud systems today." The NSFFutureCloud testbed provides the types of capabilities Brassil described. Using Chameleon, the team is testing four key aspects of the future system: Chameleon's unique, reconfigurable infrastructure lets researchers bypass some issues that would have come up if the team was running the project on a typical high-performance computing system. For instance, developing the Node Operating System requires researchers to change the operating system kernel — the computer program that controls all the hardware components of a system and allocates them to applications. "There are not a lot of places where we can do that," said Swann Perarnau, a postdoctoral researcher at Argonne National Laboratory and collaborator on the Argo Project. "HPC machines in production are strictly controlled, and nobody will let us modify such a critical component." However, Chameleon lets scientists modify and control the system from top to bottom, allowing it to support a wide variety of cloud research and methods and architectures not available elsewhere. "The Argo project didn't have the right hardware nor the manpower to maintain the infrastructure needed for proper integration and testing of the entire software stack," Perarnau added. "While we had full access to a small cluster, I think we saved weeks of additional system setup time, and many hours of maintenance work, switching to Chameleon." One of the major challenges in reaching exascale is energy usage and cost. During last year's Supercomputing Conference, the researchers demonstrated the ability to dynamically control the power usage of 20 nodes during a live demonstration running on Chameleon. They released a paper describing their approach to power management for future exascale systems and will present the results at the Twelfth Workshop on High-Performance, Power-Aware Computing (HPPAC'16) in May 2016. The Argo team is working with industry partners, including Cray, Intel and IBM, to explore which techniques and features would be best suited for the Department of Energy’s next supercomputer. "Argo was founded to design and prototype exascale operating systems and runtime software," Perarnau said. "We believe some of the new techniques and tools we have developed can be tested on petascale systems and refined for exascale platforms."


News Article
Site: http://www.scientificcomputing.com/rss-feeds/all/rss.xml/all

When a hail storm moved through Fort Worth, TX, on May 5, 1995, it battered the highly populated area with hail up to four inches in diameter and struck a local outdoor festival known as the Fort Worth Mayfest. The Mayfest storm was one of the costliest hailstorms in U.S history, causing more than $2 billion in damage and injuring at least 100 people. Scientists know that storms with a rotating updraft on their southwestern sides — which are particularly common in the spring on the U.S. southern plains — are associated with the biggest, most severe tornadoes and also produce a lot of large hail. However, clear ideas on how they form and how to predict these events in advance have proven elusive. A team based at University of Oklahoma (OU) working on the Severe Hail Analysis, Representation and Prediction (SHARP) project works to solve that mystery, with support from the National Science Foundation (NSF). Performing experimental weather forecasts using the Stampede supercomputer at the Texas Advanced Computing Center, researchers have gained a better understanding of the conditions that cause severe hail to form, and are producing predictions with far greater accuracy than those currently used operationally. To predict hail storms, or weather in general, scientists have developed mathematically based physics models of the atmosphere and the complex processes within, and computer codes that represent these physical processes on a grid consisting of millions of points. Numerical models in the form of computer codes are integrated forward in time starting from the observed current conditions to determine how a weather system will evolve and whether a serious storm will form. Because of the wide range of spatial and temporal scales that numerical weather predictions must cover and the fast turnaround required, they are almost always run on powerful supercomputers. The finer the resolution of the grid used to simulate the phenomena, the more accurate the forecast; but the more accurate the forecast, the more computation required. The highest-resolution National Weather Service's official forecasts have grid spacing of one point for every three kilometers. The model the Oklahoma team is using in the SHARP project, on the other hand, uses one grid point for every 500 meters — six times more resolved in the horizontal directions. "This lets us simulate the storms with a lot higher accuracy," says Nathan Snook, an OU research scientist. "But the trade-off is, to do that, we need a lot of computing power — more than 100 times that of three-kilometer simulations. Which is why we need Stampede." Stampede is currently one of the most powerful supercomputers in the U.S. for open science research and serves as an important part of NSF's portfolio of advanced cyberinfrastructure resources, enabling cutting-edge computational and data-intensive science and engineering research nationwide. According to Snook, there's a major effort underway to move to a "warning on forecast" paradigm — that is, to use computer-model-based, short-term forecasts to predict what will happen over the next several hours and use those predictions to warn the public, as opposed to warning only when storms form and are observed. "How do we get the models good enough that we can warn the public based on them?" Snook asks. "That's the ultimate goal of what we want to do — get to the point where we can make hail forecasts two hours in advance. 'A storm is likely to move into downtown Dallas, now is a good time to act.'" With such a system in place, it might be possible to prevent injuries to vulnerable people, divert or move planes into hangers and protect cars and other property. Looking at past storms to predict future ones To study the problem, the team first reviews the previous season's storms to identify the best cases to study. They then perform numerical experiments to see if their models can predict these storms better than the original forecasts using new, improved techniques. The idea is to ultimately transition the higher-resolution models they are testing into operation in the future. Now in the third year of their hail forecasting project, the researchers are getting promising results. Studying the storms that produced the May 20, 2013 Oklahoma–Moore tornado that led to 24 deaths, destroyed 1,150 homes and resulted in an estimated $2 billion in damage, they developed zero to 90-minute hail forecasts that captured the storm's impact better than the National Weather Service forecasts produced at the time. "The storms in the model move faster than the actual storms," Snook says. "But the model accurately predicted which three storms would produce strong hail and the path they would take." The models required Stampede to solve multiple fluid dynamics equations at millions of grid points and also incorporate the physics of precipitation, turbulence, radiation from the sun and energy changes from the ground. Moreover, the researchers had to simulate the storm multiple times — as an ensemble — to estimate and reduce the uncertainty in the data and in the physics of the weather phenomena themselves. "Performing all of these calculations on millions of points, multiple times every second, requires a massive amount of computing resources," Snook says. The team used more than a million computing hours on Stampede for the experiments and additional time on the Darter system at the National Institute for Computational Science for more recent forecasts. The resources were provided through the NSF-supported Extreme Science and Engineering Discovery Environment (XSEDE) program, which acts as a single virtual system that scientists can use to interactively share computing resources, data and expertise. Though the ultimate impacts of the numerical experiments will take some time to realize, its potential motivates Snook and the severe hail prediction team. "This has the potential to change the way people look at severe weather predictions," Snook says. "Five or 10 years down the road, when we have a system that can tell you that there's a severe hail storm coming hours in advance, and to be able to trust that — it will change how we see severe weather. Instead of running for shelter, you'll know there's a storm coming and can schedule your afternoon." Ming Xue, the leader of the project and director of the Center for Analysis and Prediction of Storms (CAPS) at OU, gave a similar assessment. "Given the promise shown by the research and the ever-increasing computing power, numerical prediction of hailstorms and warnings issued based on the model forecasts, with a couple of hours of lead time, may indeed be realized operationally in a not-too-distant future, and the forecasts will also be accompanied by information on how certain the forecasts are." The team published its results in the proceedings of the 20th Conference on Integrated Observing and Assimilation Systems for Atmosphere, Oceans and Land Surface (IOAS-AOLS); they will also be published in an upcoming issue of the American Meteorological Society journal Weather and Forecasting. "Severe hail events can have significant economic and safety impacts," says Nicholas F. Anderson, program officer in NSF's Division of Atmospheric and Geospace Sciences. "The work being done by SHARP project scientists is a step towards improving forecasts and providing better warnings for the public."


News Article
Site: http://www.scientificcomputing.com/rss-feeds/all/rss.xml/all

The Texas Advanced Computing Center (TACC) at The University of Texas at Austin (UT Austin) announced that the Lonestar 5 supercomputer is in full production and is ready to contribute to advancing science across the state of Texas. Managed by TACC, the center's second petaflop system is primed to be a leading computing resource for the engineering and science research community. The supercomputer is sponsored by UT System in partnership with UT Austin, Texas Tech University, Texas A&M University, and the Institute for Computational Engineering and Sciences (ICES) and the Center for Space Research at The University of Texas at Austin. The technology partners are Cray, Intel and DataDirect Networks. Lonestar 5 is designed for academic researchers, serving as the primary high performance computing resource in the UT Research Cyberinfrastructure (UTRC) initiative. Sponsored by The University of Texas System (UT System), UTRC provides a combination of advanced computational systems, a large data storage opportunity, and high bandwidth data access. UTRC enables researchers within all 14 UT System institutions to collaborate with each other and compete at the forefront of science and discovery. The new Lonestar 5 Cray XC40 supercomputer, which contains more than 30,000 Intel Xeon processing cores from the E5-2600 v3 product family, provides a peak performance of 1.25 petaflops. With 24 processing cores per compute node, Lonestar 5 follows the trend of more cores per node that the industry sees in every generation of microprocessors. The system is the fifth in a long line of systems available for Texas researchers, dating back over 15 years to the original Lonestar 1 system (also a Cray). The system will continue to serve its mainstay user communities with an emphasis on addressing a wide variety of research areas in engineering, medicine and the sciences. A number of researchers have been using Lonestar 5 in an "early user" mode over the last few months. Researchers from UT System institutions and contributing partners wishing to request access to Lonestar 5 should do so via the TACC User Portal.


News Article
Site: http://phys.org/technology-news/

"When you're in the world of data, there are rocks and bumps in the way, and a lot of things that you have to take care of," said Niall Gaffney, a former Hubble Space Telescope scientist who now heads the Data Intensive Computing group at the Texas Advanced Computing Center (TACC). Gaffney led the effort to bring online a new kind of supercomputer, called Wrangler. Like the old Western cowboys who tamed wild horses, Wrangler tames beasts of big data, such as computing problems that involve analyzing thousands of files that need to be quickly opened, examined and cross-correlated. Wrangler fills a gap in the supercomputing resources of XSEDE, the Extreme Science and Engineering Discovery Environment, supported by the National Science Foundation (NSF). XSEDE is a collection of advanced digital resources that scientists can easily use to share and analyze the massive datasets being produced in nearly every field of research today. In 2013, NSF awarded TACC and its academic partners Indiana University and the University of Chicago $11.2 million to build and operate Wrangler, a supercomputer to handle data-intensive high performance computing. Wrangler was designed to work closely with the Stampede supercomputer, the 10th most powerful in the world according to the bi-annual Top500 list, and the flagship of TACC at The University of Texas at Austin (UT Austin). Stampede has computed over six million jobs for open science since it came online in 2013. "We kept a lot of what was good with systems like Stampede," said Gaffney, "but added new things to it like a very large flash storage system, a very large distributed spinning disc storage system, and high speed network access. This allows people who have data problems that weren't being fulfilled by systems like Stampede and Lonestar to be able to do those in ways that they never could before." Gaffney made the analogy that supercomputers like Stampede are like racing sports cars, with fantastic compute engines optimized for going fast on smooth, well-defined race-tracks. Wrangler, on the other hand, is built like a rally car to go fast on unpaved, bumpy roads with muddy gravel. "If you take a Ferrari off-road you may want to change the way that the suspension is done," Gaffney said. "You want to change the way that the entire car is put together, even though it uses the same components, to build something suitable for people who have a different job." At the heart of Wrangler lie 600 terabytes of flash memory shared via PCI interconnect across Wrangler's over 3,000 Haswell compute cores. "All parts of the system can access the same storage," Gaffney said. "They can work in parallel together on the data that are stored inside this high-speed storage system to get larger results they couldn't get otherwise." This massive amount of flash storage comes from DSSD, a startup co-founded by Andy Bechtolsheim of Sun Microsystems fame and acquired in May of 2015 by EMC. Bechtolsheim's influence at TACC goes back to the 'Magnum' Infiniband network switch he led design on for the now-decommissioned Ranger supercomputer, the predecessor to Stampede. What's new is that DSSD took a shortcut between the CPU and the data. "The connection from the brain of the computer goes directly to the storage system. There's no translation in between," Gaffney said. "It actually allows people to compute directly with some of the fastest storage that you can get your hands on, with no bottlenecks in between." Gaffney recalled the hang-up scientists had with code called OrthoMCL, which combs through DNA sequences to find common genetic ancestry in seemingly unrelated species. The problem was that OrthoMCL let loose databases wild as a bucking bronco. "It generates a very large database and then runs computational programs outside and has to interact with this database," said biologist Rebecca Young of the Department of Integrative Biology and the Center for Computational Biology and Bioinformatics at UT Austin. She added, "That's not what Lonestar and Stampede and some of the other TACC resources were set up for." Young recounted how at first, using OrthoMCL with online resources, she was only able to pull out 350 comparable genes across 10 species. "When I run OrthoMCL on Wrangler, I'm able to get almost 2,000 genes that are comparable across the species," Young said. "This is an enormous improvement from what is already available. What we're looking to do with OrthoMCL is to allow us to make an increasing number of comparisons across species when we're looking at these very divergent, these very ancient species separated by 450 million years of evolution." "We were able to go through all of these work cases in anywhere between 15 minutes and 6 hours," Gaffney said. "This is a game changer." Gaffney added that getting results quickly lets scientists explore new and deeper questions by working with larger collections of data and driving previously unattainable discoveries. Computer scientist Joshua New with the Oak Ridge National Laboratory (ORNL) hopes to take advantage of Wrangler's ability to tame big data. New is the principal investigator of the Autotune project, which creates a software version of a building and calibrates the model with over 3,000 different data inputs from sources like utility bills to generate useful information such as what an optimal energy-efficient retrofit might be. "Wrangler has enough horsepower that we can run some very large studies and get meaningful results in a single run," New said. He currently uses the Titan supercomputer of ORNL to run 500,000 simulations and write 45 TB of data to disk in 68 minutes. He said he wants to scale out his parametric studies to simulate all 125.1 million buildings in the U.S. "I think that Wrangler fills a specific niche for us in that we're turning our analysis into an end-to-end workflow, where we define what parameters we want to vary," New said. "It creates the sampling matrix. It creates the input files. It does the computationally challenging task of running all the simulations in parallel. It creates the output. Then we run our artificial intelligence and statistic techniques to analyze that data on the back end. Doing that from beginning to end as a solid workflow on Wrangler is something that we're very excited about." When Gaffney talks about storage on Wrangler, he's talking about is a lot of data storage—a 10 petabyte Lustre-based file system hosted at TACC and replicated at Indiana University. "We want to preserve data," Gaffney said. "The system for Wrangler has been set up for making data a first-class citizen amongst what people do for research, allowing one to hold onto data and curate, share, and work with people with it. Those are the founding tenants of what we wanted to do with Wrangler." "Data is really the biggest challenge with our project," said UT Austin astronomer Steve Finkelstein. His NSF-funded project is called HETDEX, the Hobby-Eberly Telescope Dark Energy Experiment. It's the largest survey of galaxies ever attempted. Scientists expect HETDEX to map over a million galaxies in three dimensions, in the process discovering thousands of new galaxies. The main goal is to study dark energy, a mysterious force pushing galaxies apart. "Every single night that we observe—and we plan to observe more or less every single night for at least three years—we're going to make 200 GB of data," Finkelstein said. It'll measure the spectra of 34,000 points of skylight every six minutes. "On Wrangler is our pipeline," Finkelstein said. "It's going to live there. As the data comes in, it's going to have a little routine that basically looks for new data, and as it comes in every six minutes or so it will process it. By the end of the night it will actually be able to take all the data together to find new galaxies." Another example of a new HPC user Wrangler enables is an NSF-funded science initiative called PaleoCore. It hopes to take advantage of Wrangler's swiftness with databases to build a repository for scientists to dig through geospatially-aware data on all fossils related to human origins. This would combine older digital collections in formats like Excel worksheets and SQL databases with newer ways of gathering data such as real-time fossil GPS information collected from iPhones or iPads. "We're looking at big opportunities in linked open data," PaleoCore principal investigator Denne Reed said. Reed is an associate professor in the Department of Anthropology at UT Austin. Linked open data allows for queries to get meaning from the relationships of seemingly disparate pieces of data. "Wrangler is the type of platform that enables that," Reed said. "It enables us to store large amounts of data, both in terms of photo imagery, satellite imagery and related things that go along with geospatial data. Then also, it allows us to start looking at ways to effectively link those data with other data repositories in real time." Wrangler's shared memory supports data analytics on the Hadoop and Apache Spark frameworks. "Hadoop is a big buzzword in all of data science at this point," Gaffney said. "We have all of that and are able to configure the system to be able to essentially be like the Google Search engines are today in data centers. The big difference is that we are servicing a few people at a time, as opposed to Google." Users bring data in and out of Wrangler in one of the fastest ways possible. Wrangler connects to Internet2, an optical network which provides 100 gigabytes per second worth of throughput to most of the other academic institutions around the country. What's more, TACC has tools and techniques to transfer their data in parallel. "It's sort of like being at the supermarket," explained Gaffney. "If there's only one lane open, it is just as fast as one person checking you out. But if you go in and have 15 lanes open, you can spread that traffic across and get more people through in less time." Biologists, astronomers, energy efficiency experts, and paleontologists are just a small slice of the new user community Wrangler aims to attract. Wrangler is also more web-enabled than typically found in high performance computing. A web portal allows users to manage the system and gives the ability to use web interfaces such as VNC, RStudio, and Jupyter Notebooks to support more desktop-like user interactions with the system. "We need these bigger systems for science," Gaffney said. "We need more kinds of systems. And we need more kinds of users. That's where we're pushing towards with these sort of portals. This is going to be the new face, I believe, for many of these systems that we're moving forward with now. Much more web-driven, much more graphical, much less command line driven. " "The NSF shares with TACC great pride in Wrangler's continuing delivery of world-leading technical throughput performance as an operational resource available to the open science community in specific characteristics most responsive to advance data-focused research," said Robert Chadduck, the program officer overseeing the NSF award. Wrangler is primed to lead the way in computing the bumpy world of data-intensive science research. "There are some great systems and great researchers out there who are doing groundbreaking and very important work on data, to change the way we live and to change the world," Gaffney said. "Wrangler is pushing forth on the sharing of these results, so that everybody can see what's going on." Explore further: Texas Stampede supercomputer to join the eXtreme Digital (XD) program

Discover hidden collaborations