FastDL: Timely Visualization and Analysis of Large Data Sets Using IDL and Parallel Computing
Often experiments and simulations generate large data sets that require immediate processing. Scientists exploring fluid and particle dynamics, high-energy and plasma physics, astrophysics and space sciences, biophysics, protein folding and medical science are challenged to visualize and analyze increasingly complex data. Many technical professionals rely on the Interactive Data Language (IDL) from Exelis Visual Information Solutions, Inc. to visualize and analyze these large data sets.
However some analyses cannot be practically accomplished on a workstation or server with only symmetric multi-processing. And while distributed computing promises cost-effective computing power, IDL alone does not naturally take advantage of this parallel environment.
To bridge the gap between IDL and parallel computing, Tech-X Corporation has developed FastDL. With FastDL, scientists and developers can run IDL visualization and analyses applications in parallel, significantly shortening the time required to get results.
Because not every large scale visualization and analysis problem can be solved using the same parallel computing paradigm, FastDL offers two independent components that address the computational needs of parallel data analysis and visualization:
TaskDL - designed for problems where computations on each node are independent of one another and are performed on different data (task farming).
mpiDL - appropriate for complex problems where nodes need to communicate during the analysis and may need to perform different tasks.
TaskDL helps scientists and developers quickly re-purpose IDL applications to run in parallel using a task farm paradigm. Within minutes and with minimal code changes, users can run serial IDL applications in parallel. TaskDL is appropriate for improving the performance of applications where parallelized tasks do not need to communicate with one another, such as movie frame rendering and Monte Carlo simulations.
How Does TaskDL Work?
TaskDL is an IDL task farming system that uses a central task server to distribute tasks amongst multiple worker nodes running IDL on local or remote machines. Each worker node operates independently, contributing its maximum computing power without being held back by high loads or problems with other worker nodes.
TaskDL sessions can be created from within any IDL program or interactive IDL session, where users can start the task server on a specified host machine, launch or terminate IDL workers on remote hosts, or query the server for task lists and task status. IDL tasks, which may be any executable IDL command or procedure with its associated arguments, may be sent to the server via an easy-to-use object interface. TaskDL offers mechanisms to define task priorities and task dependencies.
mpiDL is designed to meet the needs of scientists and developers who are familiar with parallel computing and want to use IDL for data visualization and analysis. This add-on library implements the Message Passing Interface (MPI) standard in IDL. Through message passing, mpiDL lets users delegate computation sub-tasks to machines in the cluster. Examples of problems well-suited to mpiDL are finite element analysis and analysis of distributed data sets created by parallel plasma or weather simulations.
How Does mpiDL Work?
The MPI library of communication routines is the standard message-passing interface for distributed-memory parallel computing. Tech-X Corporation's mpiDL is an add-on library that implements MPI as native IDL function calls, helping scientists and developers familiar with parallel computing quickly leverage the power of IDL.
With mpiDL, parallel programmers can write IDL programs that call MPI functions using the same approach they would use when writing C or Fortran programs. mpiDL also gives developers access to built-in specialized parallel functionality based on collections of primitive MPI communication and data types. Developers who are new to parallel programming using explicit message passing can get up to speed quickly by modifying the mpiDL examples to create their own parallel IDL programs.
Learn More About FastDL
- Success Stories
- System Requirements