Sun Grid Engine 6.2 Update 5 is a feature update release with significant additions including data-aware scheduling, improved public cloud adapter technology, and new commands for invoking power-saving software. Additionally, Update 5 includes improvements to the Sun Grid Engine Service Domain Manager and Sun Grid Engine Inspect modules.
A summary of the new features follows:
- Data-aware job scheduling: Efficiently run Hadoop Map-Reduce jobs in a Sun Grid Engine cluster. The Sun Grid Engine qmaster is made aware of HDFS data locality and of
the data needs of jobs and will schedule the Hadoop jobs accordingly. After being assigned to a node by Sun Grid Engine, the actual processing of the job is handed over to the Hadoop framework.
- Service Domain Manager (SDM) cloud adapter improvements: Provides the ability to expand the cluster size to execute jobs on additional cloud hosts provided by the Amazon Elastic Compute Cloud (EC2). It is now possible to extend the local cluster with virtual nodes based on different Amazon EC2 AMIs. Furthermore, the user feedback on the actions executed on the cloud resources has been improved.
- Power saving improvements: A Cloud Service adapter can be configured for power savings, based on Service Level Objectives (SLOs). New commands such as "showCloudHosts", "startupCloudHosts" and "shutdownCloudHost" can be used to create new power saving scripts.
- Sun Grid Engine Inspect improvements: Inspect is a monitoring and configuration tool that provides a flexible interface for viewing current and historical data about your Sun Grid Engine cluster(s) and the Service Domain Manager. Inspect now provides a new configuration wizard for the Sun Grid Engine Parallel Environment and for the Service Domain Manger Cloud Service (EC2) Adapter for ease of use when managing parallel processing workloads with Sun Grid Engine.
- Slot-wise preemption: Allows users to configure queue subordinate relationships that affect only single slots and not whole queue instances in the subordinated queues. Thus, the slotwise preemption provides a means to ensure that high priority jobs get the resources they need, while at the same time low priority jobs on the same host are not unnecessarily preempted, maximizing the host utilization. Sun reports that this feature works well in conjunction with Topological Scheduling (see below) and thus contributes to higher overall throughput.
- Array job throttling: Enables the user to set a self-imposed limit on the maximum number of concurrently running jobs. The array job may take slightly longer to complete, but enables other jobs from the same user or other users to run earlier and at the same time. This prevents large jobs from monopolizing a cluster.
- Topological scheduling: Enables Sun Grid Engine users to schedule jobs at the socket, core or thread level according to that job's unique needs, as determined by the user. The use of topological scheduling can deliver significant performance increases where workloads can benefit from optimizations such as local memory locality. Sun reports performance optimization for multi-core processors, specifically on Nehalem, can improve by up to 2x. This feature is available for Solaris 10 and OpenSolaris (x64, x86) and Linux (x64, x86) operating systems which support the CPU core affinity setting and reporting.
Additionally, the server and client side job submission verifier (JSV) has been enhanced to support JSVs written in the Java programming language.
For users without a Sun Grid Engine license, a 90-day evaluation use is offered. The courtesy binaries based on the Grid Engine open source project will be made available soon, writes Thijs Metsch with Sun. They will continue to allow unlimited use but will not include the Amazon EC2 and power saving adapters, the SGE Inspect module or the Hadoop integration.
The product, itself, is issued either as a subscription agreement or as a perpetual license, with entitlements according to the number of CPU sockets and/or for unlimited sockets according to the number of employees in the organization.
After 90 days from installation, the user must purchase a Sun software subscription or the perpetual right to use license and, if support is desired, a service contract to legally use the software.
Paid licenses of Sun Grid Engine 6.0 and higher qualify for a free upgrade to Sun Grid Engine 6.2, update 5. Users who do not have a license or a valid subscription must purchase a license in order to upgrade.
Subscription entitlements qualify for the latest upgrades as long as the subscriptions are renewed and up-to-date.
Sun Grid Engine 6.2 Update 5 - Sun product page
Sun Grid Engine 6.2 Update 5 Documentation Wiki
The following links provide general information on some of the Sun Grid Engine features but are not specific to Update 5:
How the Cloud Adapter Works
Amazon EC2 Overview
Service Domain Manager
Read More ...