A duel in the Suns of sorts involving a Sun Fire V40z and a Sun Fire X4450 connected back-to-back was performed to discover their ability to exchange UDP or TCP messages with a round-trip latency under 100 ms per message. In addition, both servers were occupied with some business logic, to bring the overall CPU utilization to 60-80% on each machine. Both were running Solaris 10 Update 5 and Java RTS 2.1 (build 1.5.0_16_Java-RTS-2.1_fcs-b11_RTSJ-1.0.2). Frederic Pariente reports the results in his blog Real Time opens system and network surveillance to Java, a proof-of-concept to evaluate Java RTS 2.1 on Solaris 10.

The Sun Fire V40z (Machine 1) featured 32GB memory, 8x 2.2GHz cores and 2x bge interfaces while the Sun Fire X4450 (Machine 2) ran 8x 3GHz cores, 8GB memory and 2x e1000g interfaces.

Pariente writes that, in addition to an average workload of 400 CPU-bound application threads -- part of them being real-time threads -- on each system, Machine 1 was configured to run 4 real-time sender threads sending messages in different sizes and Machine 2 was configured to run 2 real-time sender threads. Finally, each system had one real-time thread receiving and sending an acknowledgement message back, and another real-time thread reading the acknowledgements (and calculating round-trip times). All the above communication-handling threads were given highest real-time priority (among all application threads).

Machine 1

Thread 1 – sends 120KB every 50ms Thread 2 – sends 1KB every 50ms Thread 3 – sends 100B every 50ms Thread 4 – sends 200B every 50ms

Machine 2

Thread 1 – sends 300 every 50ms Thread 2 – sends 350 every 50ms

The first tests revealed periodic delays in communication, Pariente reports, adding that a message round trip between the servers could take more than 500 ms. Monitoring the systems revealed a correlation between those delays and real-time application threads being block by GC threads, whose priority is raised when there is a severe stress on memory. Such situations can be signs of memory leaks.

Given those results, DTrace was applied, which identified the problem as memory allocations that grew unintentionally to very large String allocations. They were easily tracked down by 2 DTrace scripts using RTS probes, the author notes.

What the researchers found when they ran Script 1 which prints the number of allocated bytes per thread, every second and Script 2, which prints the top allocating stack trace, each second, for a specified thread was that there was some sort of recursive concatenation of ever-growing strings, which caused garbage collection to block from time to time the application threads, leading to significant delays.

Further sources of delay were found to involve CPU handling interrupts. Since the application's real-time threads have a higher priority, they could and would preempt the TCP squeues processes, thus getting in the way of the TCP/IP processing.

A long-standing Solaris feature allows processor sets that enable the binding of specific processes onto a given set of processors, or the reverse of that: the isolation of these processors from the rest of the system. Pariente and his colleagues defined a processor set of 7 cores on each machine, bounded the application process to this set (psrset -c 1-7; psrset -b 1 <JRTS pid>) and moved all interrupts away from the application's processor set (psrset -f 1).

With the tuning concluded, max latency for a message round trip, including simple processing, had been reduced to under 45 milliseconds for 50KB messages and 80 ms for 120KB messages. Processors were running mostly in the range of 60-80% CPU utilization. The remaining (incompressible) delays were attributed to TCP/IP networking, none of the application critical threads being interrupted.

The system integrator was very satisfied with the proof-of-concept, Pariente writes, adding that multiple licenses for commercial development were ordered. Java RTS brought the best of both worlds into play: the coding productivity of the Java language -- that translates into lower cost of development and maintenance for the software house -- and the performance and predictability typically achieved by a native language like C, he concludes.

More Information

Sun Java Real-Time System (Java RTS) - Evaluation Downloads

Real-Time - Real-Time Specification for Java (RTSJ) Forum [...read more...]

		fullsource
Print Email Current Profile Archive Blog Generate newsletter