Porting, monitoring and tuning UPC on NUMA architectures

In this work we report on our experience in porting NAS NPB benchmark using the recently developed GCC-SGI UPC compiler on the Origin 03800 NUMA machine. In fact, the SGI NUMA environment has provided new opportunities for UPC For example, by coupling Unix P-threads with standard UPC threads one is able to code solutions to problems using pipelining, divide-and-conquer, and speculative parallelization styles. This task-level parallelism was never before possible in UPC that relies mainly on distributed shared memory fine-grain data parallelism. This has led to having multi-threads per processor and provided further opportunities for optimization through load balancing. The SGI CC-NUMA environment also provided memory consistency optimizations to mask the latency of remote accesses, convert aggregate accesses into more efficient bulk operations, and cache data locally. UPC allows programmers to specify memory accesses with "relaxed" consistency semantics. These explicit consistency "hints" are exploited by the CC-NUMA environment very effectively to hide latency and reduce coherence overheads further by, for example, allowing two or more processors to modify their local copies of shared data concurrently and merging modifications at synchronization points. This characteristic alleviates the effect of false sharing. Yet another opportunity that was made possible by the spectrum of performance analysis and profiler tools within the SGI NUMA environment is the development of new monitoring and tuning strategy that aims at improving the efficiency of parallel UPC applications. We are able to project the physically monitored parameters back to the data structures and high-level program constructs within the UPC source code. This increases a programmer's ability to effectively understand, develop, and optimize UPC programs; enabling an exact analysis of a program's data and code layouts. Using this visualized information, programmers are able to detect communication, data/threads layouts, and I/O bottlenecks and further optimizes UPC programs with a better data and threads layouts potentially resulting in significant performance improvements.