[esp-r] Re: ESPr validation with gfortran/gcc4
Ferguson, Alex
Alex.Ferguson at NRCan-RNCan.gc.ca
Fri Feb 27 21:14:44 GMT 2009
Hi Scott,
On linux, we presently compile and use the vanilla versions of gcc shipped with gentoo, along with the gcc-config tool, which ensures gcc finds its corresponding glibc/libstdc++ libraries. Off hand, I do know which use flags we've presently configured for portage, but I'll check next week.
- Alex
-----Original Message-----
From: Scott Bucking [mailto:sbucking at gmail.com]
Sent: February 27, 2009 16:07
To: Ferguson, Alex
Cc: esp-r at lists.strath.ac.uk
Subject: Re: [esp-r] ESPr validation with gfortran/gcc4
Hi Alex,
Thanks for your reply.
One more question (with regards to gcc4.1.2/gfortran for Linux).
Did you use a vanilla tool chain (if so what distribution and version?), or did you compile gcc separately for each test (if so what enable/disable/with switches did you compile with?)
As I am working on optimization, I need to avoid any numerically induced noise.
Thanks again,
Scott
On Fri, Feb 27, 2009 at 12:37 PM, Ferguson, Alex <Alex.Ferguson at nrcan-rncan.gc.ca> wrote:
>
>
>>Has anyone recently performed validation on more recent toolchains
>>under a Linux plaform >(gcc4.3/gfortran)?
>>
>>I tend to keep my system updated, and am wondering if a separate
>>version of gcc3/g77 is still necessary to avoid truncation/numerical
>>discrepancies between compilers.
>
> Hi Scott,
>
> We've undertaken validation with gcc/gfortran 4.1 and 4.2, as well as
> the Intel compiler. Results are mixed; 4.1.2 seems the most stable
> (and is now recommended over gcc 3.4.X/ g77, while 4.2+ produces
> numerical exceptions. The next release (11.7) will likely default the
> Install script to use gFortran.
>
> A complete summary of the pre-release testing (including
> g77/gfortran/ifort/sun-f90 comparisons) completed for 11.6 follows.
>
> - Alex
>
> Alex Ferguson
> Sustainable Buildings and Communities (Housing Group) CanmetENERGY
> Natural Resources Canada
> P:+1 613 995 3294
> F:+1 613 996 9909
> Alex.Ferguson at NRCan-RNCan.gc.ca
>
>
>
> ======================
> SUMMARY:
> ======================
>
> - ESP-r compiles correctly on Linux, Cygwin, MinGW, Solaris and OSX.
> - Portability testing suggests that GCC 4.1.2 builds are actually
> more
> reliable than GCC3 builds.
> - GCC 4.2.3 builds are not as robust.
> - Some uncertainty remains in Intel builds, as well as builds on OSX.
>
> Recommendations:
> - Proceed with release 11.6.
> - The ESP-r 'reference' platform should be changed from
> Linux / GCC 3.4.6 to Linux / GCC 4.1.2
> - Cygwin / GCC 3.4.6 and SUN CC builds should continue to be
> designated as stable
> - GCC 4.2+ and Intel builds should be used with caution.
> - Compiler optimization should be used with caution.
> - OSX / GCC 3.4.6 and MinGW / GCC 3.4.6 builds should be used with
> caution.
>
> Detailed results follow.
>
>
>
> ======================
> Static Analysis:
> ======================
>
> The Forcheck static analyzer warns of no additional coding errors in
> the current version of pre-release-patches. The total number of
> errors, warnings and informational numbers as declined from the last release.
>
>
>
> ======================
> Regression testing:
> ======================
>
> Changes since the last revision have introduced significant numerical
> differences into the results. But for the first time, all the
> numerical differences were anticipated by developers, and no
> unintended effects were observed during testing.
>
> Two revisions introduced numerical differences in to the results:
>
> r2838: This commit patched a long-standing bug in the water tank
> storage model. Predicted water temperatures in the plant
> network may differ by as much as 3oC; predicted casual gains
> in the space may differ by as much as 3W.
>
> r3088: This commit corrected an error in the application of
> relaxation
> to the plant network solution scheme. Predicted heat transfer
> in the plant may differ by as much as 400W (1%), predicted
> temperatures in the plant may differ by as much as 18oC in
> extreme cases.
>
> ======================
> Portability testing:
> ======================
> The release candidate can be successfully installed on the following
> platform, compiler and graphics library combinations. We've also added
> support for the Intel compiler suite on linux. Jon Hand has been
> experimenting with support for 64-bit architecture, but this work has
> not yet been generalized for inclusion in the public release.
>
> We've successfully installed ESP-r using the following 32-bit
> platform, compiler and graphics library combinations:
>
>
>
> Support for ESP-r on various compiler, platform
> and graphics library combinations.
> =============================================================
> Compiler Library Linux Cygwin MinGW Sun OSX
> =============================================================
> GCC 3.4.X X11 O O
> SUN CC X11 O
> GCC 3.4.4 X11 O
> noX O
> GCC 3.4.5 noX *
> GCC 3.4.6 X11 O
> GTK O
> noX O
> INTEL noX O
> X11 O
> GTK O
> GCC 4.1.2 X11 O
> GCC 4.2.3 X11 O
> -------------------------------------------------------------
> Notes:
> O: Installs correctly.
> *: MinGW only installs correctly to absolute DOS paths
> (i.e. C:/ESRU)
> =============================================================
>
>
>
> We've also undertaken a comprehensive comparison of the numerical
> results produced by ESP-r on various platforms. Previously, these
> comparisons were limited to various versions of the GCC compiler suite
> on Linux and cygwin. Since the last release, Jon Hand has worked to
> add support for XML-enabled builds on OSX and Solaris, using both the
> GCC and Sun compiler suites. Along with new support for the Intel
> compiler, these new capabilities add important reference points for
> our ongoing evaluation of GCC4/gFortran
>
> For the last few years, we've designated GCC 3.4.6/Linux builds as the
> reference version of ESP-r. Most ESP-r development and automated
> testing is based on this platform, and bps's numerical predictions on
> this platform receive scrutiny than any other. GCC 3's fortran 77
> compiler
> (g77) has been obsoleted by gFortran, a fortran 90 compiler released
> with GCC 4. Results differ somewhat between g77 and gFortran builds,
> and we've been wondering if they're caused by bugs in g77 or gFortran,
> or perhaps poor ESP-r code that one of the compilers is more sensitive to.
> For the first time, we have additional reference points (Intel and Sun
> CC/f90) to perform these comparisons with.
>
> The following chart summarizes the significant observed differences
> between the and other platform and compiler combinations. The
> "Maximum difference" reflects the maximum observed difference observed
> in the test cases, excluding exceptional test cases that are discussed
> in section 'Problematic test cases', below..
>
>
>
> Comparison of results from GCC4.1.2-compiled versions
> (X11, Linux) with builds from other platform/compilers.
> =============================================================
> Arch. Platform Compiler Maximum difference (%)
> -------------------------------------------------------------
> IA-32 CYGWIN GCC 3.4.4 3.18
> LINUX GCC 3.4.6 3.18
> GCC 4.2.3 0. (^1)
> Intel -O0 0.75 (^2)
> Intel -O2 38.781
> PowerPC OSX GCC 3.4 16.1
> Sparc SOLARIS Sun CC/f90 1.80
> -------------------------------------------------------------
> NOTES:
> 1. Some test cases produced numerical exceptions and
> floating point errors. See "Problematic Test Cases".
> 2. One test case produced a larger error (~13%). See below.
> =============================================================
>
> Detailed scrutiny of the results revealed:
>
> - gFortran vs Intel iFort / SUN f90
> ---------------------------------
> With optimization disabled, the results from SUN and Intel compilers
> consistently agree more closely with GCC 4.1.2 / gFortran than they
> do with GCC 3.4.6 / g77. These results suggest GCC 4.1.2 can be used
> with confidence, and indeed, should be recommended in place of g77.
>
> - Stability of gFortran
> ---------------------
> GCC 4.2.3 builds encounter numerical exceptions in a handful of test
> cases that ran correctly in all other compilers (See "problematic
> test cases", below). I recommend GCC 4.2+ builds be regarded as
> 'beta' until this issue is resolved.
>
> - Intel iFort results: effect of optimization
> -------------------------------------------
> With optimization deactivated, the Intel compiler consistently
> produced
> results close to the gFortran and Sun f90 builds. But when the
> default
> optimization is activated (using the -O2) option, agreement between
> the Intel and GCC 4.1.2 compiler deteriorates considerably. Nearly
> all of the models exhibit small, non-trace differences, and in every
> case these differences were slightly, or significantly larger than
> those observed with optimization deactivated.
>
> Its possible this error is due to bugs in Intel's optimization
> algorithm, but poor code in ESP-r is more likely the culprit. For
> instance, when -O0 is specified, the compiler ensures that arguement
> mismatches in procedure calls are properly converted, but does not
> perform these conversions by default.
>
> This problem might occur in GCC builds as well. By default, GCC uses
> no optimization. I suspect similar issues might appear if -O2 is
> specified for g77/gFortran builds.
>
> For these reasons, application of optimization options with ESP-r
> is not presently recommended.
>
> - Intel iFort results: test case bld_hc_ISO15099 /HC
> --------------------------------------------------
> Newly added for this release, this test case continues to cause
> headaches. Intel-compiled versions of bps exhibit significant
> differences from g77/gFortran builds:
>
> - MAX error (W) 199.37 W ( 13.293 %)
> - Predicted value - g77 1499.8 W
> Intel 1300.4 W
> [ observed in:
> building:zone_05:thermal_loads:net_load:month_01 (max) ]
>
> The corresponding test case that does not activate the ISO15099
> correlation does not exhibit the same error, suggesting the Intel
> compiler is exposing a sensitive compoment of the ISO15099
> algorithm,
>
> or vice-versa.
>
> The SUN CC/f90 results we have do not include this test case,
> precluding assessment of whether the g77/gFortran or Intel
> predictions
> are more accurate. For this reason, the Intel compiler suite is only
> recommended for beta testing for the time being.
>
>
> - OSX / GCC builds
> ----------------
> A handful of test cases exhibited suprising sensitivity when run
> with GCC 3.4 compiled builds on OSX. The cause of these differences
> is not known; OSX builds should be used with caution for the time
> being.
>
> ===========================
> Problematic test cases
> ===========================
>
> - esru_benchmark_model / bld_basic_af2_summer
> / bld_basic_af2_winter
> -------------------------------------------
> We've previously observed the that this coarse-timestep air flow
> network test case produces significant differences in g77 and
> gFortran builds. The long, half-hour timesteps cause small
> differences in the numerical computations to be exaggerated in
> the aggregated output --- increasing time resolution vastly improves
> agreement between the compilers.
>
> The Linux / Intel, Solaris / SUN CC-f90, and OSX / g77 platforms all
> exhibit the same sensitivity. They produce differing results at
> short-timesteps, but agreement with Linux / g77 improves at higher
> time-resolutions.
>
> Since we've observed that each compiler combination produces
> dissimilar
> results and that the high-resolution version is more useful,
> this test case has little value. For this reason, I recommend we
> delete it.
>
> - plt_boundary_conditions / connected_flow
> connected_temperature
> unconnected_controls
> unconnected_flow
> unconnected_temperature
> -------------------------------------------------
> While they run correctly in every other compiler (including GCC
> 4.1.2),
> these test cases appear to produce numerical exceptions in GCC 4.2.3.
>
>
More information about the esp-r
mailing list