Difference between revisions of "Netcdf"

Revision as of 11:56, 10 March 2009

NetCDF (network Common Data Form) is an interface for array-oriented data access and a library that provides an implementation of the interface. The netCDF library also defines a machine-independent format for representing scientific data. Together, the interface, library, and format support the creation, access, and sharing of scientific data.

Netcdf library

The netcdf library provides some useful binary tools suchs as ncdump. It is required to link against ConvertToHdf5 and ConvertToNetcdf tools.

Netcdf 4

Netcdf 4 allows to read hdf5 files as well and is fully backwards compatible.

Netcdf 3.6.x

How to compile netcdf 3.5.1 for intel fortran in windows

This how to will only explain how to compile the static libraries.

• Get the netcdf source distribution version 3.5.1.
• Unpack somwhere
• From the netcdf-3.5.1/src/libsrc

nmake /f msofts.mak

• From the netcdf-3.5.1/src/fortran

nmake /f msofts.mak

• From the netcdf-3.5.1/src/f90

ifort -c  -Gm typeSizes.f90
ifort -c  -Gm netcdf.f90
ar cru ../libsrc/netcdfs.lib netcdf.obj typeSizes.obj
ranlib ../libsrc/netcdfs.lib

• Copy the netcdfs.lib and the mod files to IntelLibs and that's it!

Netcdf tools

Nc bins

• ncgen
• ncdump

NcBrowse

Matlab

NCO NetCdf Operators

Perl

To compile the NetCDF module in perl in linux:

• 1 Make sure you know where the netcdf libraries are installed in your system:

> sudo find / | grep libnetcdf.a
> sudo find / | grep netcdf.h

• 2 Go to this site and extract the module source code;
• 3 Unpack the module

> tar -xvf netcdf-perl-x.tar.gz
> cd netcdf-perl-x/src
> ./configure
> make
> sudo make install

• 4 Locate where it was installed

> sudo find / | grep NetCDF.pm

• 5 Copy the ncmanipulator file in the same directory as NetCDF.pm
• 6 That should do it!

NOTE: Couldn't do it in windows :(

Check out a sample perl program to manipulate a netcdf file.

Problems while compiling netcdf for fortran in windows or in linux

The binaries distributed for windows are currently only for the c and c++ interface. This is a problem since we want the netcdf libraries with the fortran interface.

What the compilation thread should be

Here's the log of a linux compilation with make:

First we build the c library:

make[2]: Entering directory `/home/guillaume/Transferências/netcdf-3.6.1/src/libsrc
'
cc -c -g -O2 -I.. -I.   -DpgiFortran attr.c
cc -c -g -O2 -I.. -I.   -DpgiFortran dim.c
cc -c -g -O2 -I.. -I.   -DpgiFortran error.c
cc -c -g -O2 -I.. -I.   -DpgiFortran -DVERSION=`cat ../VERSION` libvers.c
cc -c -g -O2 -I.. -I.   -DpgiFortran nc.c
cc -c -g -O2 -I.. -I.   -DpgiFortran ncio.c
cc -c -g -O2 -I.. -I.   -DpgiFortran ncx.c
cc -c -g -O2 -I.. -I.   -DpgiFortran putget.c
cc -c -g -O2 -I.. -I.   -DpgiFortran string.c
cc -c -g -O2 -I.. -I.   -DpgiFortran v1hpg.c
cc -c -g -O2 -I.. -I.   -DpgiFortran v2i.c
cc -c -g -O2 -I.. -I.   -DpgiFortran var.c
ar cru libnetcdf.a attr.o dim.o error.o libvers.o nc.o ncio.o ncx.o putget.o string
.o v1hpg.o v2i.o var.o
ranlib libnetcdf.a
make[2]: Leaving directory `/home/guillaume/Transferências/netcdf-3.6.1/src/libsrc'

Then we build the C interface with fortran:

make[2]: Entering directory `/home/guillaume/Transferências/netcdf-3.6.1/src/fortra
n'
cc -c -g -O2 -I.. -I../libsrc   -DpgiFortran fort-attio.c
cc -c -g -O2 -I.. -I../libsrc   -DpgiFortran fort-control.c
cc -c -g -O2 -I.. -I../libsrc   -DpgiFortran fort-dim.c
cc -c -g -O2 -I.. -I../libsrc   -DpgiFortran fort-genatt.c
cc -c -g -O2 -I.. -I../libsrc   -DpgiFortran fort-geninq.c
cc -c -g -O2 -I.. -I../libsrc   -DpgiFortran fort-genvar.c
cc -c -g -O2 -I.. -I../libsrc   -DpgiFortran fort-lib.c
cc -c -g -O2 -I.. -I../libsrc   -DpgiFortran fort-misc.c
cc -c -g -O2 -I.. -I../libsrc   -DpgiFortran fort-v2compat.c
cc -c -g -O2 -I.. -I../libsrc   -DpgiFortran fort-vario.c
cc -c -g -O2 -I.. -I../libsrc   -DpgiFortran fort-var1io.c
cc -c -g -O2 -I.. -I../libsrc   -DpgiFortran fort-varaio.c
cc -c -g -O2 -I.. -I../libsrc   -DpgiFortran fort-varmio.c
cc -c -g -O2 -I.. -I../libsrc   -DpgiFortran fort-varsio.c
ar cru ../libsrc/libnetcdf.a fort-attio.o fort-control.o fort-dim.o fort-genatt.o f
ort-geninq.o fort-genvar.o fort-lib.o fort-misc.o fort-v2compat.o fort-vario.o fort
-var1io.o fort-varaio.o fort-varmio.o fort-varsio.o
ranlib ../libsrc/libnetcdf.a
make[2]: Leaving directory `/home/guillaume/Transferências/netcdf-3.6.1/src/fortran

Finally we build the fortran library:

make[2]: Entering directory `/home/guillaume/Transferências/netcdf-3.6.1/src/f90'
ifort -c  typeSizes.f90
ifort -c  netcdf.f90
ar cru ../libsrc/libnetcdf.a netcdf.o typeSizes.o
ranlib ../libsrc/libnetcdf.a
make[2]: Leaving directory `/home/guillaume/Transferências/netcdf-3.6.1/src/f90'

Follow these steps and you should be fine.

Problems arose from name mangling

However different compilers will produce different name decorations and then linkers will have a hell of a time trying to resolve the references from different object files and/or libraries. In netcdf, it is likely for this to be the case.

in windows

For example, here's an extract of the external references of ftest.obj after compiling with df (compaq Fortran):

df /c /nologo /Zi /Od ftest.for
dumpbin /SYMBOLS ftest.obj

094 00000000 UNDEF  notype ()    External     | _NCDREN@20
095 00000000 UNDEF  notype ()    External     | _NCVID@16
096 00000000 UNDEF  notype ()    External     | _NCVREN@20
097 00000000 UNDEF  notype ()    External     | _NCAREN@28
098 00000000 UNDEF  notype ()    External     | _NCVDEF@28
099 00000000 UNDEF  notype ()    External     | _NCVGT@24
09A 00000000 UNDEF  notype ()    External     | _NCVGTC@32
09B 00000000 UNDEF  notype ()    External     | _NCVGT1@20
09C 00000000 UNDEF  notype ()    External     | _NCVG1C@24
09D 00000000 UNDEF  notype ()    External     | _NCVPT@24
09E 00000000 UNDEF  notype ()    External     | _NCVPTC@32
09F 00000000 UNDEF  notype ()    External     | _NCVPT1@20
0A0 00000000 UNDEF  notype ()    External     | _NCVP1C@24

and here's an extract after compiling ftest.for with ifort (Intel Fortran):

ifort /c /nologo /Zi /Od ftest.for
dumpbin /SYMBOLS ftest.obj

1D8 00000000 UNDEF  notype ()    External     | _NCDREN
1D9 00000000 UNDEF  notype ()    External     | _NCVID
1DA 00000000 UNDEF  notype ()    External     | _NCVREN
1DB 00000000 UNDEF  notype ()    External     | _NCAREN
...
1E5 00000000 UNDEF  notype ()    External     | _NCVDEF
...
205 00000000 UNDEF  notype ()    External     | _NCVGT
206 00000000 UNDEF  notype ()    External     | _NCVGTC

Hence the same object will have problems linking to other objects or libraries if the name mangling convention is diferent! How do you solve that?

Well, let's take a look at the static version of the netcdf library, netcdfs.lib:

071 00000DE0 SECT5  notype ()    External     | _NCDREN@20
072 00000ED0 SECT5  notype ()    Static       | _c_ncdren
073 00000000 UNDEF  notype ()    External     | _ncdimrename
074 00000F00 SECT5  notype ()    External     | _NCVINQ@36
075 00000000 UNDEF  notype ()    External     | _c2f_dimids
076 000010B0 SECT5  notype ()    Static       | _c_ncvinq
077 00000000 UNDEF  notype ()    External     | _ncvarinq
078 000010F0 SECT5  notype ()    External     | _NCVPT1@20
079 00000000 UNDEF  notype ()    External     | _f2c_coords
07A 00001150 SECT5  notype ()    Static       | _c_ncvpt1

As you can see the name mangling convention is compatible with the df compiler default settings and not with the ifort compiler default settings. Thus it will link correctly with the df-obj whereas it will return unreferenced errors with the ifort-obj.

So, what's the solution? Well, you have to tell the compiler to use the correct name mangling convention:

ifort /Gm /c /nologo /Zi /Od ftest.for

And if you try link it against the netcds.lib, it'll work! Thanks for the /Gm option defined in ifort.

in linux

In linux machines, sometimes, such name mangling may occur. Let's check which name convention is used with the ifort compiler:

> ifort -c -fpp ModuleMERCATORFormat.F90
> nm ModuleMERCATORFormat.F90 | grep netcdf
                U netcdf_mp_nf90_close_
                U netcdf_mp_nf90_get_att_text_
                U netcdf_mp_nf90_get_var_1d_eightbytereal_
                U netcdf_mp_nf90_get_var_1d_fourbytereal_
                U netcdf_mp_nf90_get_var_2d_fourbytereal_
                U netcdf_mp_nf90_get_var_3d_fourbytereal_
                U netcdf_mp_nf90_get_var_4d_fourbytereal_
                U netcdf_mp_nf90_inq_dimid_
                U netcdf_mp_nf90_inquire_
                U netcdf_mp_nf90_inquire_dimension_
                U netcdf_mp_nf90_inquire_variable_
                U netcdf_mp_nf90_inq_varid_
                U netcdf_mp_nf90_open_

Now let's verify what the libnetcdf.a library uses the same name decoration:

> nm /usr/lib/libnetcdf.a | grep netcdf | grep '_$'
...
0000000000001796 T netcdf_mp_nf90_put_var_text_
000000000000563a T netcdf_mp_nf90_put_var_twobyteint_
0000000000000190 T netcdf_mp_nf90_redef_
0000000000000452 T netcdf_mp_nf90_rename_att_
00000000000003b0 T netcdf_mp_nf90_rename_dim_
000000000000178c T netcdf_mp_nf90_rename_var_
...

In this case, it turns out that both the netcdf library and the calling module use the same name decoration, thus there will be no error at link time arose from calling conventions for netcdf functions.

External references

• Netcdf at unidata
• NCO homepage
• Matlab interface for netcdf