https://www.reproducibility.org/wiki2020/api.php?action=feedcontributions&user=Sergey&feedformat=atomMadagascar - User contributions [en]2024-03-29T01:35:01ZUser contributionsMediaWiki 1.34.0https://www.reproducibility.org/wiki2020/index.php?title=Guide_to_programming_with_madagascar&diff=3933Guide to programming with madagascar2021-05-06T02:29:52Z<p>Sergey: update + python</p>
<hr />
<div><center><font size="-1">''This page was created from the LaTeX source in [https://github.com/ahay/src/blob/master/book/rsf/rsf/demo.tex book/rsf/rsf/demo.tex] using [[latex2wiki]]''</font></center><br />
<br />
This guide demonstrates a simple time-domain<br />
finite-differences modeling code in RSF.<br />
<br />
==Introduction==<br />
<br />
This section presents time-domain <br />
finite-difference modeling <ref>"Hello world" of seismic imaging.</ref><br />
written with the RSF library.<br />
The program is demonstrated with the C, C++ and Fortran 90 interfaces.<br />
The acoustic wave-equation<br />
<center><math><br />
\Delta U - \frac{1}{v^2} \frac{\partial^2 U}{\partial t^2} = f(t)<br />
</math></center><br />
can be written as<br />
<center><math><br />
\left[ \Delta U - f(t) \right] v^2 =<br />
\frac{\partial^2 U}{\partial t^2} \;.<br />
</math></center><br />
<math>\Delta</math> is the Laplacian symbol,<br />
<math>f(t)</math> is the source wavelet,<br />
<math>v</math> is the velocity, and<br />
<math>U</math> is a scalar wavefield.<br />
A discrete time-step involves the following computations:<br />
<center><math><br />
U_{i+1} = \left[ \Delta U -f(t) \right] v^2 \Delta t^2 + 2 U_{i} - U_{i-1} \;,<br />
</math></center><br />
where <math>U_{i-1}</math>, <math>U_{i}</math> and <math>U_{i+1}</math><br />
represent the propagating wavefield at various time steps.<br />
<br />
==C program==<br />
<br />
[[Image:wavec.png|frame|center|Wave propagation snapshot.]]<br />
<br />
==C program==<br />
<br />
<syntaxhighlight lang="c"><br />
/* time-domain acoustic FD modeling */<br />
#include <rsf.h><br />
int main(int argc, char* argv[])<br />
{<br />
/* Laplacian coefficients */<br />
float c0=-30./12.,c1=+16./12.,c2=- 1./12.;<br />
<br />
bool verb; /* verbose flag */<br />
sf_file Fw=NULL,Fv=NULL,Fr=NULL,Fo=NULL; /* I/O files */<br />
sf_axis at,az,ax; /* cube axes */<br />
int it,iz,ix; /* index variables */<br />
int nt,nz,nx;<br />
float dt,dz,dx,idx,idz,dt2;<br />
<br />
float *ww,**vv,**rr; /* I/O arrays*/<br />
float **um,**uo,**up,**ud;/* tmp arrays */<br />
<br />
sf_init(argc,argv);<br />
if(! sf_getbool("verb",&verb)) verb=0; /* verbose flag */<br />
<br />
/* setup I/O files */<br />
Fw = sf_input ("in" );<br />
Fo = sf_output("out");<br />
Fv = sf_input ("vel");<br />
Fr = sf_input ("ref");<br />
<br />
/* Read/Write axes */<br />
at = sf_iaxa(Fw,1); nt = sf_n(at); dt = sf_d(at);<br />
az = sf_iaxa(Fv,1); nz = sf_n(az); dz = sf_d(az);<br />
ax = sf_iaxa(Fv,2); nx = sf_n(ax); dx = sf_d(ax);<br />
<br />
sf_oaxa(Fo,az,1); <br />
sf_oaxa(Fo,ax,2); <br />
sf_oaxa(Fo,at,3);<br />
<br />
dt2 = dt*dt;<br />
idz = 1/(dz*dz);<br />
idx = 1/(dx*dx);<br />
<br />
/* read wavelet, velocity & reflectivity */<br />
ww=sf_floatalloc(nt); sf_floatread(ww ,nt ,Fw);<br />
vv=sf_floatalloc2(nz,nx); sf_floatread(vv[0],nz*nx,Fv);<br />
rr=sf_floatalloc2(nz,nx); sf_floatread(rr[0],nz*nx,Fr);<br />
<br />
/* allocate temporary arrays */<br />
um=sf_floatalloc2(nz,nx);<br />
uo=sf_floatalloc2(nz,nx);<br />
up=sf_floatalloc2(nz,nx);<br />
ud=sf_floatalloc2(nz,nx);<br />
<br />
for (ix=0; ix<nx; ix++) {<br />
for (iz=0; iz<nz; iz++) {<br />
um[ix][iz]=0;<br />
uo[ix][iz]=0;<br />
up[ix][iz]=0;<br />
ud[ix][iz]=0;<br />
}<br />
}<br />
<br />
/* MAIN LOOP */<br />
if(verb) fprintf(stderr,"<br />
");<br />
for (it=0; it<nt; it++) {<br />
if(verb) fprintf(stderr,"\b\b\b\b\b %d",it);<br />
<br />
/* 4th order laplacian */<br />
for (ix=2; ix<nx-2; ix++) {<br />
for (iz=2; iz<nz-2; iz++) {<br />
ud[ix][iz] = <br />
c0* uo[ix ][iz ] * (idx+idz) + <br />
c1*(uo[ix-1][iz ] + uo[ix+1][iz ])*idx +<br />
c2*(uo[ix-2][iz ] + uo[ix+2][iz ])*idx +<br />
c1*(uo[ix ][iz-1] + uo[ix ][iz+1])*idz +<br />
c2*(uo[ix ][iz-2] + uo[ix ][iz+2])*idz; <br />
}<br />
}<br />
<br />
/* inject wavelet */<br />
for (ix=0; ix<nx; ix++) {<br />
for (iz=0; iz<nz; iz++) {<br />
ud[ix][iz] -= ww[it] * rr[ix][iz];<br />
}<br />
}<br />
<br />
/* scale by velocity */<br />
for (ix=0; ix<nx; ix++) {<br />
for (iz=0; iz<nz; iz++) {<br />
ud[ix][iz] *= vv[ix][iz]*vv[ix][iz];<br />
}<br />
}<br />
<br />
/* time step */<br />
for (ix=0; ix<nx; ix++) {<br />
for (iz=0; iz<nz; iz++) {<br />
up[ix][iz] = <br />
2*uo[ix][iz] <br />
- um[ix][iz] <br />
+ ud[ix][iz] * dt2; <br />
<br />
um[ix][iz] = uo[ix][iz];<br />
uo[ix][iz] = up[ix][iz];<br />
}<br />
}<br />
<br />
/* write wavefield to output */<br />
sf_floatwrite(uo[0],nz*nx,Fo);<br />
}<br />
if(verb) fprintf(stderr,"\n");<br />
<br />
exit (0);<br />
}<br />
</syntaxhighlight><br />
<br />
<br />
#Declare input, output and auxiliary file tags: <tt>Fw</tt> for input wavelet, <tt>Fv</tt> for velocity, <tt>Fr</tt> for reflectivity, and <tt>Fo</tt> for output wavefield. <syntaxhighlight lang="c"><br />
sf_file Fw=NULL,Fv=NULL,Fr=NULL,Fo=NULL; /* I/O files */<br />
</syntaxhighlight> <br />
#Declare RSF cube axes: <tt>at</tt> time axis, <tt>ax</tt> space axis, <tt>az</tt> depth axis. \tiny <syntaxhighlight lang="c"><br />
sf_axis at,az,ax; /* cube axes */<br />
</syntaxhighlight> <br />
#Declare multi-dimensional arrays for input, output and computations. <syntaxhighlight lang="c"><br />
<br />
float *ww,**vv,**rr; /* I/O arrays*/<br />
</syntaxhighlight> <br />
#Open files for input/output. <syntaxhighlight lang="c"><br />
Fw = sf_input ("in" );<br />
Fo = sf_output("out");<br />
Fv = sf_input ("vel");<br />
Fr = sf_input ("ref");<br />
</syntaxhighlight> <br />
#Read axes from input files; write axes to output file. <syntaxhighlight lang="c"><br />
at = sf_iaxa(Fw,1); nt = sf_n(at); dt = sf_d(at);<br />
az = sf_iaxa(Fv,1); nz = sf_n(az); dz = sf_d(az);<br />
ax = sf_iaxa(Fv,2); nx = sf_n(ax); dx = sf_d(ax);<br />
<br />
sf_oaxa(Fo,az,1); <br />
sf_oaxa(Fo,ax,2); <br />
sf_oaxa(Fo,at,3);<br />
</syntaxhighlight> <br />
#Allocate arrays and read wavelet, velocity and reflectivity. <syntaxhighlight lang="c"><br />
ww=sf_floatalloc(nt); sf_floatread(ww ,nt ,Fw);<br />
vv=sf_floatalloc2(nz,nx); sf_floatread(vv[0],nz*nx,Fv);<br />
rr=sf_floatalloc2(nz,nx); sf_floatread(rr[0],nz*nx,Fr);<br />
</syntaxhighlight> <br />
#Allocate temporary arrays. <syntaxhighlight lang="c"><br />
um=sf_floatalloc2(nz,nx);<br />
uo=sf_floatalloc2(nz,nx);<br />
up=sf_floatalloc2(nz,nx);<br />
ud=sf_floatalloc2(nz,nx);<br />
</syntaxhighlight> <br />
#Loop over time. <syntaxhighlight lang="c"><br />
for (it=0; it<nt; it++) {<br />
</syntaxhighlight> <br />
#Compute Laplacian: <math>\Delta U</math>. <syntaxhighlight lang="c"><br />
for (ix=2; ix<nx-2; ix++) {<br />
for (iz=2; iz<nz-2; iz++) {<br />
ud[ix][iz] = <br />
c0* uo[ix ][iz ] * (idx+idz) + <br />
c1*(uo[ix-1][iz ] + uo[ix+1][iz ])*idx +<br />
c2*(uo[ix-2][iz ] + uo[ix+2][iz ])*idx +<br />
c1*(uo[ix ][iz-1] + uo[ix ][iz+1])*idz +<br />
c2*(uo[ix ][iz-2] + uo[ix ][iz+2])*idz; <br />
}<br />
}<br />
</syntaxhighlight> <br />
#Inject source wavelet: <math>\left[ \Delta U - f(t) \right]</math> <syntaxhighlight lang="c"><br />
for (ix=0; ix<nx; ix++) {<br />
for (iz=0; iz<nz; iz++) {<br />
ud[ix][iz] -= ww[it] * rr[ix][iz];<br />
}<br />
}<br />
</syntaxhighlight> <br />
#Scale by velocity: <math>\left[ \Delta U - f(t) \right] v^2</math> <syntaxhighlight lang="c"><br />
for (ix=0; ix<nx; ix++) {<br />
for (iz=0; iz<nz; iz++) {<br />
ud[ix][iz] *= vv[ix][iz]*vv[ix][iz];<br />
}<br />
}<br />
</syntaxhighlight> <br />
#Time step: <math>U_{i+1} = \left[ \Delta U -f(t) \right] v^2 \Delta t^2 + 2 U_{i} - U_{i-1}</math> <syntaxhighlight lang="c"><br />
for (ix=0; ix<nx; ix++) {<br />
for (iz=0; iz<nz; iz++) {<br />
up[ix][iz] = <br />
2*uo[ix][iz] <br />
- um[ix][iz] <br />
+ ud[ix][iz] * dt2; <br />
<br />
um[ix][iz] = uo[ix][iz];<br />
uo[ix][iz] = up[ix][iz];<br />
}<br />
}<br />
</syntaxhighlight> <br />
<br />
<br />
\newpage<br />
==C++ program==<br />
<br />
<syntaxhighlight lang="cpp"><br />
// time-domain acoustic FD modeling<br />
#include <valarray><br />
#include <iostream><br />
#include <rsf.hh><br />
#include <cub.hh><br />
<br />
#include "vai.hh"<br />
<br />
using namespace std;<br />
<br />
int main(int argc, char* argv[])<br />
{<br />
// Laplacian coefficients<br />
float c0=-30./12.,c1=+16./12.,c2=- 1./12.;<br />
<br />
sf_init(argc,argv);// init RSF<br />
bool verb; // vebose flag<br />
if(! sf_getbool("verb",&verb)) verb=0;<br />
<br />
// setup I/O files<br />
CUB Fw("in", "i"); Fw.headin(); //Fw.report();<br />
CUB Fv("vel","i"); Fv.headin(); //Fv.report();<br />
CUB Fr("ref","i"); Fr.headin(); //Fr.report();<br />
CUB Fo("out","o"); Fo.setup(3); <br />
<br />
// Read/Write axes<br />
sf_axis at = Fw.getax(0); int nt = sf_n(at); float dt = sf_d(at);<br />
sf_axis az = Fv.getax(0); int nz = sf_n(az); float dz = sf_d(az);<br />
sf_axis ax = Fv.getax(1); int nx = sf_n(ax); float dx = sf_d(ax);<br />
<br />
Fo.putax(0,az); <br />
Fo.putax(1,ax); <br />
Fo.putax(2,at);<br />
Fo.headou();<br />
<br />
float dt2 = dt*dt;<br />
float idz = 1/(dz*dz);<br />
float idx = 1/(dx*dx);<br />
<br />
// read wavelet, velocity and reflectivity<br />
valarray<float> ww( nt ); ww=0; Fw >> ww;<br />
valarray<float> vv( nz*nx ); vv=0; Fv >> vv;<br />
valarray<float> rr( nz*nx ); rr=0; Fr >> rr;<br />
<br />
// allocate temporary arrays<br />
valarray<float> um(nz*nx); um=0;<br />
valarray<float> uo(nz*nx); uo=0;<br />
valarray<float> up(nz*nx); up=0;<br />
valarray<float> ud(nz*nx); ud=0;<br />
<br />
// init ValArray Index counter<br />
VAI k(nz,nx);<br />
<br />
// MAIN LOOP<br />
if(verb) cerr << endl;<br />
for (int it=0; it<nt; it++) {<br />
if(verb) cerr << "\b\b\b\b\b" << it;<br />
<br />
// 4th order laplacian<br />
for (int ix=2; ix<nx-2; ix++) {<br />
for (int iz=2; iz<nz-2; iz++) {<br />
ud[k(iz,ix)] = <br />
c0* uo[ k(iz ,ix )] * (idx+idz) +<br />
c1*(uo[ k(iz ,ix-1)]+uo[ k(iz ,ix+1)]) * idx + <br />
c1*(uo[ k(iz-1,ix )]+uo[ k(iz+1,ix )]) * idz + <br />
c2*(uo[ k(iz ,ix-2)]+uo[ k(iz ,ix+2)]) * idx + <br />
c2*(uo[ k(iz-2,ix )]+uo[ k(iz+2,ix )]) * idz;<br />
}<br />
}<br />
<br />
// inject wavelet<br />
ud -= ww[it] * rr;<br />
<br />
// scale by velocity<br />
ud *= vv*vv;<br />
<br />
// time step<br />
up=(float)2 * uo - um + ud * dt2;<br />
um = uo;<br />
uo = up;<br />
<br />
// write wavefield to output output<br />
Fo << uo;<br />
}<br />
if(verb) cerr << endl;<br />
<br />
exit(0);<br />
}<br />
<br />
</syntaxhighlight><br />
<br />
#Declare input, output and auxiliary file cubes (of type <tt>CUB</tt>). <syntaxhighlight lang="cpp"><br />
CUB Fw("in", "i"); Fw.headin(); //Fw.report();<br />
CUB Fv("vel","i"); Fv.headin(); //Fv.report();<br />
CUB Fr("ref","i"); Fr.headin(); //Fr.report();<br />
CUB Fo("out","o"); Fo.setup(3); <br />
</syntaxhighlight> <br />
#Declare, read and write RSF cube axes: <tt>at</tt> time axis, <tt>ax</tt> space axis, <tt>az</tt> depth axis. <syntaxhighlight lang="cpp"><br />
sf_axis at = Fw.getax(0); int nt = sf_n(at); float dt = sf_d(at);<br />
sf_axis az = Fv.getax(0); int nz = sf_n(az); float dz = sf_d(az);<br />
sf_axis ax = Fv.getax(1); int nx = sf_n(ax); float dx = sf_d(ax);<br />
</syntaxhighlight> <br />
#Declare multi-dimensional <tt>valarrays</tt> for input, output and read data. <syntaxhighlight lang="cpp"><br />
valarray<float> ww( nt ); ww=0; Fw >> ww;<br />
valarray<float> vv( nz*nx ); vv=0; Fv >> vv;<br />
valarray<float> rr( nz*nx ); rr=0; Fr >> rr;<br />
</syntaxhighlight> <br />
#Declare multi-dimensional <tt>valarrays</tt> for temporary storage. <syntaxhighlight lang="cpp"><br />
valarray<float> um(nz*nx); um=0;<br />
valarray<float> uo(nz*nx); uo=0;<br />
valarray<float> up(nz*nx); up=0;<br />
valarray<float> ud(nz*nx); ud=0;<br />
</syntaxhighlight> <br />
#Initialize multidimensional <tt>valarray</tt> index counter (of type <tt>VAI</tt>). <syntaxhighlight lang="cpp"><br />
VAI k(nz,nx);<br />
</syntaxhighlight> <br />
#Loop over time. <syntaxhighlight lang="cpp"><br />
for (int it=0; it<nt; it++) {<br />
</syntaxhighlight> <br />
#Compute Laplacian: <math>\Delta U</math>. <syntaxhighlight lang="cpp"><br />
for (int ix=2; ix<nx-2; ix++) {<br />
for (int iz=2; iz<nz-2; iz++) {<br />
ud[k(iz,ix)] = <br />
c0* uo[ k(iz ,ix )] * (idx+idz) +<br />
c1*(uo[ k(iz ,ix-1)]+uo[ k(iz ,ix+1)]) * idx + <br />
c1*(uo[ k(iz-1,ix )]+uo[ k(iz+1,ix )]) * idz + <br />
c2*(uo[ k(iz ,ix-2)]+uo[ k(iz ,ix+2)]) * idx + <br />
c2*(uo[ k(iz-2,ix )]+uo[ k(iz+2,ix )]) * idz;<br />
}<br />
}<br />
</syntaxhighlight> <br />
#Inject source wavelet: <math>\left[ \Delta U - f(t) \right]</math> <syntaxhighlight lang="cpp"><br />
ud -= ww[it] * rr;<br />
</syntaxhighlight> <br />
#Scale by velocity: <math>\left[ \Delta U - f(t) \right] v^2</math> <syntaxhighlight lang="cpp"><br />
ud *= vv*vv;<br />
</syntaxhighlight> <br />
#Time step: <math>U_{i+1} = \left[ \Delta U -f(t) \right] v^2 \Delta t^2 + 2 U_{i} - U_{i-1}</math> <syntaxhighlight lang="cpp"><br />
up=(float)2 * uo - um + ud * dt2;<br />
um = uo;<br />
uo = up;<br />
</syntaxhighlight> <br />
<br />
==Fortran 90 program==<br />
<br />
<syntaxhighlight lang="fortran"><br />
! time-domain acoustic FD modeling<br />
program AFDMf90<br />
use rsf<br />
<br />
implicit none<br />
<br />
! Laplacian coefficients<br />
real :: c0=-30./12.,c1=+16./12.,c2=- 1./12.<br />
<br />
logical :: verb ! verbose flag<br />
type(file) :: Fw,Fv,Fr,Fo ! I/O files<br />
type(axa) :: at,az,ax ! cube axes<br />
integer :: it,iz,ix ! index variables<br />
integer :: nt,nz,nx<br />
real :: dt,dz,dx<br />
real :: idx,idz,dt2<br />
<br />
real, allocatable :: vv(:,:),rr(:,:),ww(:) ! I/O arrays<br />
real, allocatable :: um(:,:),uo(:,:),up(:,:),ud(:,:) ! tmp arrays<br />
<br />
call sf_init() ! init RSF<br />
call from_par("verb",verb,.false.)<br />
<br />
! setup I/O files<br />
Fw=rsf_input ("in")<br />
Fv=rsf_input ("vel")<br />
Fr=rsf_input ("ref")<br />
Fo=rsf_output("out")<br />
<br />
! Read/Write axes<br />
call iaxa(Fw,at,1); nt = at%n; dt = at%d<br />
call iaxa(Fv,az,1); nz = az%n; dz = az%d<br />
call iaxa(Fv,ax,2); nx = ax%n; dx = ax%d<br />
<br />
call oaxa(Fo,az,1)<br />
call oaxa(Fo,ax,2)<br />
call oaxa(Fo,at,3)<br />
<br />
dt2 = dt*dt<br />
idz = 1/(dz*dz)<br />
idx = 1/(dx*dx) <br />
<br />
! read wavelet, velocity & reflectivity<br />
allocate(ww(nt)); call rsf_read(Fw,ww)<br />
allocate(vv(nz,nx)); call rsf_read(Fv,vv)<br />
allocate(rr(nz,nx)); call rsf_read(Fr,rr)<br />
<br />
! allocate temporary arrays<br />
allocate(um(nz,nx)); um=0.<br />
allocate(uo(nz,nx)); uo=0.<br />
allocate(up(nz,nx)); up=0.<br />
allocate(ud(nz,nx)); ud=0.<br />
<br />
! MAIN LOOP<br />
do it=1,nt<br />
if(verb) write (0,*) it<br />
<br />
ud(3:nz-2,3:nx-2) = &<br />
c0* uo(3:nz-2,3:nx-2) * (idx + idz) + &<br />
c1*(uo(3:nz-2,2:nx-3) + uo(3:nz-2,4:nx-1))*idx + &<br />
c2*(uo(3:nz-2,1:nx-4) + uo(3:nz-2,5:nx ))*idx + &<br />
c1*(uo(2:nz-3,3:nx-2) + uo(4:nz-1,3:nx-2))*idz + &<br />
c2*(uo(1:nz-4,3:nx-2) + uo(5:nz ,3:nx-2))*idz<br />
<br />
! inject wavelet<br />
ud = ud - ww(it) * rr<br />
<br />
! scale by velocity<br />
ud= ud *vv*vv<br />
<br />
! time step<br />
up = 2*uo - um + ud * dt2<br />
um = uo<br />
uo = up<br />
<br />
! write wavefield to output<br />
call rsf_write(Fo,uo)<br />
end do<br />
<br />
call exit(0)<br />
end program AFDMf90<br />
</syntaxhighlight><br />
<br />
#Declare input, output and auxiliary file tags. <syntaxhighlight lang="fortran"><br />
type(file) :: Fw,Fv,Fr,Fo ! I/O files<br />
</syntaxhighlight> <br />
#Declare RSF cube axes: <tt>at</tt> time axis, <tt>ax</tt> space axis, <tt>az</tt> depth axis. <syntaxhighlight lang="fortran"><br />
type(axa) :: at,az,ax ! cube axes<br />
</syntaxhighlight> <br />
#Declare multi-dimensional arrays for input, output and computations. <syntaxhighlight lang="fortran"><br />
real, allocatable :: vv(:,:),rr(:,:),ww(:) ! I/O arrays<br />
real, allocatable :: um(:,:),uo(:,:),up(:,:),ud(:,:) ! tmp arrays<br />
</syntaxhighlight> <br />
#Open files for input/output. <syntaxhighlight lang="fortran"><br />
Fw=rsf_input ("in")<br />
Fv=rsf_input ("vel")<br />
Fr=rsf_input ("ref")<br />
Fo=rsf_output("out")<br />
</syntaxhighlight> <br />
#Read axes from input files; write axes to output file. <syntaxhighlight lang="fortran"><br />
call iaxa(Fw,at,1); nt = at%n; dt = at%d<br />
call iaxa(Fv,az,1); nz = az%n; dz = az%d<br />
call iaxa(Fv,ax,2); nx = ax%n; dx = ax%d<br />
<br />
call oaxa(Fo,az,1)<br />
call oaxa(Fo,ax,2)<br />
call oaxa(Fo,at,3)<br />
</syntaxhighlight> <br />
#Allocate arrays and read wavelet, velocity and reflectivity. <syntaxhighlight lang="fortran"><br />
allocate(ww(nt)); call rsf_read(Fw,ww)<br />
allocate(vv(nz,nx)); call rsf_read(Fv,vv)<br />
allocate(rr(nz,nx)); call rsf_read(Fr,rr)<br />
</syntaxhighlight> <br />
#Allocate temporary arrays. <syntaxhighlight lang="fortran"><br />
allocate(um(nz,nx)); um=0.<br />
allocate(uo(nz,nx)); uo=0.<br />
allocate(up(nz,nx)); up=0.<br />
allocate(ud(nz,nx)); ud=0.<br />
</syntaxhighlight> <br />
#Loop over time. <syntaxhighlight lang="fortran"><br />
do it=1,nt<br />
</syntaxhighlight> <br />
#Compute Laplacian: <math>\Delta U</math>. <syntaxhighlight lang="fortran"><br />
ud(3:nz-2,3:nx-2) = &<br />
c0* uo(3:nz-2,3:nx-2) * (idx + idz) + &<br />
c1*(uo(3:nz-2,2:nx-3) + uo(3:nz-2,4:nx-1))*idx + &<br />
c2*(uo(3:nz-2,1:nx-4) + uo(3:nz-2,5:nx ))*idx + &<br />
c1*(uo(2:nz-3,3:nx-2) + uo(4:nz-1,3:nx-2))*idz + &<br />
c2*(uo(1:nz-4,3:nx-2) + uo(5:nz ,3:nx-2))*idz<br />
</syntaxhighlight> <br />
#Inject source wavelet: <math>\left[ \Delta U - f(t) \right]</math> <syntaxhighlight lang="fortran"><br />
ud = ud - ww(it) * rr<br />
</syntaxhighlight> <br />
#Scale by velocity: <math>\left[ \Delta U - f(t) \right] v^2</math> <syntaxhighlight lang="fortran"><br />
ud= ud *vv*vv<br />
</syntaxhighlight> <br />
#Time step: <math>U_{i+1} = \left[ \Delta U -f(t) \right] v^2 \Delta t^2 + 2 U_{i} - U_{i-1}</math> <syntaxhighlight lang="fortran"><br />
up = 2*uo - um + ud * dt2<br />
um = uo<br />
uo = up<br />
</syntaxhighlight> <br />
<br />
==Python program==<br />
<br />
<syntaxhighlight lang="python"><br />
#!/usr/bin/env python<br />
<br />
import sys<br />
import numpy<br />
import m8r<br />
<br />
c0=-30./12.<br />
c1=+16./12.<br />
c2=- 1./12.<br />
<br />
par = m8r.Par()<br />
verb = par.bool("verb",False) # verbosity<br />
<br />
# setup I/O files<br />
Fw=m8r.Input()<br />
Fv=m8r.Input ("vel")<br />
Fr=m8r.Input ("ref")<br />
Fo=m8r.Output()<br />
<br />
# Read/Write axes<br />
at = Fw.axis(1); nt = at['n']; dt = at['d']<br />
az = Fv.axis(1); nz = az['n']; dz = az['d']<br />
ax = Fv.axis(2); nx = ax['n']; dx = ax['d']<br />
<br />
Fo.putaxis(az,1)<br />
Fo.putaxis(ax,2)<br />
Fo.putaxis(at,3)<br />
<br />
dt2 = dt*dt<br />
idz = 1/(dz*dz)<br />
idx = 1/(dx*dx) <br />
<br />
# read wavelet, velocity & reflectivity<br />
ww = numpy.zeros(nt,'f'); Fw.read(ww)<br />
vv = numpy.zeros([nz,nx],'f'); Fv.read(vv)<br />
rr = numpy.zeros([nz,nx],'f'); Fr.read(rr)<br />
<br />
# allocate temporary arrays<br />
um = numpy.zeros([nz,nx],'f')<br />
uo = numpy.zeros([nz,nx],'f')<br />
up = numpy.zeros([nz,nx],'f')<br />
ud = numpy.zeros([nz,nx],'f')<br />
<br />
# MAIN LOOP<br />
for it in range(nt):<br />
if verb:<br />
sys.stderr.write("\b\b\b\b\b %d" % it)<br />
<br />
ud[2:-2,2:-2] = \<br />
c0* uo[2:-2,2:-2] * (idx + idz) + \<br />
c1*(uo[2:-2,1:-3] + uo[2:-2,3:-1])*idx + \<br />
c2*(uo[2:-2, :-4] + uo[2:-2,4: ])*idx + \<br />
c1*(uo[1:-3,2:-2] + uo[3:-1,2:-2])*idz + \<br />
c2*(uo[ :-4,2:-2] + uo[4: ,2:-2])*idz<br />
<br />
# inject wavelet<br />
ud = ud - ww[it] * rr<br />
<br />
# scale by velocity<br />
ud= ud *vv*vv<br />
<br />
# time step<br />
up = 2*uo - um + ud * dt2<br />
um = uo<br />
uo = up<br />
<br />
if verb:<br />
sys.stderr.write("\n")<br />
Fo.write(uo)<br />
<br />
sys.exit(0)<br />
</syntaxhighlight><br />
<br />
[[Image:wavepython.png|frame|center|Wave propagation snapshot.]]<br />
<br />
#Open files for input/output. <syntaxhighlight lang="python"><br />
Fw=m8r.Input()<br />
Fv=m8r.Input ("vel")<br />
Fr=m8r.Input ("ref")<br />
Fo=m8r.Output()<br />
</syntaxhighlight> <br />
#Read axes from input files; write axes to output file. <syntaxhighlight lang="python"><br />
at = Fw.axis(1); nt = at['n']; dt = at['d']<br />
az = Fv.axis(1); nz = az['n']; dz = az['d']<br />
ax = Fv.axis(2); nx = ax['n']; dx = ax['d']<br />
<br />
Fo.putaxis(az,1)<br />
Fo.putaxis(ax,2)<br />
Fo.putaxis(at,3)<br />
</syntaxhighlight> <br />
#Allocate arrays and read wavelet, velocity and reflectivity. <syntaxhighlight lang="python"><br />
ww = numpy.zeros(nt,'f'); Fw.read(ww)<br />
vv = numpy.zeros([nz,nx],'f'); Fv.read(vv)<br />
rr = numpy.zeros([nz,nx],'f'); Fr.read(rr)<br />
</syntaxhighlight> <br />
#Allocate temporary arrays. <syntaxhighlight lang="python"><br />
um = numpy.zeros([nz,nx],'f')<br />
uo = numpy.zeros([nz,nx],'f')<br />
up = numpy.zeros([nz,nx],'f')<br />
ud = numpy.zeros([nz,nx],'f')<br />
</syntaxhighlight> <br />
#Loop over time. <syntaxhighlight lang="python"><br />
for it in range(nt):<br />
</syntaxhighlight> <br />
#Compute Laplacian: <math>\Delta U</math>. <syntaxhighlight lang="python"><br />
ud[2:-2,2:-2] = \<br />
c0* uo[2:-2,2:-2] * (idx + idz) + \<br />
c1*(uo[2:-2,1:-3] + uo[2:-2,3:-1])*idx + \<br />
c2*(uo[2:-2, :-4] + uo[2:-2,4: ])*idx + \<br />
c1*(uo[1:-3,2:-2] + uo[3:-1,2:-2])*idz + \<br />
c2*(uo[ :-4,2:-2] + uo[4: ,2:-2])*idz<br />
</syntaxhighlight> <br />
#Inject source wavelet: <math>\left[ \Delta U - f(t) \right]</math> <syntaxhighlight lang="python"><br />
ud = ud - ww[it] * rr<br />
</syntaxhighlight> <br />
#Scale by velocity: <math>\left[ \Delta U - f(t) \right] v^2</math> <syntaxhighlight lang="python"><br />
ud= ud *vv*vv<br />
</syntaxhighlight> <br />
#Time step: <math>U_{i+1} = \left[ \Delta U -f(t) \right] v^2 \Delta t^2 + 2 U_{i} - U_{i-1}</math> <syntaxhighlight lang="python"><br />
up = 2*uo - um + ud * dt2<br />
um = uo<br />
uo = up<br />
</syntaxhighlight> <br />
<br />
==References==<br />
<references/></div>Sergeyhttps://www.reproducibility.org/wiki2020/index.php?title=Main_Page&diff=3932Main Page2021-05-06T00:54:39Z<p>Sergey: /* Latest News */</p>
<hr />
<div><big>'''Madagascar'''</big> is an open-source software package for multidimensional data analysis and [[Reproducibility|reproducible]] computational experiments. Its mission is to provide<br />
* a convenient and powerful environment<br />
* a convenient technology transfer tool<br />
for researchers working with digital image and data processing in geophysics and related fields. Technology developed using the Madagascar project management system is transferred in the form of recorded processing histories, which become "computational recipes" to be verified, exchanged, and modified by users of the system.<br />
<br />
== Features ==<br />
<br />
[[Image:Crystal_Clear_app_katomic.png|left|30px|Modern]]<br />
Madagascar is a <u>modern</u> package. Started in 2003 and publicly released in 2006, it was developed almost entirely from scratch. Being a relatively new package, it follows modern software engineering practices such as module encapsulation and test-driven development. A rapid development of a project of this scope (more than 1,000 main programs and more than 5,000 tests) would not be possible without standing on the shoulders of giants and learning from the 30 years of previous experience in open packages such as SEPlib and Seismic Unix. We have borrowed and reimplemented functionality and ideas from these other packages. <br />
<br />
[[Image:Crystal_Clear_app_vcalendar.png|left|30px|Test-driven]]<br />
Madagascar is a <u>test-driven</u> package. Test-driven development is not only an agile software programming practice but also a way of bringing scientific foundation to geophysical research that involves numerical experiments. Bringing reproducibility and peer review, the backbone of any real science, to the field of computational geophysics is the main motivation for Madagascar development. The package consists of two levels: low-level main programs (typically developed in the C programming language and working as data filters) and high-level processing flows (described using the Python programming language) that combine main programs and completely document data processing histories for testing and reproducibility. Experience shows that high-level programming is easily mastered even by beginning students who have no previous programming experience. <br />
<br />
[[Image:Crystal_Clear_app_gadu.png|left|30px|Open]]<br />
Madagascar is an <u>open-source</u> package. It is distributed under the standard GPL open-source license, which places no restriction on the usage and modification of the code. Moreover, access to modifying the source repository is not controlled by one organization but shared equally among different developers. This enables an open collaboration among different groups spread all over the world, in the true spirit of the open-source movement. <br />
<br />
[[Image:Crystal_Clear_filesystem_blockdevice.png|left|30px|Simple,flexible]]<br />
Madagascar uses a <u>simple, flexible, and universal</u> data format that can handle very large datasets but is not tied specifically to seismic data or data of any other particular kind. This "regularly sampled" format is borrowed from the traditional SEPlib. A universal data format allows us to share general-purpose data processing tools with scientists from other disciplines such as petroleum engineers working on large-scale reservoir simulations.<br />
<br />
== Latest News ==<br />
<br />
{{#widget:Feed<br />
|feedurl=https://reproducibility.org/blog/feed/<br />
|num=7<br />
|desc=100<br />
|date=n<br />
|targ=n<br />
}}</div>Sergeyhttps://www.reproducibility.org/wiki2020/index.php?title=File:Online2021.png&diff=3931File:Online2021.png2021-05-05T23:27:01Z<p>Sergey: Sergey uploaded a new version of File:Online2021.png</p>
<hr />
<div></div>Sergeyhttps://www.reproducibility.org/wiki2020/index.php?title=Advanced_Installation&diff=3930Advanced Installation2021-05-05T00:17:39Z<p>Sergey: /* Mac OS X */ updated</p>
<hr />
<div>[[Image:Fotolia_419157_XS.jpg|right|]]<br />
Before reading this document, please familiarize yourself with the [[Installation|short Installation guide]].<br />
=What the installation process does=<br />
The term "installation" in the title is used for brevity, and it actually covers all three steps: configuration, build and install.<br />
# Configure: determine what tools are available on the system and how they should be used to built the software. Creates a layer of abstraction so that the build is platform-independent. Should ideally either solve or flag all problems, so that the build either works, or does not proceed at all.<br />
# Build: compiles the software and documentation using RSFSRC/build as a "workplace"<br />
# Install: moves the compiled executables and the documentation to the final locations in $RSFROOT, sometimes changing filenames. Kept separate from build so that it can be done by root, and to avoid build failures leaving junk files all over the system.<br />
A successful installation will have created in <tt>$RSFROOT</tt> the following directories:<br />
* <tt>bin/</tt>: executable programs<br />
* <tt>doc/</tt>: auto-generated HTML documentation<br />
* <tt>include/</tt>: header files with info on library procedures; fonts<br />
* <tt>lib/</tt>: libraries and Python modules<br />
<br />
=Old (1.0 and 1.1) version installation=<br />
<br />
Use <tt>$RSFROOT/etc/madagascar/</tt> as the location of <tt>env</tt> scripts instead of <tt>$RSFROOT/share/madagascar/etc/</tt>.<br />
<br />
=Old (0.9.9) version installation=<br />
<br />
====Environment variables====<br />
Example configuration for <tt>bash</tt>:<br />
<syntaxhighlight lang="bash"><br />
export RSFROOT=/usr/local/rsf # directory where Madagascar will be installed. <br />
if [ -n "$PYTHONPATH" ]; then<br />
export PYTHONPATH=${PYTHONPATH}:$RSFROOT/lib<br />
else<br />
export PYTHONPATH=$RSFROOT/lib<br />
fi<br />
export PATH=$RSFROOT/bin:$PATH<br />
export DATAPATH=/var/tmp/<br />
export MANPATH=$RSFROOT/share/man:$(manpath)<br />
export LD_LIBRARY_PATH=$RSFROOT/lib:$LD_LIBRARY_PATH<br />
</syntaxhighlight><br />
Notice the slash at the end of the <tt>DATAPATH</tt> variable. <br />
<br />
Example configuration for <tt>csh</tt> and <tt>tcsh</tt>:<br />
<pre><br />
setenv RSFROOT /usr/local/rsf<br />
if ($?PYTHONPATH) then<br />
setenv PYTHONPATH ${PYTHONPATH}:$RSFROOT/lib<br />
else<br />
setenv PYTHONPATH $RSFROOT/lib<br />
endif<br />
set path = ($RSFROOT/bin $path)<br />
setenv DATAPATH /var/tmp/<br />
setenv MANPATH $RSFROOT/share/man:`manpath`<br />
setenv LD_LIBRARY_PATH $RSFROOT/lib:$LD_LIBRARY_PATH<br />
</pre><br />
Notice the backticks surrounding the call to <tt>manpath</tt>.<br />
<br />
Be aware that on some systems /var/tmp gets automatically cleaned at some intervals, so if you want to keep your data binaries for a long time, set <tt>DATAPATH</tt> to another location where you have write access and that allows large files.<br />
<br />
====Software construction====<br />
<br />
#Configuration. Change to the top source directory and run <pre>./configure</pre> You can examine the <tt>config.py</tt> file that this command generates. Additional options are available. You can obtain a full list of customizable variables by running <tt>scons -h</tt>. For example, to install C++ and Fortran-90 API bindings in addition to the basic package, run <pre>./configure API=c++,fortran-90</pre> <br />
#Building and installing the package. Run <tt>scons install</tt> or the following two commands in succession: <pre>make; make install</pre> or <pre>scons; scons install</pre> If you need "root" privileges for installing under <tt>&#36;RSFROOT</tt>, you may need to run <pre>su; scons install </pre> or <pre>sudo scons install</pre><br />
#Cleaning. To clean all intermediate files generated by SCons, run <pre>make clean</pre> or <pre>scons -c</pre><br />
<br />
=Prerequisites=<br />
Basic prerequisites are described in the [[Installation|short Installation guide]]. Here are some additional details. <br />
==Compilers==<br />
Madagascar has been built successfully with the following compilers, and possibly with others:<br />
* gcc<br />
* Intel (icc/ifort)<br />
* open64<br />
* clang<br />
* cc (Solaris)<br />
<br />
==Python and SCons==<br />
As described below under [[Advanced Installation#Platform-specific installation advice | Platform-specific installation advice]], Madagascar supports the oldest non-deprecated Python version currently supported by the latest stable version of [http://scons.org/ SCons]. If your version of Python is older and you experience problems you should probably [http://www.python.org/ upgrade].<br />
<br />
Madagascar includes the latest stable version of SCons and the configure scripts will try to install it for you in RSFROOT if you don't have it already. However, if you have an older version of SCons the configure scripts will not try to install the newer version. Your older version might work fine, but Madagascar attempts to support only the latest stable version of SCons, so if you have problems you should upgrade.<br />
<br />
To install the SCons bundled with Madagascar go to <tt>RSFSRC/scons</tt>, unpack the tar file, and type<br />
<br />
<syntaxhighlight lang="bash"><br />
python setup.py install<br />
</syntaxhighlight><br />
<br />
This will install SCons in the standard location. You might need root privileges. If you don't have root privileges, or you don't want to interfere with the system SCons you can install it somewhere else with a --prefix option. A logical choice is to put it in RSFROOT like this:<br />
<br />
<syntaxhighlight lang="bash"><br />
python setup.py install --prefix=$RSFROOT<br />
</syntaxhighlight><br />
<br />
==Location==<br />
As long as you set the environment variables and directory permissions correctly, it does not matter in what part of your filesystem you place the install. If you have the luxury of installing anywhere, it is good practice to follow the [http://en.wikipedia.org/wiki/Filesystem_Hierarchy_Standard Filesystem Hierarchy Standard] and either:<br />
# Install everything (including <tt>figs</tt> if you do testing) under <tt>/usr/local/rsf</tt>, with the source tree in <tt>/usr/local/rsf/src</tt>, OR <br />
# Put the source tree in <tt>/usr/local/src/rsf</tt>, and specify <tt>RSFROOT=/usr/local</tt>, so that header files and binaries go in <tt>/usr/local/bin</tt> and <tt>/usr/local/include</tt>. To follow the standard, before installing set <tt>RSFDOC=/usr/local/share/rsf/doc</tt> and create the appropriate directories. The auto-generated HTML documentation will get put there. Also, if installed, the figs directory for testing should be <tt>/usr/local/share/rsf/figs/</tt>.<br />
# Package Madagascar (i.e. build a RPM, etc) and install it in the default locations. For RPMs, those are as like the ones from the previous option, just directly in the <tt>/usr/</tt> hierarchy, instead of in the <tt>/usr/local/</tt> one.<br />
<br />
==Disk space==<br />
At present (Feb 2007, r2530), the source directory containing the build tree from the development version was approx. 200Mb, the full installation (<tt>bin/</tt>, <tt>doc/</tt>, <tt>include/</tt> and <tt>lib</tt>) is 31Mb, and <tt>figs/</tt> (the optional directory if you want to do testing) is about 10 Gb. The stable version is significantly smaller.<br />
<br />
The only Madagascar-related directory where disk space can be an issue is <tt>$DATAPATH</tt>. Real 3-D seismic datasets can be measured in Terabytes. Buggy programs/processing flows can fill up <tt>$DATAPATH</tt>. A real problem are "disk memory leaks" -- removing header files with anything else than <tt>sfrm</tt> will leave the binaries intact. Crashed jobs which start to write to binary but never get to write the header also produce "leaks". Experience has shown that over time <tt>$DATAPATH</tt> inexorably fills up. You may need to <br />
# keep irreplaceable data and expensive results in a separate place;<br />
# remove the oldest files in <tt>$DATAPATH</tt> whenever the amount of free space declines under a preset threshold.<br />
<br />
==Dependencies==<br />
Some platforms feature complete lists of dependencies. See [[Advanced Installation#Platform-specific installation advice | Platform-specific installation advice]] for details.<br />
===C++ API===<br />
A C++ compiler. SCons is smart and will try to find it for you. If it does not work specify the path to your compiler in the <tt>CXX</tt> environment variable (can be passed as an option to the configuration script, like the <tt>API</tt> one).<br />
===F77 API===<br />
A Fortran 77 compiler. If SCons does not find one, then you can either specify its path through the <tt>F77</tt> variable, or if the executable is in your path, add its name to the list of F77 compilers in <tt>RSFSRC/configure.py</tt> .<br />
===F90 API===<br />
Same as for Fortran 77 &ndash; just substitute <tt>F90</tt>. If using the <tt>gfortran</tt> compiler, make sure to get [http://gcc.gnu.org/wiki/GFortranBinaries the latest version]. If you have more than one compiler installed on your system, specify the desired one at configuration time:<br />
<bash><br />
./configure API=f90 F90=/path/to/preferred/compiler<br />
</bash><br />
<br />
===Java API===<br />
There are two styles of API, old (1.0 release and previous) and new. The two APIs are not compatible with each other.<br />
<br />
The "old" interface requires the Mines Java Toolkit for fast Java file IO. The Mines JTK, is an open-source Java package that can be downloaded from [http://inside.mines.edu/~dhale/jtk/ Mines JTK]. Currently, the "old" interface will also build alongside the new one if the MINESJTK environment variable exists, and can be used as was previously.<br />
<br />
To install the new API, you need the (Oracle) JDK. Set your JAVA_HOME environment variable to the location of the jdk (on Ubuntu 10.04 this is: /usr/lib/jvm/java-6-openjdk), then reconfigure (./configure API=java ...) and reinstall. <br />
<br />
Ignore the SWIG warnings (there are lots). <br />
<br />
Make sure to set your LD_LIBRARY_PATH to $RSFROOT/lib .<br />
<br />
If you want to include additional Java packages, you can set them using your shell's CLASSPATH variable. This environment variable is now automatically passed onto all Java classes in SCons.<br />
<br />
The installation can be tested using the example demonstrating the new API in api/java/test .<br />
<br />
===Matlab API===<br />
Besides Matlab itself, you need Mex, which compiles C code into regular Matlab functions. Use the <tt>MATLAB</tt> and <tt>MEX</tt> environment variables to specify their paths if they are installed, but not found.<br />
===Octave API===<br />
The Octave function compiler (<tt>mkoctfile</tt>) is sometimes bundled in a separate package, so it may be missing from the Octave installation.<br />
<br />
===Python API===<br />
This API requires [http://www.swig.org/ SWIG], [http://numpy.scipy.org/ numpy] and the Python development kit. Numpy requires Python 2.4 or newer (i.e. RHEL 5 or newer). However, these dependencies are unnecessary for the common case when Python is just used as [http://en.wikipedia.org/wiki/Glue_language glue] to create chains of programs, and it only needs to read the RSF header, and not the binary. To allow Python [http://en.wikipedia.org/wiki/Meta-programs metaprograms] in madagascar to function, and programming in this style to be done, a fallback development kit implementing only the header-related functionality will be installed in the lack of these dependencies.<br />
<br />
===Python modules in user space===<br />
Python is an evolving language. Many large systems have old versions for stability reasons, and administrators of such large systems tend to not install all software users may wish, and to not allow access to rpm either. To install a module in your user space, download the tarball, unzip it, cd into the directory and run: <br />
<br />
<pre>python setup.py install --prefix=/path/to/your/place</pre><br />
<br />
The installer will create a subdirectory named <tt>lib</tt>, or <tt>lib64</tt> under the directory above. These <tt>lib*</tt> dirs will have a directory named <tt>python</tt>, or <tt>python2.3</tt> for example, and those will have a subdirectory named <tt>site-packages</tt>. Add all paths to these <tt>site-packages</tt> subdirectories in your <tt>PYTHONPATH</tt> environment variable. Some (<tt>numpy</tt>) may create a <tt>bin</tt> directory that needs to be added to <tt>PATH</tt>.<br />
<br />
=Environment variables=<br />
Besides the variables defined in env.sh or env.csh (see the [[Installation|short Installation guide]]), Madagascar programs may read the variables below. They usually have reasonable defaults and were introduced just to provide more power to the advanced user.<br />
<br />
For future documentation writers: the environment variables read by Madagascar that have not been documented below can be found by running the script <tt>$RSFSRC/admin/find_env_var.py</tt>. If the script does not exist or does not work, a summary of all environment variable calls can be obtained by going to $RSFSRC, temporarily moving the directory <tt>build/</tt> outside RSFSRC, and typing<br />
<syntaxhighlight lang="bash"><br />
grep environ.get *.py */*.py */*/*.py */*/*/*.py<br />
grep getenv */*.c */*/*.c */*/*/*.c<br />
</syntaxhighlight><br />
<br />
==Used by the Madagascar core==<br />
<br />
{|class="wikitable" align="center" cellspacing="0" border="1"<br />
|-<br />
! colspan="3" style="background:#ffdead;"|Variables introduced by Madagascar's non-graphic programs<br />
|-<br />
| '''Name''' || '''Default''' || Meaning<br />
|-<br />
| RSF_DATASERVER || <nowiki>ftp://egl.beg.utexas.edu/</nowiki> || Data server for benchmark datasets<br />
|-<br />
| RSFDOC || $RSFROOT/doc || Directory for the HTML self-doc<br />
|-<br />
| RSFFIGS || $RSFROOT/figs || Directory with figures for testing examples in $RSFSRC/book<br />
|-<br />
|-<br />
| RSFALTFIGS || $RSFFIGS || Alternate directory with figures for testing examples not in $RSFSRC/book<br />
|-<br />
| RSFMEMSIZE || 100 || Maximum RAM (Mb) to be used by some programs <br />
|-<br />
| RSFSRC || undefined || Root of the Madagascar source tree<br />
|-<br />
| TMPDATAPATH || $DATAPATH || Datapath for temporary files on local disk.<br />
|-<br />
| LATEX2HTML || undefined || LateX2HTML customization directory<br />
|}<br />
<br />
<br />
{|class="wikitable" align="center" cellspacing="0" border="1"<br />
|-<br />
! colspan="3" style="background:#ffdead;"|Variables introduced by Madagascar graphics programs <br />
|-<br />
| '''Name''' || '''Default''' || Meaning<br />
|-<br />
| DEFAULT_PAPER_SIZE || "letter" || For pspen. Other options: legal, a3, a4, a5.<br />
|-<br />
| FATMULT || ? || Fatness multiplication factor. <br />
|-<br />
| GIFBORDER || 0.25 || For vplot2gif (spacing)<br />
|-<br />
| GIFDELAY || 100 || For vplot2gif (for animations)<br />
|-<br />
| IMAGE_TYPE || 'png' || Icon type for LateX2HTML <br />
|-<br />
| PATTERNMULT || None || Pattern multiplication factor <br />
|-<br />
| PLOTSTYLE || None || Used in vplot<br />
|-<br />
| PPI || 75 || For vplot2gif (screen resolution)<br />
|-<br />
| PPMSCALE || 1 || For vplot2gif<br />
|-<br />
| PSBORDER || 0.05 || For vplot2eps (border around the plot)<br />
|-<br />
| PSPRINTER || postscript or colorps || For pspen<br />
|-<br />
| PSTEXPENOPTS || color=n fat=1 fatmult=1.5 invras=y || Other vplot2eps options <br />
|-<br />
| VPLOTFONTDIR || $RSFROOT/include || Dir with backup fonts in case the runtime-loaded vplot fonts are not found<br />
|-<br />
| VPLOTSPOOLDIR || /tmp || Where to put vplot tmp files<br />
|-<br />
| WSTYPE || "default" || Workstation type.<br />
|}<br />
<br />
<br />
{|class="wikitable" align="center" cellspacing="0" border="1"<br />
|-<br />
! colspan="2" style="background:#ffdead;"| Variables set by OS/other apps, read-only to Madagascar<br />
|-<br />
| '''Name''' || '''Primarily used/set by'''<br />
|-<br />
| CWPROOT || Seismic Unix<br />
|-<br />
| DISPLAY || Operating System (OS)<br />
|-<br />
| HOME || OS<br />
|-<br />
| LD_LIBRARY_PATH || linker<br />
|-<br />
| MATLABPATH || Matlab<br />
|-<br />
| XAUTHORITY || X-Windows<br />
|}<br />
<br />
==Used by the Madagascar build process and parallelization utilities==<br />
Type <tt>scons -h</tt> in RSFSRC to get a list of environment variables that affect the build process, with explanations, defaults and actual values. Below are more detailed explanations for some of them:<br />
* <tt>RSF_THREADS</tt>: used by <tt>pscons</tt> to determine on how many threads to run on the local node, overriding the number of threads detected by Madagascar<br />
* <tt>RSF_CLUSTER</tt>: used by <tt>pscons</tt> to determine on which cluster nodes to run, and on how many CPUs<br />
<br />
==Used by the Matlab API==<br />
To use the Matlab API, you need to add <tt>$RSFROOT/lib</tt> to <tt>MATLABPATH</tt><br />
==Used by the Octave API==<br />
To use the Octave API, you need to add <tt>$RSFROOT/lib</tt> to Octave's path. Determine Octave's version with<br />
<syntaxhighlight lang="bash"><br />
octave -v | head -1<br />
</syntaxhighlight><br />
If your version is lower than 2.9.6, type at a Unix command line:<br />
<syntaxhighlight lang="bash"><br />
echo 'LOADPATH = "::$RSFROOT/lib/octave"' >> ~/.octaverc<br />
</syntaxhighlight><br />
For later versions, use:<br />
<syntaxhighlight lang="bash"><br />
echo 'addpath([getenv("RSFROOT") "/lib/octave"])' >> ~/.octaverc<br />
</syntaxhighlight><br />
==Used by the Java API==<br />
New-style (post-1.0) API: Needs <tt>JAVA_HOME</tt><br />
<br />
Old-style API (1.0 and before): The path to the downloaded Mines JTK must be specified in the MINESJTK environment variable in order to install the Java API. For example:<br />
<syntaxhighlight lang="bash"><br />
export MINESJTK=/home/user/edu_mines_jtk.jar<br />
</syntaxhighlight><br />
<br />
==RSFROOT for NFS-shared user home directories==<br />
Heterogeneous networks with user home directories shared through [http://en.wikipedia.org/wiki/Network_File_System_(protocol) NFS] are quite common in many institutions. In addition, even when the architecture is the same (i.e. 64-bit) and the operating system is the same (i.e. [http://en.wikipedia.org/wiki/RHEL RHEL]), the difference between operating system versions may be very significant because clusters may run legacy versions, while desktop workstations may run the latest-and-greatest (even beta), and entirely different Madagascar versions may be needed to support both. <br />
<br />
One possible solution of detecting the distribution version and architecture and setting RSFROOT appropriately is shown below. In the example network, all RHEL4 machines have the same architecture, but there are RHEL 3 machines with several architectures:<br />
<syntaxhighlight lang="bash"><br />
REDHAT_RELEASE=`awk -F'release' '{ print $2 }' /etc/redhat-release | awk -F' ' '{ print $1 }'`<br />
<br />
RSFROOT=/usr/local/rsf/rhel$REDHAT_RELEASE<br />
<br />
if [ $REDHAT_RELEASE == '4' ] ; then<br />
export RSFROOT<br />
elif [ $REDHAT_RELEASE == '3' ] ; then<br />
export RSFROOT=$RSFROOT/$ARCH<br />
fi<br />
</syntaxhighlight><br />
Of course, the Madagascar administrator will have to download appropriate versions of Madagascar to each $RSFROOT, and compile them on the appropriate system.<br />
<br />
If you have many kinds of systems to maintain, with multiple versions of Madagascar, and users have more than one shell, you may find it easy to outsource the complex logic to the easy-to-debug Python, i.e.:<br />
<br />
<syntaxhighlight lang="bash"><br />
export RSFROOT=`$M8R_SETUP/get_rsfroot.py`<br />
export PYTHONPATH=`$M8R_SETUP/edit_pythonpath.py`<br />
export PATH=`$M8R_SETUP/edit_path.py`<br />
</syntaxhighlight><br />
<br />
and similarly for (t)csh. The Python scripts determine the operating system and its version, determine the machine name, and simply print to stdout the desired string.<br />
<br />
==Eclipse + Pydev==<br />
If you use [http://eclipse.org/ Eclipse] with [http://pydev.org/ Pydev], [http://pydev.org/manual_101_interpreter.html#id2 configure the interpreter] by adding <tt>$RSFROOT/lib</tt> to the <tt>PYTHONPATH</tt> for your chosen interpreter.<br />
<br />
=Platform-specific installation advice=<br />
==Supported platforms==<br />
Madagascar attempts to support any [http://en.wikipedia.org/wiki/POSIX POSIX-compliant] operating system demanded by users. For systems that bundle Python (i.e. Linux distributions, BSDs), backwards compatibility will attempt to cover those systems that were bundled with the oldest non-deprecated Python version currently supported by the latest stable version of [http://scons.org/ SCons]. For example, in early 2009 the stable SCons release (1.2) supported Python 2.2 or newer. [http://distrowatch.com/table.php?distribution=redhat Python 2.2 was bundled by RHEL3], so RHEL 3 and newer are supported. <br />
<br />
Attempts for backward compatibility with a given operating system are also stopped if the operating system itself becomes unsupported. For example, Python 2.2 was bundled by Fedora 1 and newer, but in January 2010 only Fedora 11 and 12 are actively maintained. Thus, in January 2010 Madagascar was not attempting to support Fedora 1, even though it included Python 2.2.<br />
<br />
Please keep in mind that the above statements constitute only general guidelines for what will be attempted, and do not constitute in any way a warranty of support. An application of the above guidelines to some Linux distributions follows:<br />
<br />
'''Support info'''<br />
{| class="wikitable" align="center" cellspacing="0" border="1"<br />
|-<br />
! style="background:#ffdead;" | Distribution<br />
! style="background:#ffdead;" | Life Cycle<br />
! style="background:#ffdead;" | Supported versions<br />
|-<br />
| RHEL/CentOS/Scientific Linux<br />
| [https://www.redhat.com/security/updates/errata/ 7 years]<br />
| <br />
* 6 until 2017-11-30<br />
* 5 until 2014-03-31<br />
|-<br />
| Fedora<br />
| [http://fedoraproject.org/wiki/Fedora_Release_Life_Cycle Release X maintained until one month after the release of X+2]<br />
| <br />
* 15 until 2012-06-24<br />
* 14 until 2011-12-02<br />
|-<br />
| Ubuntu<br />
| [https://wiki.ubuntu.com/Releases Releases every 6 mo, maintained for 1.5 yrs; LTS versions every 2 yrs, maintained for 5 yrs]<br />
| <br />
* 15.10 until July 2016<br />
* 14.04 LTS until end of April 2019<br />
* 12.04.4 LTS until end of April 2017<br />
|-<br />
| Debian<br />
| [http://wiki.debian.org/DebianLenny Usually: stable releases every 1.5-3 yrs, release X maintained 1 yr after release X+1]<br />
| <br />
* 6 until its TBD end of life (approx. 2014)<br />
* 5 until 2012-04<br />
|- <br />
| openSUSE<br />
| [http://en.opensuse.org/Lifetime openSUSE releases Lifetime of 1.5-2.5 years]<br />
| <br />
* 11.4 until 2012-09-10<br />
* 11.3 until 2012-01-15<br />
|}<br />
<br />
==Ubuntu==<br />
In '''Ubuntu 18.04''', you can install the essential Madagascar depenency by running<br />
<pre><br />
sudo apt-get install libxaw7-dev freeglut3-dev libnetpbm10-dev libgd-dev libplplot-dev \<br />
libavcodec-dev libcairo2-dev libjpeg-dev swig python-dev python-numpy g++ gfortran \<br />
libopenmpi-dev libfftw3-dev libsuitesparse-dev python-epydoc scons git emacs25<br />
<br />
</pre><br />
In '''Ubuntu 15.10 ''Wily Werewolf''''' (as well as some previous versions), you can install the essential Madagascar depenency by running<br />
<pre><br />
sudo apt-get install libxaw7-dev <br />
</pre><br />
and (optionally) all of Madagascar's dependencies by running<br />
<pre><br />
sudo apt-get install libxaw7-dev freeglut3-dev libnetpbm10-dev libgd-dev libplplot-dev \<br />
libavcodec-dev libcairo2-dev libjpeg-dev swig python-dev python-numpy g++ gfortran \<br />
libopenmpi-dev libfftw3-dev libsuitesparse-dev python-epydoc<br />
</pre><br />
<br />
In Ubuntu 13.04 ''Raring Ringtail'', you can install all of Madagascar's dependencies by running<br />
<pre><br />
sudo apt-get install freeglut3-dev g++ gfortran libgd2-xpm-dev libx11-dev libxaw7-dev \<br />
libnetpbm10-dev swig python-dev python-numpy libblas-dev libcairo2-dev liblapack-dev \<br />
libavcodec-dev python-epydoc scons<br />
</pre><br />
<br />
In Ubuntu 12.10 ''Quantal Quetzal'', you can install all of Madagascar's dependencies by running<br />
<pre><br />
sudo apt-get install freeglut3-dev g++ gfortran libgd2-xpm-dev libx11-dev libxaw7-dev \<br />
libnetpbm10-dev swig python-dev python-numpy libblas-dev libcairo2-dev liblapack-dev \<br />
libavcodec-dev python-epydoc<br />
</pre><br />
<br />
In Ubuntu 12.04 ''Precise Pangolin'', you can install all of Madagascar's dependencies by running<br />
<pre><br />
sudo apt-get install scons openmpi-bin libopenmpi-dev freeglut3-dev g++ gfortran libgd2-xpm-dev \<br />
libglew1.6-dev libx11-dev libxaw7-dev libnetpbm10-dev swig python-dev python-scipy python-numpy \<br />
libtiff4-dev scons units libblas-dev libcairo2-dev liblapack-dev libavcodec-dev python-epydoc<br />
</pre><br />
<br />
In Ubuntu 10.10 ''Maverick Meerkat'', you can install all of Madagascar's dependencies by running <br />
<pre><br />
sudo apt-get install freeglut3-dev g++ gfortran libgd2-xpm-dev libglew1.5-dev libjpeg62-dev \<br />
libx11-dev libxaw7-dev libnetpbm10-dev swig python-dev python-scipy python-numpy libtiff4-dev \<br />
scons units libblas-dev libcairo2-dev libavcodec-dev libplplot-dev <br />
</pre><br />
<br />
In Ubuntu 9.04 ''Jaunty Jackalope'', the corresponding command is<br />
<pre><br />
sudo apt-get install freeglut3-dev g++ gfortran libc6-dev libgd2-xpm-dev libglew1.5-dev libjpeg62-dev \<br />
libx11-dev libxaw7-dev libnetpbm10-dev swig python-dev python-scipy python-numpy libtiff4-dev scons units <br />
</pre><br />
<br />
Earlier versions may work with<br />
<pre><br />
sudo apt-get install mesa-libGL-devel g++ g77 libc6-dev libgd2-xpm-dev libglew-dev libjpeg62-dev \<br />
libx11-dev libxaw7-dev libnetpbm10-dev swig python-dev python-scipy python-numpy libtiff4-dev scons units <br />
</pre><br />
<br />
If working with the development version, you will also need <tt>git</tt> or <tt>subversion</tt>.<br />
<br />
==Fedora, RedHat, CentOS, Scientific Linux, openSUSE==<br />
<br />
Install essential Madagascar dependencies with<br />
<pre><br />
sudo yum install gcc libtirpc-devel libXaw-devel<br />
</pre><br />
<br />
To install Python in the latest versions, you can use<br />
<pre><br />
yum install python3<br />
alternatives --set python /usr/bin/python3<br />
</pre><br />
<br />
Dependency package names, sorted by Linux distribution and Madagascar feature they provide. Packages that are not included in the standard distro repositories are hyperlinked to their providers. The tables below cover build dependencies. <br />
<br />
Names of packages that are runtime dependencies are '''highlighted''' in the tables below (task under construction).<br />
<br />
''Note: In the future, it should be possible for the configuration scripts to output the dependency tables below, so that they are guaranteed to be in synch with a given Madagascar version''<br />
<br />
'''Minimal install ("Core"), publishing and development'''<br />
<br />
{| class="wikitable" align="center" cellspacing="0" border="1"<br />
|-<br />
|<br />
! style="background:#ffdead;" | Core<br />
! style="background:#ffdead;" | LaTeX<br />
! style="background:#ffdead;" | Development version<br />
! style="background:#ffdead;" | C++ API<br />
! style="background:#ffdead;" | F77 API, F90 API<br />
! style="background:#ffdead;" | Python API<br />
! style="background:#ffdead;" | Java API<br />
! style="background:#ffdead;" | Octave API<br />
! style="background:#ffdead;" | Matlab API<br />
|-<br />
! style="background:#ffdead;" | Fedora 15<br />
| binutils, gcc, glibc-headers, python<br />
| texlive-latex<br />
| subversion<br />
| gcc-c++<br />
| gcc-gfortran<br />
| numpy, swig, python-devel<br />
| Java (Sun's? IcedTea?)<br />
| octave, octave-devel<br />
| [http://www.mathworks.com/ Matlab] with Mex<br />
|-<br />
! style="background:#ffdead;" | Fedora 14<br />
| binutils, gcc, glibc-headers, python<br />
| texlive-latex<br />
| subversion<br />
| gcc-c++<br />
| gcc-gfortran<br />
| numpy, swig<br />
| Java (Sun's? IcedTea?)<br />
| octave, octave-devel<br />
| [http://www.mathworks.com/ Matlab] with Mex<br />
|-<br />
! style="background:#ffdead;" | Fedora 13<br />
| binutils, gcc, glibc-headers, python<br />
| texlive-latex<br />
| subversion<br />
| gcc-c++<br />
| gcc-gfortran<br />
| numpy, swig<br />
| Java (Sun's? IcedTea?)<br />
| octave, octave-devel<br />
| [http://www.mathworks.com/ Matlab]<br />
|-<br />
! style="background:#ffdead;" | CentOS 5<br />
| binutils, gcc, glibc-headers; python (needs [http://dag.wieers.com/rpm/FAQ.php#B2 RPMforge's RHEL5 repository])<br />
| ?<br />
| subversion<br />
| gcc-c++<br />
| gcc-gfortran<br />
| numpy, swig<br />
| Java (Sun's? IcedTea?), [http://inside.mines.edu/~dhale/jtk/ Mines JTK]<br />
| [http://www.gnu.org/software/octave/ Octave, octave-devel]<br />
| [http://www.mathworks.com/ Matlab]<br />
|-<br />
! style="background:#ffdead;" | openSUSE 11.0<br />
| gcc, python<br />
| texlive-latex<br />
| subversion<br />
| gcc-c++<br />
| gcc-fortran<br />
| [http://numpy.scipy.org/ NumPy], swig<br />
| Java (Sun's? IcedTea?), [http://inside.mines.edu/~dhale/jtk/ Mines JTK]<br />
| [http://www.gnu.org/software/octave/ Octave]<br />
| [http://www.mathworks.com/ Matlab]<br />
|}<br />
<br />
<br />
'''Graphics and visualization'''<br />
<br />
The minimum requirement is X11 graphics for running '''xtpen'''.<br />
<br />
{| class="wikitable" align="center" cellspacing="0" border="1"<br />
|-<br />
|<br />
! style="background:#ffdead;" | X11 graphics<br />
! style="background:#ffdead;" | vplot2gif<br />
! style="background:#ffdead;" | vplot2avi<br />
! style="background:#ffdead;" | Some sort of movies?<br />
! style="background:#ffdead;" | TIFF output<br />
! style="background:#ffdead;" | JPEG output<br />
! style="background:#ffdead;" | PLplot graphics<br />
! style="background:#ffdead;" | OpenGL graphics<br />
! style="background:#ffdead;" | ppm (?)<br />
! style="background:#ffdead;" | unknown<br />
|-<br />
! style="background:#ffdead;" | Fedora 13<br />
| libXaw-devel<br />
| gifsicle<br />
| ffmpeg (needs the [http://rpmfusion.org/ RPM Fusion] repository enabled)<br />
| ffmpeg-devel (needs the [http://rpmfusion.org/ RPM Fusion] repository enabled)<br />
| libtiff-devel<br />
| libjpeg-devel<br />
| plplot-devel<br />
| mesa-libGL-devel, freeglut, freeglut-devel<br />
| netpbm-devel<br />
| ?<br />
|-<br />
! style="background:#ffdead;" | openSUSE 11.0<br />
| xorg-x11-devel<br />
| ?<br />
| ?<br />
| ?<br />
| libtiff-devel<br />
| libjpeg-devel<br />
| plplot-devel<br />
| mesa-libGL-devel, freeglut, freeglut-devel<br />
| netpbm-devel<br />
| cairo-devel, gd-devel, glew-devel<br />
|}<br />
<br />
'''Numerical and file manipulation utilities'''<br />
<br />
{| class="wikitable" align="center" cellspacing="0" border="1"<br />
|-<br />
|<br />
! style="background:#ffdead;" | OpenMP<br />
! style="background:#ffdead;" | MPI<br />
! style="background:#ffdead;" | BLAS/ATLAS<br />
! style="background:#ffdead;" | Interface to the Fast Discrete Curvelet Transform<br />
! style="background:#ffdead;" | sfunits<br />
|-<br />
! style="background:#ffdead;" | Fedora 13<br />
| libgomp<br />
| openmpi, openmpi-devel; openmpi-libs (?)<br />
| blas, blas-devel, atlas, atlas-devel<br />
| [https://wave.eos.ubc.ca/Software/Licenced/ pyct]<br />
| units<br />
|-<br />
! style="background:#ffdead;" | CentOS 5<br />
| ?<br />
| ?<br />
| ?<br />
| [https://wave.eos.ubc.ca/Software/Licenced/ pyct]<br />
| [http://www.gnu.org/software/units/units.html Gnu Units]<br />
|}<br />
<br />
'''Other'''<br />
<br />
{| class="wikitable" align="center" cellspacing="0" border="1"<br />
|-<br />
|<br />
! style="background:#ffdead;" | Inclusions from Seismic Unix<br />
|-<br />
! style="background:#ffdead;" | Fedora 14, 15<br />
| [http://www.cwp.mines.edu/cwpcodes/ Seismic Unix]<br />
|-<br />
! style="background:#ffdead;" | CentOS 5<br />
| [http://www.cwp.mines.edu/cwpcodes/ Seismic Unix]<br />
<br />
|}<br />
<br />
'''Command to install all dependencies present in the public repositories'''<br />
<br />
Usually package management software will not install again a package that is already installed, so it should be safe to copy and paste the command below to a command line:<br />
<br />
{| class="wikitable" align="center" cellspacing="0" border="1"<br />
|-<br />
! style="background:#ffdead;" | Fedora 15<br />
| '''yum -y install''' binutils gcc glibc-headers scons texlive-latex subversion gcc-c++ gcc-gfortran numpy python python-devel swig octave octave-devel libgomp openmpi openmpi-devel blas blas-devel atlas atlas-devel units gifsicle ffmpeg ffmpeg-devel libtiff-devel libjpeg-devel plplot-devel mesa-libGL-devel freeglut freeglut-devel libXaw-devel netpbm-devel<br />
|-<br />
! style="background:#ffdead;" | Fedora 13<br />
| '''yum -y install''' binutils gcc glibc-headers scons texlive-latex subversion gcc-c++ gcc-gfortran numpy python swig octave octave-devel libgomp openmpi openmpi-devel blas blas-devel atlas atlas-devel units gifsicle ffmpeg ffmpeg-devel libtiff-devel libjpeg-devel plplot-devel mesa-libGL-devel freeglut freeglut-devel libXaw-devel netpbm-devel<br />
|-<br />
! style="background:#ffdead;" | CentOS 5<br />
| '''yum -y install''' binutils freeglut freeglut-devel gcc gcc-c++ gcc-gfortran glibc-headers libjpeg-devel libXaw-devel netpbm-devel<br />
|-<br />
! style="background:#ffdead;" | openSUSE 11.0<br />
| '''zypper install''' cairo-devel gcc gcc-c++ gcc-fortran gd-devel glew-devel libjpeg-devel libtiff-devel octave scons subversion texlive-latex xorg-x11-devel<br />
|}<br />
<br />
'''List of runtime dependencies only'''<br />
(needed by packagers of Madagascar in order to properly list dependencies):<br />
UNDER CONSTRUCTION<br />
<br />
===Fedora 25===<br />
The following explains how to install madagascar on Fedora 25.<br />
<br />
We use the following compiler: <code>gcc</code>, <code>g++</code> and <code>gfortran</code>.<br />
<br />
* Install some packages<br />
<syntaxhighlight lang="bash" inline style="white-space:wrap">sudo dnf install scons lapack-static lapack64-static atlas-static fftw-static gd gd-devel libtool-ltdl-devel \<br />
libXaw-devel mesa-libGL-devel freeglut-devel netpbm-devel plplot-devel ffmpeg-devel swig openmpi openmpi-devel suitesparse-devel</syntaxhighlight><br />
This installed the following packages:<br />
<pre><br />
scons-2.5.1-1.fc25.noarch<br />
lapack-static-3.6.1-2.fc25.x86_64<br />
lapack64-static-3.6.1-2.fc25.x86_64<br />
atlas-static-3.10.2-14.fc25.x86_64<br />
fftw-static-3.3.5-3.fc25.x86_64<br />
gd-2.2.4-1.fc25.x86_64<br />
gd-devel-2.2.4-1.fc25.x86_64<br />
libtool-ltdl-devel-2.4.6-13.fc25.x86_64<br />
libXaw-devel-1.0.13-4.fc25.x86_64<br />
mesa-libGL-devel-17.0.5-3.fc25.x86_64<br />
freeglut-devel-3.0.0-3.fc24.x86_64<br />
netpbm-devel-10.78.00-1.fc25.x86_64<br />
plplot-devel-5.11.1-13.fc25.x86_64<br />
ffmpeg-devel-3.1.9-1.fc25.x86_64<br />
swig-3.0.11-1.fc25.x86_64<br />
openmpi-1.10.5-1.fc25.x86_64<br />
openmpi-devel-1.10.5-1.fc25.x86_64<br />
suitesparse-devel-4.4.6-7.fc25.x86_64<br />
</pre><br />
and many other dependencies.<br />
<br />
* Get the sources from the git repository<br />
<syntaxhighlight lang="bash">git clone https://github.com/ahay/src</syntaxhighlight><br />
<br />
If you run now the <code>configure</code> script, you will get:<br />
<br />
<syntaxhighlight lang="bash"><br />
checking for Python ... /usr/bin/python<br />
checking Python version ... 2.7.13<br />
checking for RSFROOT ... /usr/local/rsf<br />
checking for SCons ... /usr/bin/scons<br />
checking SCons version ... v2.5.1.rel_2.5.1:3735:9dc6cee5c168[MODIFIED]<br />
Running RSFROOT=/usr/local/rsf /usr/bin/scons config ...<br />
------------------------<br />
scons: Reading SConscript files ...<br />
checking platform ... (cached) linux [fedora]<br />
checking for C compiler ... (cached) gcc<br />
checking if gcc works ... yes<br />
checking if gcc accepts '-x c -std=gnu99 -Wall -pedantic' ... yes<br />
checking for ar ... (cached) ar<br />
checking for libraries ... ['m']<br />
checking complex support ... yes<br />
checking for X11 headers ... /usr/include<br />
checking for X11 libraries ... /usr/lib<br />
checking for OpenGL ... yes<br />
checking for sfpen ... (cached) xtpen<br />
checking for ppm ... yes<br />
checking for tiff ... yes<br />
checking for GD (PNG) ... yes<br />
checking for GD (GIF) ... yes<br />
checking for plplot ... no<br />
<br />
Optional package: plplot-devel<br />
checking for ffmpeg ... yes<br />
checking for cairo (PNG) ... yes<br />
checking for cairo (SVG) ... yes<br />
checking for cairo (PDF) ... yes<br />
checking for jpeg ... yes<br />
checking for BLAS ... yes<br />
checking for LAPACK ... no<br />
<br />
Optional package: blas + blas-devel + atlas + atlas-devel<br />
checking for SWIG ... (cached) /usr/bin/swig<br />
checking for numpy ... (cached) yes<br />
checking API options ... (cached) []<br />
checking for C++ compiler ... (cached) g++<br />
checking if g++ works ... yes<br />
checking if g++ accepts '-std=c++11 -U__STRICT_ANSI__ -Wall -pedantic' ... yes<br />
checking for MPICC ... (cached) /usr/lib64/openmpi/bin/mpicc<br />
checking if /usr/lib64/openmpi/bin/mpicc works ... yes<br />
checking for MPICXX ... (cached) /usr/lib64/openmpi/bin/mpicxx<br />
checking if /usr/lib64/openmpi/bin/mpicxx works ... yes<br />
checking for MPIRUN ... (cached) /usr/lib64/openmpi/bin/mpirun<br />
checking for Posix threads ... yes<br />
checking for OpenMP ... yes<br />
checking for CUDA ... (cached) no<br />
checking for FFTW ... yes<br />
checking if FFTW supports threads ... yes<br />
checking for SuiteSparse ... yes<br />
checking for pfft ... no<br />
scons: done reading SConscript files.<br />
scons: Building targets ...<br />
shell_script(["env.sh"], [])<br />
shell_script(["env.csh"], [])<br />
scons: done building targets.<br />
------------------------<br />
Done with configuration.<br />
</syntaxhighlight><br />
<br />
Libraries <code>plplot</code> and <code>lapack</code> are not found. File <code>config.log</code> says<br />
<pre><br />
/bin/ld: cannot find -lplplotd<br />
</pre><br />
<br />
and<br />
<br />
<pre><br />
gcc -o .sconf_temp/sfconftest_21 .sconf_temp/conftest_21.o -L/usr/lib64/atlas -lm -lf77blas -lcblas -latlas -llapack -lcblas<br />
/usr/lib64/atlas/liblapack.a(dgesv.o): In function `dgesv_':<br />
(.text+0x69): undefined reference to `xerbla_'<br />
/usr/lib64/atlas/liblapack.a(ATL_f77wrap_dgesv.o): In function `atl_f77wrap_dgesv_':<br />
(.text+0x10d): undefined reference to `ATL_xerbla'<br />
/usr/lib64/atlas/liblapack.a(ATL_dgetf2.o): In function `ATL_dgetf2':<br />
(.text+0x184): undefined reference to `ATL_dger'<br />
...<br />
</pre><br />
<br />
So library <code>plplotd</code> doesn't exist on Fedora 25 and the <code>configure</code> script is looking for the wrong <code>lapack</code> library.<br />
<br />
If compiling now, 2 other libraries won't be found: <code>gfortran</code> and <code>quadmath</code><br />
<br />
* Regarding the <code>plplotd</code> library, edit file <code>framework/configure.py</code> and change the line 724:<br />
<syntaxhighlight lang="python" line start="724">plplot = context.env.get('PLPLOT','plplotd')</syntaxhighlight><br />
to<br />
<syntaxhighlight lang="python" line start="724">plplot = context.env.get('PLPLOT','plplot')</syntaxhighlight><br />
<br />
* As for the libraries<br />
** <code>lapack</code><br />
** <code>gfortran</code><br />
** <code>quadmath</code><br />
We must use the <code>lapack</code> library installed in <code>/usr/lib64</code>, not the one installed in <code>/usr/lib64/atlas</code>. And libraries <code>gfortran</code> and <code>quadmath</code> are installed in <code>/usr/lib/gcc/x86_64-redhat-linux/6.3.1/</code>.<br />
Edit file <code>framework/configure.py</code> and add the following 4 lines:<br />
<pre><br />
['/usr/lib/gcc/x86_64-redhat-linux/6.3.1/',<br />
'/usr/lib64/',<br />
</pre><br />
and<br />
<pre><br />
LIBS.append('gfortran')<br />
LIBS.append('quadmath')<br />
</pre><br />
<br />
so that we end up with the following <code>blas</code> function: <br />
<syntaxhighlight lang="python" line start="971">def blas(context):<br />
context.Message("checking for BLAS ... ")<br />
text = '''<br />
#ifdef __APPLE__<br />
#include <Accelerate/Accelerate.h><br />
#else<br />
#ifdef HAVE_MKL<br />
#include <mkl.h><br />
#else<br />
#include <cblas.h><br />
#endif<br />
#endif<br />
int main(int argc,char* argv[]) {<br />
float d, x[]={1.,2.,3.}, y[]={3.,2.,1.};<br />
d = cblas_sdot(3,x,1,y,1);<br />
return 0;<br />
}\n'''<br />
<br />
if plat['OS'] == 'cygwin':<br />
context.env['ENV']['PATH'] = context.env['ENV']['PATH'] + \<br />
':/lib/lapack'<br />
<br />
res = context.TryLink(text,'.c')<br />
if res: <br />
context.Result(res)<br />
context.env['BLAS'] = True <br />
else:<br />
# first try blas<br />
LIBS = path_get(context,'LIBS')<br />
blas = context.env.get('BLAS','blas')<br />
LIBS.append(blas)<br />
res = context.TryLink(text,'.c')<br />
if res: <br />
context.Result(res)<br />
context.env['LIBS'] = LIBS <br />
context.env['BLAS'] = blas <br />
else:<br />
# some systems require cblas and atlas<br />
for atlas_dir in filter(os.path.isdir,<br />
['/usr/lib/gcc/x86_64-redhat-linux/6.3.1/', # <--- add this line<br />
'/usr/lib64/', # <--- add this line<br />
'/usr/lib64/atlas/',<br />
'/usr/lib/atlas/']):<br />
context.env['LIBPATH'].append(atlas_dir)<br />
LIBS.pop()<br />
LIBS.append('f77blas')<br />
LIBS.append('cblas')<br />
LIBS.append('atlas')<br />
LIBS.append('gfortran') # <----------------------------------------------- add this line<br />
LIBS.append('quadmath') # <----------------------------------------------- add this line<br />
res = context.TryLink(text,'.c')<br />
if res: <br />
context.Result(res)<br />
context.env['LIBS'] = LIBS <br />
context.env['BLAS'] = 'cblas'<br />
else:<br />
context.Result(context_failure)<br />
context.env['CPPDEFINES'] = \<br />
path_get(context,'CPPDEFINES','NO_BLAS')<br />
LIBS.pop()<br />
LIBS.pop()<br />
LIBS.pop()<br />
context.env['BLAS'] = None <br />
need_pkg('blas', fatal=False)<br />
</syntaxhighlight><br />
* Run the <code>configure</code> script, for example:<br />
<syntaxhighlight lang="bash">./configure --prefix=/usr/local/rsf</syntaxhighlight><br />
Output should give:<br />
<syntaxhighlight lang="bash"><br />
checking for Python ... /usr/bin/python<br />
checking Python version ... 2.7.13<br />
checking for RSFROOT ... /usr/local/rsf<br />
checking for SCons ... /usr/bin/scons<br />
checking SCons version ... v2.5.1.rel_2.5.1:3735:9dc6cee5c168[MODIFIED]<br />
Running RSFROOT=/usr/local/rsf /usr/bin/scons config ...<br />
------------------------<br />
scons: Reading SConscript files ...<br />
checking platform ... (cached) linux [fedora]<br />
checking for C compiler ... (cached) gcc<br />
checking if gcc works ... yes<br />
checking if gcc accepts '-x c -std=gnu99 -Wall -pedantic' ... yes<br />
checking for ar ... (cached) ar<br />
checking for libraries ... ['m']<br />
checking complex support ... yes<br />
checking for X11 headers ... /usr/include<br />
checking for X11 libraries ... /usr/lib<br />
checking for OpenGL ... yes<br />
checking for sfpen ... (cached) xtpen<br />
checking for ppm ... yes<br />
checking for tiff ... yes<br />
checking for GD (PNG) ... yes<br />
checking for GD (GIF) ... yes<br />
checking for plplot ... yes<br />
checking for ffmpeg ... yes<br />
checking for cairo (PNG) ... yes<br />
checking for cairo (SVG) ... yes<br />
checking for cairo (PDF) ... yes<br />
checking for jpeg ... yes<br />
checking for BLAS ... yes<br />
checking for LAPACK ... yes<br />
checking for SWIG ... (cached) /usr/bin/swig<br />
checking for numpy ... (cached) yes<br />
checking API options ... (cached) []<br />
checking for C++ compiler ... (cached) g++<br />
checking if g++ works ... yes<br />
checking if g++ accepts '-std=c++11 -U__STRICT_ANSI__ -Wall -pedantic' ... yes<br />
checking for MPICC ... (cached) /usr/lib64/openmpi/bin/mpicc<br />
checking if /usr/lib64/openmpi/bin/mpicc works ... yes<br />
checking for MPICXX ... (cached) /usr/lib64/openmpi/bin/mpicxx<br />
checking if /usr/lib64/openmpi/bin/mpicxx works ... yes<br />
checking for MPIRUN ... (cached) /usr/lib64/openmpi/bin/mpirun<br />
checking for Posix threads ... yes<br />
checking for OpenMP ... yes<br />
checking for CUDA ... (cached) no<br />
checking for FFTW ... yes<br />
checking if FFTW supports threads ... yes<br />
checking for SuiteSparse ... yes<br />
checking for pfft ... no<br />
scons: done reading SConscript files.<br />
scons: Building targets ...<br />
shell_script(["env.sh"], [])<br />
shell_script(["env.csh"], [])<br />
scons: done building targets.<br />
------------------------<br />
Done with configuration.<br />
</syntaxhighlight><br />
* Then run the compilation:<br />
<syntaxhighlight lang="bash">make</syntaxhighlight><br />
<br />
* and finally the installation<br />
<syntaxhighlight lang="bash">sudo make install</syntaxhighlight><br />
<br />
==Debian 5==<br />
Specific dependencies:<br />
* Debian 5.0 ("Lenny"): Please make sure you have the <tt>libc6-dev</tt> package before trying to compile from source. The <tt>libXaw7-dev</tt> package might be a dependency for <tt>xtpen</tt> (was in Debian 4.0)<br />
<br />
==Yellow Dog Linux 6.1 on Sony PlayStation 3==<br />
See [http://www.reproducibility.org/rsflog/uploads/Friday_Seminar_Madagascar_on_PS3.ppt Will Burnett's guide (PowerPoint)]<br />
<br />
==Mac OS X==<br />
<br />
Install necessary Mac OS X applications using:<br />
* [http://www.macports.org/ MacPorts], an easy-to-use system for compiling, installing, and upgrading open-source software on Mac OS X.<br />
<br />
Installation on Mac OS X requires the following:<br />
# <b>Xcode</b>: Download and install [https://developer.apple.com/xcode/ XCode] from Apple using App Store application. In Xcode, make sure to enable and install the command-line tools in Xcode/Preferences/Downloads <br />
# (Optionally) <b>X11</b>: Install X11 libraries from [http://xquartz.macosforge.org Xquartz]. <br />
# (Optionally) <b>gcc</b>: Install the Gnu C compiler from [http://hpc.sourceforge.net HPC Mac OS X]. <br />
# (Optionally) [[SEGTeX]]: To use <b>SEGTeX</b>, you may need [http://www.tug.org/texlive/ TeX Live]. <b>MacPorts</b> provide an easy way to install it with commands <pre>sudo port install texlive texlive-latex-extra texlive-fontutils</pre><br />
# (Optionally) <b>FFTW</b>: Using <b>MacPorts</b>, run <pre>sudo port install fftw-3-single</pre><pre>sudo port install fftw-3 +gfortran</pre><br />
# (Optionally) <b>MPI</b>: Using <b>MacPorts</b>, run <pre>sudo port install openmpi</pre><pre>sudo port select --set mpi openmpi-mp-fortran</pre><br />
# (Optionally) <b>ppmpen</b>: Using <b>MacPorts</b>, run <pre>sudo port install libnetpbm</pre><br />
# (Optionally) <b>Open vplots using double click</b>: [[machacks|Instructions]].<br />
# (Optionally) You can install Python with libraries including jupyter using the Anaconda distribution, available at https://www.anaconda.com/download<br />
# (Optionally) <b>SWIG</b> can be installed with <pre>conda install swig</pre> or <pre>sudo port install swig-python</pre> Additionally, run<pre>ln -s `which swig` /usr/local/bin</pre> to get ./configure to find swig. <br />
#Proceed with configuration and installation following the normal procedure. Build the system with: <pre>make install</pre><br />
# To use Apple's compiler (clang) instead of gcc, run <tt>./configure</tt> with options <pre>./configure CC=clang CXX=clang++</pre><br />
<br />
==OpenSolaris==<br />
<br />
Use <tt>pkg</tt> to install missing components such as X11 headers.<br />
<br />
<syntaxhighlight lang="bash"><br />
pfexec pkg install SUNWxorg-headers<br />
</syntaxhighlight><br />
<br />
==MS Windows==<br />
Due to its size, this topic has been assigned [[Windows | its own Wiki page]].<br />
<br />
==How to adapt Madagascar to a new platform==<br />
The most laborious part of adapting madagascar to a new platform is finding the proper dependency names. This usually proceeds as follows: dependency X fails with a "missing file" error either as a header file in <tt>config.log</tt>, or a missing library during the build step. Possible package names are found through an internet search for the missing file name and the distribution name or by using specific [http://rpm.pbone.net/ rpm search tools]. Packages are installed and the configure (and, if necessary) build processes are repeated until the error goes away.<br />
<br />
=Multi-user installs=<br />
Some organizations may find it desirable to deny write access of some users to all RSFSRC/RSFROOT except their own user directory. Fortunately, this can be easily done by placing the restricted user dirs outside RSFSRC/RSFROOT, i.e. in their home dirs, say /home/joe/rsfsrc. In order to move a user's directory out of RSFSRC, you must:<br />
* "tell" the SConstruct in the user's dir where to find RSFSRC so that when the user builds in his directory, it can import <tt>configure.py</tt> and <tt>config.py</tt> You do that by setting the environment variable RSFSRC to the absolute path of the Madagascar source root, and by making sure that lines 2 and 3 in the users' SConstruct files are<br />
<syntaxhighlight lang="python"><br />
srcroot = os.environ.get('RSFSRC', '../..')<br />
sys.path.append(srcroot)<br />
</syntaxhighlight> <br />
and then replace <tt>../..</tt> throughout the SConstruct using <tt>os.path.join</tt> and the <tt>srcroot</tt> variable.<br />
* "tell" the build scripts about the user's dir, so that it is included in the builds launched from RSFSRC. You do that with a symbolic link:<br />
<syntaxhighlight lang="bash"><br />
ln -s /home/joe/rsfsrc $RSFSRC/user/joe<br />
</syntaxhighlight><br />
''When the link exists'', those of Joe's programs that are mentioned in the "prog" string in SConstruct get included in the distribution, complete with self-doc. If Joe is just learning how to code and his stuff breaks the build, just remove the symbolic link. Even if build+installs are done after the link is removed, his stable programs and self-doc will continue to remain installed system-wide as long as the admin does not type <tt>scons -c install</tt> (not likely).<br />
* point the user's RSFDOC environment variable to a location where the user has write access<br />
* edit the users' SConstruct so that it uses the RSF library and headers already installed in $RSFROOT/lib and $RSFROOT/include , instead of building again the whole <tt>librsf</tt> with user-specific flags in <tt>RSFSRC/filt/lib/</tt>. To do that, replace in the user's SConstruct the env.Prepend statement with<br />
<syntaxhighlight lang="python"><br />
rsfroot = os.environ.get('RSFROOT','/usr/local/rsf')<br />
<br />
env.Prepend(CPPPATH=[os.path.join(rsfroot,'include')],<br />
LIBPATH=[os.path.join(rsfroot,'lib')],<br />
LIBS=['rsf'])<br />
</syntaxhighlight><br />
* If the link from RSFSRC to Joe's directory was not made, add Joe's directory to his own path so that he can execute his own binaries.<br />
<br />
To understand how $DATAPATH disk space issues may become an issue in a multi-user environment, refer to the [[Advanced_Installation#Disk_space|Disk Space subsection]] at the beginning of this document.<br />
<br />
=Keeping your stuff separate=<br />
A user may add his own programs and recipes to the Madagascar system. He may also create his own computational examples, data, and locked figures for testing. All of these components can be placed in their default locations, but it is not necessary to make them public. To keep these items private simply do not add them to the repository.<br />
<br />
However, it might be desirable to keep these components in separate places. For example, if you keep your private programs in RSFSRC/user you will have to remember to make a copy somewhere else if you ever want to delete the Madagascar installation to perform a fresh install. Yup, I deleted all my programs that way once. Good thing I had a back up! Fortunately, it is easy to keep each of these components in a separate place if desired.<br />
<br />
==Keeping programs separate==<br />
User programs are ordinarily kept in a subdirectory of RSFSRC/user. However, if you want to keep your programs separate all you have to do is put your subdirectory somewhere else and make a link to it in RSFSRC/user:<br />
<bash><br />
ln -s path_to_my_programs $RSFSRC/user/my_programs<br />
</bash><br />
The additional instructions above for "multi-user installs" are for the case where the other users do not have write access to RSFSRC. However, if you have full write access and only want to keep the programs in a separate place the link is the only thing you need.<br />
<br />
===Problems when symbolic linking and using SVN=== <br />
If you decide to use a symbolic link e.g. "ln -s ..." as above, you will have difficulties checking in your code into the SVN repository. SVN typically does not follow the symbolic link but will only the save the symbolic link. <br />
<br />
See this answer from Stackoverflow (http://stackoverflow.com/questions/24850844/follow-symlinks-in-svn)<br />
<br />
<br />
<blockquote><br />
<br />
<br />
If I understand your problem, you have:<br />
<br />
project1/<br />
project1/link1 -> ../../some/where/else<br />
<br />
If you do a simple svn add link1 it adds a symlink entry to the subversion repository, but what you're trying to accomplish is getting the stuff under somewhere else into the tree.<br />
<br />
If this is the case, then you're fighting in the wrong direction, you should make the real files and directories under link1, and make the target locations symlinks into the link1 directory. That would be a simple solution to the problem.<br />
<br />
Another solution would be to make the location ../../some/where/else an svn location in it's own right, and make link1 an externals definition to that location. When you commit in the main directory, the externals would be committed at the same time which would achieve storing the information; however to ensure that the other location was kept in sync, you would have to make sure to update it to the same version as the stored data.<br />
<br />
In my case, on my desktop, I have a settings directory:<br />
<br />
$HOME/settings<br />
<br />
This is a checked out copy of a directory containing .bashrc, .profile, .vimrc, .vim, etc. files and folders from an svn repo. All my profile files and directories were symlinks into this folder, so for example .profile -> settings/.profile, etc. When I make changes, they are reflected in the svn tree, which I can commit back to in order to ensure that I don't lose my config settings when I go to another system.<br />
<br />
If you really want svn to follow symlinks as hardlinks (You can't make directory hardlinks because it would be a bad thingâ„¢), then you'll have to hack the svn client source so that it did what you wanted; but this would be a maintenance nightmare.<br />
<br />
You could get away with making the symlink into a bind mount point directed at the target, but that has it's own issues - you would need to be root to accomplish this, and it ends up leaving funny entries in your /proc/mounts to accomplish this:<br />
<br />
mount --bind /absolute/path/to/some/where/else project1/link1<br />
</blockquote><br />
<br />
==Keeping recipes separate==<br />
Computational recipes written in Python and imported by the SConstruct file of a workflow are normally stored in RSFSRC/book/Recipes. The install process copies these recipes to a directory like $RSFROOT/lib/python2.5/site-packages/rsf/recipes and adds this directory to your PYTHONPATH so that Python can find them.<br />
<br />
However, you can put you own recipes anywhere you want. You only have to add that place to your PYTHONPATH like this (bash):<br />
<syntaxhighlight lang="bash"><br />
export PYTHONPATH=${PYTHONPATH}:path_to_my_recipes<br />
</syntaxhighlight><br />
Or like this (csh):<br />
<syntaxhighlight lang="bash"><br />
setenv PYTHONPATH ${PYTHONPATH}:path_to_my_recipes<br />
</syntaxhighlight><br />
<br />
==Keeping examples separate==<br />
Madagascar's public collection of example workflows are stored in RSFSRC/book, but you can put your private workflows anywhere you want. No special instructions are required.<br />
<br />
However, Madagascar assumes that the workflows are organized into a three-level book/chapter/section directory hierarchy when it creates a directory tree for the data and locked figures associated with your workflow. It is not required, but it might be easier to find the data and locked figures if you put your workflows in a three-level directory tree something like this: path_to_my_book/chapter/section/SConstruct.<br />
<br />
==Keeping data separate==<br />
The location of the data portion of your *.rsf files is controlled by your DATAPATH environment variable. However, you may want to keep the data for your private workflows in a different place, or several different places, than the data created by the public examples in RSFSRC/book. The way to do that is to temporarily change the DATAPATH variable in the SConstruct ''before'' importing rsf.proj like this:<br />
<syntaxhighlight lang="python"><br />
import os<br />
os.environ['DATAPATH'] = 'path_to_my_private_data'<br />
<br />
from rsf.proj import *<br />
</syntaxhighlight><br />
<br />
==Keeping locked figures separate==<br />
The command "scons lock" in the directory of a workflow will store a "locked" copy of your figures for regression testing. Normally these figures are stored in the location pointed to by your RSFFIGS variable, and that is where the figures from the figures repository should be stored for testing in RSFSRC/book.<br />
<br />
However, you may want to keep your private figures in a different place. The way to do that is to temporarily change the RSFFIGS variable in the SConstruct ''before'' importing rsf.proj like this:<br />
<syntaxhighlight lang="python"><br />
import os<br />
os.environ['RSFFIGS'] = 'path_to_my_private_figures'<br />
<br />
from rsf.proj import *<br />
</syntaxhighlight><br />
If you also create a RSFALTFIGS environment variable pointing to path_to_my_private_figures, then the testing script sffiglist will automatically test your figures against those in RSFALTFIGS when the sffiglist command is executed from a location outside of RSFSRC/book.<br />
<br />
=Capturing error and warning messages=<br />
The messages during configuration are few and their importance quite high, so they should be watched "in person". A complete log of the configuration process is recorded in RSFSRC/configure.log<br />
<br />
Console messages generated during the build step can be captured to a log file and observed at the same time with a command like this (tcsh):<br />
<syntaxhighlight lang="bash"><br />
nice +10 nohup /usr/bin/time -p scons -k |& tee log_build.asc<br />
</syntaxhighlight><br />
The log file can be of course named otherwise than <tt>log_build.asc</tt>. The file can be later grepped for error and warnings with commands such as:<br />
<syntaxhighlight lang="bash"><br />
grep -c error log_build.asc<br />
grep error log_build.asc | awk '/error.c/ {next}; /error.h/ {next}; /error.o/ {next}; {print}'<br />
grep -c warning log_build.asc<br />
grep warning log_build.asc | awk '/imaginary constants are a GCC extension/ {next}; {print}'<br />
</syntaxhighlight><br />
<br />
=Advanced troubleshooting=<br />
* If you removed one of your programs or changed its name, and <tt>scons install</tt> fails with "Source <tt>oldprogname</tt> not found, needed by target install", and you cleaned everything there was to clean but still get this message, remove <tt>RSFSRC/.sconsign*</tt><br />
* If during <tt>scons install</tt> you get a <tt>DBAccessError : (13, 'Permission denied')</tt> in some reproducible papers, check permissions in your <tt>$DATAPATH</tt> directory. This is where SCons places database ".sconsign" files for its dependencies (according to the rules in <tt>rsf.proj</tt> and <tt>rsf.tex</tt>).<br />
* If <tt>scons</tt> or <tt>scons install</tt> fail due to an a bug introduced in a tool you are certain you will not use, a quick workaround for the problem is already built into scons: the <tt>-k</tt> option, which means "keep going". Thus, if you use <tt>scons -k</tt> or <tt>scons -k install</tt>, SCons will not be able to build the failed component, or anything that depends on it, but it will keep going and make everything else that it can.<br />
<br />
=Further support=<br />
Subscribe to the [https://lists.sourceforge.net/lists/listinfo/rsf-user rsf-user mailing list].</div>Sergeyhttps://www.reproducibility.org/wiki2020/index.php?title=Guide_to_RSF_file_format&diff=3926Guide to RSF file format2021-03-08T23:09:32Z<p>Sergey: /* Reading and writing CSV files */</p>
<hr />
<div><center><font size="-1">''This page was created from the LaTeX source in [https://github.com/ahay/src/blob/master/book/rsf/rsf/format.tex book/rsf/rsf/format.tex] using [[latex2wiki]]''</font></center><br />
<br />
<br />
[[Image:Fotolia_9592362_XS.jpg|right|]]<br />
<br />
==Principles==<br />
The main design principle behind the RSF data format is [http://en.wikipedia.org/wiki/KISS_principle KISS] ("Keep It<br />
Short and Simple"). The RSF format is borrowed from the SEPlib data format<br />
originally designed at the Stanford Exploration Project<br />
(Claerbout, 1991<ref>Claerbout, J. F., 1991, Introduction to Seplib and SEP utility software, ''in'' SEP-70, 413--436. Stanford Exploration Project.</ref>). The format is made as simple as possible for<br />
maximum convenience, transparency and flexibility.<br />
According to the Unix tradition, common file formats should be in a readable<br />
textual form so that they can be easily examined and processed with universal<br />
tools. Raymond (2004<ref>Raymond, E. S., 2004, The art of UNIX programming: Addison-Wesley.</ref>) writes:<br />
<blockquote><br />
To design a perfect anti-Unix, make all file formats binary and opaque, and<br />
require heavyweight tools to read and edit them.<br />
</blockquote><br />
<blockquote><br />
If you feel an urge to design a complex binary file format, or a complex<br />
binary application protocol, it is generally wise to lie down until the<br />
feeling passes.<br />
</blockquote><br />
Storing large-scale datasets in a text format may not be economical. RSF<br />
chooses the next best thing: it allows data values to be stored in a binary<br />
format but puts all data attributes in text files that can be read by humans<br />
and processed with universal text-processing utilities.<br />
===Example===<br />
Let us first create some synthetic RSF data.<br />
<pre><br />
bash$ sfmath n1=1000 output='sin(0.5*x1)' > sin.rsf<br />
</pre><br />
Open and read the file <tt>sin.rsf</tt>.<br />
<pre><br />
bash$ cat sin.rsf<br />
sfmath rsf/rsf/rsftour: fomels@egl Sun Jul 31 07:18:48 2005<br />
<br />
o1=0<br />
data_format="native_float"<br />
esize=4<br />
in="/tmp/sin.rsf@"<br />
x1=0<br />
d1=1<br />
n1=1000<br />
</pre><br />
The file contains nine lines with simple readable text. The first line<br />
shows the name of the program, the working directory, the user and<br />
computer that created the file and the time it was created (that<br />
information is recorded for accounting purposes). Other lines contain<br />
parameter-value pairs separated by the "=" sign. The "in"<br />
parameter points to the location of the binary data. Before we discuss<br />
the meaning of parameters in more detail, let us plot the data.<br />
<pre><br />
bash$ < sin.rsf sfwiggle title='One Trace' | sfpen<br />
</pre><br />
On your screen, you should see a plot similar to the figure below.<br />
[[Image:sin1.png|frame|center|An example sinusoid plot.]]<br />
Suppose you want to reformat the data so that instead of one trace of a<br />
thousand samples, it contains twenty traces with fifty samples each. Try<br />
running<br />
<pre><br />
bash$ < sin.rsf sed 's/n1=1000/n1=50 n2=20/' > sin10.rsf <br />
bash$ < sin10.rsf sfwiggle title=Traces | sfpen<br />
</pre><br />
or (using pipes)<br />
<pre><br />
bash$ < sin.rsf sed 's/n1=1000/n1=50 n2=20/' | sfwiggle title=Traces | sfpen<br />
</pre><br />
On your screen, you should see a plot similar to the figure below:<br />
[[Image:sin2.png|frame|center|An example sinusoid plot, with data reformatted to twenty traces.]]<br />
What happened? We used <tt>sed</tt>, a standard Unix line editing utility to<br />
change the parameters describing the data dimensions. Because of the<br />
simplicity of this operation, there is no need to create specialized data<br />
formatting tools or to make the <tt>sfwiggle</tt> program accept additional<br />
formatting parameters. Other general-purpose Unix tools that can be applied on<br />
RSF files include <tt>cat</tt>, <tt>echo</tt>, <tt>grep</tt>, etc. <br />
An alternative way to obtain the previous result is to run<br />
<pre><br />
bash$ ( cat sin.rsf; echo n1=50 n2=20 ) > sin10.rsf <br />
bash$ < sin10.rsf sfwiggle title=Traces | sfpen<br />
</pre><br />
In this case, the <tt>cat</tt> utility simply copies the contents of the<br />
previous file, and the <tt>echo</tt> utility appends new line "<tt>n1=50<br />
n2=20</tt>". A new value of the <tt>n1</tt> parameter overwrites the old value<br />
of <tt>n1=1000</tt>, and we achieve the same result as before.<br />
Of course, one could also edit the file by hand with one of the general<br />
purpose text editors. For recording the history of data processing, it is<br />
usually preferable to be able to process files with non-interactive tools.<br />
<br />
==Header and Data files==<br />
A simple way to check the layout of an RSF file is with the <tt>sfin</tt><br />
program.<br />
<pre><br />
bash$ sfin sin10.rsf<br />
sin10.rsf:<br />
in="/tmp/sin.rsf@"<br />
esize=4 type=float form=native<br />
n1=50 d1=1 o1=0<br />
n2=20 d2=? o2=?<br />
1000 elements 4000 bytes<br />
</pre><br />
The program reports the following information: the location of the data file<br />
(<tt>/tmp/sin.rsf\@</tt>), the element size (4 bytes), the element<br />
type (floating point), the element form (native), the hypercube dimensions<br />
(<math>50 \times 20</math>), axis scaling (1 and unspecified), and axis origin (0 and<br />
unspecified). It also checks the total number of elements and bytes in the<br />
data file.<br />
Let us examine this information in detail. First, we can verify that the data<br />
file exists and contains the specified number of bytes:<br />
<pre><br />
bash$ ls -l /tmp/sin.rsf@<br />
-rw-r--r-- 1 sergey users 4000 2004-10-04 00:35 /tmp/sin.rsf@<br />
</pre><br />
4000 bytes in this file are required to store <math>50 \times 20</math> floating-point<br />
4-byte numbers in a binary form. Thus, the data file contains nothing but the<br />
raw data in a contiguous binary form.<br />
===Datapath===<br />
How did the RSF program (<tt>sfmath</tt>) decide where to put the data file?<br />
In the order of priority, the rules for selecting the data file name and the<br />
data file directory are as follows:<br />
<br />
#Check <tt>--out=</tt> parameter on the command line. The parameter specifies the output data file location explicitly. <br />
#Specify the path and the file name separately. <br />
#*The rules for the path selection are: <br />
#*#Check <tt>datapath=</tt> parameter on the command line. The parameter specifies a string to prepend to the file name. The string may contain the file directory. <br />
#*#Check <tt>DATAPATH</tt> environmental variable. It has the same meaning as the parameter specified with <tt>datapath=</tt>. <br />
#*#Check for <tt>.datapath</tt> file in the current directory. The file may contain a line <pre> datapath=/path/to_file/ </pre> or <pre> machine_name datapath=/path/to_file/ </pre> if you indent to use different paths on different platforms. <br />
#*#Check for <tt>.datapath</tt> file in the user home directory. <br />
#*#Put the data file in the current directory (similar to <tt>datapath=./</tt>). <br />
#*: <br />
#*The rules for the filename selection are: <br />
#*#If the output RSF file is in the current directory, the name of the data file is made by appending \@. <br />
#*#If the output file is not in the current directory or if it is created temporarily by a program, the name is made by appending random characters to the name of the program and selected to be unique. <br />
<br />
Examples:<br />
<br />
<pre><br />
bash$ sfspike n1=10 --out=test1 > spike.rsf <br />
bash$ grep in spike.rsf <br />
in="test1" <br />
</pre> <br />
<br />
<pre><br />
bash$ sfspike n1=10 datapath=/tmp/ > spike.rsf <br />
bash$ grep in spike.rsf <br />
in="/tmp/spike.rsf@" <br />
</pre> <br />
<br />
<pre><br />
bash$ DATAPATH=/tmp/ sfspike n1=10 > spike.rsf <br />
bash$ grep in spike.rsf <br />
in="/tmp/spike.rsf@" <br />
</pre> <br />
<br />
<pre><br />
bash$ sfspike n1=10 datapath=/tmp/ > /tmp/spike.rsf <br />
bash$ grep in /tmp/spike.rsf <br />
in="/tmp/sfspikejcARVf" <br />
</pre> <br />
<br />
====Packing header and data together====<br />
While the header and data files are separated by default, it is also possible<br />
to pack them together into one file. To do that, specify the program's<br />
"<tt>--out</tt>" parameter as <tt>--out=stdout</tt>. Example:<br />
<pre><br />
bash$ sfspike n1=10 --out=stdout > spike.rsf<br />
bash$ grep in spike.rsf<br />
Binary file spike.rsf matches<br />
bash$ sfin spike.rsf<br />
spike.rsf:<br />
in="stdin"<br />
esize=4 type=float form=native<br />
n1=10 d1=0.004 o1=0 label1="Time" unit1="s"<br />
10 elements 40 bytes<br />
bash$ ls -l spike.rsf<br />
-rw-r--r-- 1 sergey users 196 2004-11-10 21:39 spike.rsf<br />
</pre><br />
If you examine the contents of <tt>spike.rsf</tt>, you will find that it<br />
starts with the text header information, followed by special<br />
symbols, followed by binary data. <br />
Packing headers and data together may not be a good idea for data processing<br />
but it works well for storing data: it is easier to move the packed file<br />
around than to move two different files (header and binary) together while<br />
remembering to preserve their connection. Packing header and data together is<br />
also the current mechanism used to push RSF files through Unix pipes.<br />
<br />
===Type===<br />
The data stored with RSF can have different types: character, unsigned<br />
character, integer, floating point, or complex. By default, single precision<br />
is used for numbers (<tt>int</tt> and <tt>float</tt> data types in the C<br />
programming language) but double precision and other<br />
integer types (<tt>short</tt> and <tt>long</tt>) are also<br />
supported. The number of bytes required for represent these<br />
numbers may depend on the platform.<br />
===Form===<br />
The data stored with RSF can also be in a different form: ASCII, native<br />
binary, and XDR binary. Native binary is often used by default. It is the<br />
binary format employed by the machine that is running the application. On<br />
Linux-running PC, the native binary format will typically correspond to the<br />
so-called little-endian byte ordering. On some other platform, it might be<br />
big-endian ordering. XDR is a binary format designed by Sun for exchanging<br />
files over network. It typically corresponds to big-endian byte ordering. It<br />
is more efficient to process RSF files in the native binary format but, if you<br />
intend to access data from different platforms, it might be a good idea to<br />
store the corresponding file in an XDR format. RSF also allows for an ASCII<br />
(plain text) form of data files. <br />
Conversion between different types and forms is accomplished with<br />
<tt>sfdd</tt> program. Here are some examples. First, let us create synthetic<br />
data.<br />
<pre><br />
bash$ sfmath n1=10 output='10*sin(0.5*x1)' > sin.rsf<br />
bash$ sfin sin.rsf<br />
sin.rsf:<br />
in="/tmp/sin.rsf@"<br />
esize=4 type=float form=native<br />
n1=10 d1=1 o1=0<br />
10 elements 40 bytes<br />
bash$ < sin.rsf sfdisfil<br />
0: 0 4.794 8.415 9.975 9.093<br />
5: 5.985 1.411 -3.508 -7.568 -9.775<br />
</pre><br />
Converting the data to the integer type:<br />
<pre><br />
bash$ < sin.rsf sfdd type=int > isin.rsf<br />
bash$ sfin isin.rsf<br />
isin.rsf:<br />
in="/tmp/isin.rsf@"<br />
esize=4 type=int form=native<br />
n1=10 d1=1 o1=0<br />
10 elements 40 bytes<br />
bash$ < isin.rsf sfdisfil<br />
0: 0 4 8 9 9 5 1 -3 -7 -9<br />
</pre><br />
Converting the data to the ASCII form:<br />
<pre><br />
bash$ < sin.rsf sfdd form=ascii > asin.rsf<br />
bash$ < asin.rsf sfdisfil<br />
0: 0 4.794 8.415 9.975 9.093<br />
5: 5.985 1.411 -3.508 -7.568 -9.775<br />
bash$ sfin asin.rsf<br />
asin.rsf:<br />
in="/tmp/asin.rsf@"<br />
esize=0 type=float form=ascii<br />
n1=10 d1=1 o1=0<br />
10 elements<br />
bash$ cat /tmp/asin.rsf@<br />
0 4.79426 8.41471 9.97495 9.09297 5.98472 1.4112 -3.50783<br />
-7.56803 -9.7753<br />
</pre><br />
<br />
===Hypercube===<br />
While RSF stores binary data in a contiguous 1-D array, the conceptual<br />
data model is a multidimensional hypercube. By convention, the<br />
dimensions of the cube are defined with parameters <tt>n1</tt>,<br />
<tt>n2</tt>, <tt>n3</tt>, etc. The fastest axis is <tt>n1</tt>.<br />
Additionally, the grid sampling can be given by parameters<br />
<tt>d1</tt>, <tt>d2</tt>, <tt>d3</tt>, etc. The axes origins are given<br />
by parameters <tt>o1</tt>, <tt>o2</tt>, <tt>o3</tt>, etc. Optionally,<br />
you can also supply the axis label strings: <tt>label1</tt>,<br />
<tt>label2</tt>, <tt>label3</tt>, etc., and axis units strings:<br />
<tt>unit1</tt>, <tt>unit2</tt>, <tt>unit3</tt>, etc. <br />
==Compatibility with other file formats==<br />
It is possible to exchange RSF-formatted data with several other popular data formats.<br />
===Compatibility with SEPlib===<br />
RSF is mostly compatible with its predecessor, the SEPlib file format.<br />
However, there are several important differences:<br />
<br />
#SEPlib programs typically use the element size (<tt>esize=</tt> parameter) to distinguish between different data types: <tt>esize=4</tt> corresponds to floating point data, while <tt>esize=8</tt> corresponds to complex data. The RSF type handling mechanism is different: data types are determined from the value of the <tt>data_format</tt> parameter. Madagascar computational programs typically output files with <tt>data_format="native_float"</tt> or <tt>native_complex</tt>.<br />
#The default data form in SEPlib programs is typically XDR and not native as it is in RSF. Thus, to make a dataset created with SEPlib readable by Madagascar programs, you would typically need to add to the history file <tt>data_format="xdr_float"</tt> or <tt>data_format="xdr_complex"</tt> . <ref group="note">For SEPlib 6.5.3 and older: Note that xdr_complex is not a valid SEPlib value, so for datasets of complex numbers encoded as pairs of floats, a dataset cannot be at the same time valid in both SEPlib and Madagascar. A valid SEPlib dataset will have esize=8 and data_format="xdr_float", but sfin will show it as having "200% of expected" data. Adding data_format="xdr_complex" to such a dataset will make sfin work as expected, but SEPlib's In or In3d will give a segmentation fault because of an unknown data type. To patch SEPlib to accept <tt>native_complex</tt> and <tt>xdr_complex</tt> data, the following changes must be made:<br />
* In <tt>$SEPSRC/seplib_base/lib/corelibs/sep/strformats.c</tt>:<br />
** Add "xdr_complex" and "native_complex" to the str_fmt_names structure<br />
** Set FMT_LENGTH to 15<br />
* In <tt>$SEPSRC/seplib_base/lib/corelibs/include/strformats.h</tt>:<br />
** Add preprocessor directives to define FMT_XDR_COMPLEX as 8 and FMT_NATIVE_COMPLEX as 9<br />
** Set NUM_FMT to 10<br />
</ref><br />
#It is possible to pipe the output of Madagascar programs to SEPlib: <pre>bash$ sfspike n1=1 | Attr want=min</pre> (output should be: <tt>minimum value = 1 at 1</tt>). However, piping the output of SEPlib programs to RSF (or, for that matter, any other non-SEPlib programs) will result in an unterminated process. For example, the command <pre> bash$ Spike n1=1 | sfattr want=min </pre> will hang. This is because SEPlib uses sockets for piping and expects a socket connection from the receiving program, while Madagascar passes data through regular Unix pipes. <br />
#SEP3D is an extension of SEPlib for operating with irregularly sampled data (Biondi et al., 1996<ref>Biondi, B., R. Clapp, and S. Crawley, 1996, SEPlib90: SEPlib for 3-D prestack data, ''in'' SEP-92, 343--364. Stanford Exploration Project.</ref>). There is no equivalent of it in RSF for the reasons explained in the beginning of this guide. Operations with irregular datasets are supported through the use of auxiliary input files that represent the geometry information.<br />
<br />
;Notes<br />
<references group="note" /><br />
<br />
===Reading and writing SEG-Y and SU files===<br />
The SEG-Y format is based on the proposal of Barry et al. (1975<ref>[http://www.seg.org/SEGportalWEBproject/prod/SEG-Publications/Pub-Technical-Standards/Documents/seg_y_rev0.pdf Barry, K. M., D. A. Cavers, and C. W. Kneale, 1975, Report on recommended standards for digital tape formats: Geophysics, '''40''', 344--352]</ref>).<br />
It was revised in 2002<ref>See http://www.seg.org/SEGportalWEBproject/prod/SEG-Publications/Pub-Technical-Standards/Documents/seg_y_rev1.pdf</ref>. The<br />
SU format is a modification of SEG-Y used in Seismic Unix<br />
(Stockwell, 1997<ref>Stockwell, J. W., 1997, Free software in education: A case study of CWP/SU: Seismic Unix: The Leading Edge, '''16''', 1045--1049.</ref>).<br />
To convert files from SEG-Y or SU format to RSF, use the <tt>sfsegyread</tt><br />
program. Let us first manufacture an example file using SU utilities<br />
(Stockwell, 1999<ref>-------- 1999, The CWP/SU: Seismic Un*x package: Computers and Geosciences, '''25''', 415--419.</ref>):<br />
<pre><br />
bash$ suplane > plane.su<br />
bash$ segyhdrs < plane.su | segywrite tape=plane.segy<br />
</pre><br />
To convert it to RSF, use either<br />
<pre><br />
bash$ sfsuread < plane.su tfile=tfile.rsf endian=0 > plane.rsf<br />
</pre><br />
or<br />
<pre><br />
bash$ sfsegyread < plane.segy tfile=tfile.rsf \<br />
hfile=file.asc bfile=file.bin > plane.rsf<br />
</pre><br />
The endian flag is needed if the SU file originated from a little-endian<br />
machine such as Linux PC.<br />
Several files are generated. The standard output contains an RSF file with the<br />
data (32 traces with 64 samples each):<br />
<pre><br />
bash$ sfin plane.rsf<br />
plane.rsf:<br />
in="/tmp/plane.rsf@"<br />
esize=4 type=float form=native<br />
n1=64 d1=0.004 o1=0<br />
n2=32 d2=? o2=?<br />
2048 elements 8192 bytes<br />
</pre><br />
The contents of this file are displayed in the figure.<br />
[[Image:plane.png|frame|center|The output of suplane, converted to RSF and<br />
displayed with <tt>sfwiggle</tt>.]]<br />
The <tt>tfile</tt> is an RSF integer-type file with the trace headers (32<br />
headers with 71 traces each):<br />
<pre><br />
bash$ sfin tfile.rsf<br />
tfile.rsf:<br />
in="/tmp/tfile.rsf@"<br />
esize=4 type=int form=native<br />
n1=71 d1=? o1=?<br />
n2=32 d2=? o2=?<br />
2272 elements 9088 bytes<br />
</pre><br />
The contents of trace headers can be quickly examined with the <br />
<tt>sfheaderattr</tt> program.<br />
The <tt>file.asc</tt> is the ASCII header file for the whole record.<br />
<pre><br />
bash$ head -c 242 file.asc<br />
C This tape was made at the<br />
C <br />
C Center for Wave Phenomena <br />
</pre><br />
The <tt>file.bin</tt> is the binary header file.<br />
<br />
To convert files back from RSF to SEG-Y or SU, use the <tt>sfsegywrite</tt><br />
program and reverse the input and output:<br />
<pre><br />
bash$ sfsuwrite > plane.su tfile=tfile.rsf endian=0 < plane.rsf<br />
</pre><br />
or<br />
<pre><br />
bash$ sfsegywrite > plane.segy tfile=tfile.rsf \<br />
hfile=file.asc bfile=file.bin < plane.rsf<br />
</pre><br />
<br />
If <tt>hfile=</tt> and <tt>bfile=</tt> are not supplied to <tt>sfsegywrite</tt>, the corresponding headers will be generated on the fly. The trace header file can be generated with <tt>sfsegyheader</tt>. Here is an example:<br />
<pre><br />
bash$ sfheadermath < plane.rsf output=N+1 | sfdd type=int > tracl.rsf<br />
bash$ sfsegyheader < plane.rsf tracl=tracl.rsf > tfile.rsf<br />
bash$ sfsegywrite < plane.rsf tfile=tfile.rsf > plane.segy<br />
</pre><br />
<br />
====Unusual trace header keys====<br />
Sometimes, SEG-Y files deviate from the standard by creating additional<br />
trace header keys. If, for example, you find out that the SEG-Y file<br />
contains an additional trace header key stored in bytes 225-226, you<br />
can either remap one of the standard two-byte keys<br />
<pre><br />
bash&#36; sfsegyread < file.segy tfile=tfile.rsf gut=224 > file.rsf<br />
</pre><br />
or create a new key<br />
<pre><br />
bash&#36; sfsegyread < file.segy tfile=tfile.rsf \<br />
key1=mykey key1_len=2 mykey=224 > file.rsf<br />
</pre><br />
Any number of additional keys can be created this way.<br />
<br />
===Reading and writing ASCII files===<br />
Reading and writing ASCII files can be accomplished with the <tt>sfdd</tt><br />
program. For example, let us take an ASCII file with numbers<br />
<pre><br />
bash$ cat file.asc<br />
1.0 1.5 3.0<br />
4.8 9.1 7.3<br />
</pre><br />
Converting it to RSF is as simple as<br />
<pre><br />
bash$ echo in=file.asc n1=3 n2=2 data_format=ascii_float > file.rsf<br />
bash$ sfin file.rsf<br />
file.rsf:<br />
in="file.asc"<br />
esize=0 type=float form=ascii<br />
n1=3 d1=? o1=?<br />
n2=2 d2=? o2=?<br />
6 elements<br />
</pre><br />
For more efficient input/output operations, it might be advantageous to<br />
convert the data type to native binary, as follows:<br />
<pre><br />
bash$ echo in=file.asc n1=3 n2=2 data_format=ascii_float | \<br />
sfdd form=native > file.rsf<br />
bash$ sfin file.rsf<br />
file.rsf:<br />
in="/tmp/file.rsf@"<br />
esize=4 type=float form=native<br />
n1=3 d1=? o1=?<br />
n2=2 d2=? o2=?<br />
6 elements 24 bytes<br />
</pre><br />
Converting from RSF to ASCII is equally simple:<br />
<pre><br />
bash$ sfdd form=ascii --out=file.asc < file.rsf > /dev/null<br />
bash$ cat file.asc<br />
1 1.5 3 4.8 9.1 7.3<br />
</pre><br />
You can use the <tt>line=</tt> and <tt>format=</tt> parameters in<br />
<tt>sfdd</tt> to control the ASCII formatting:<br />
<pre><br />
bash$ sfdd form=ascii --out=file.asc \<br />
line=3 format="%3.1f " < file.rsf > /dev/null<br />
bash$ cat file.asc<br />
1.0 1.5 3.0<br />
4.8 9.1 7.3<br />
</pre><br />
An alternative is to use <tt>sfdisfil</tt>.<br />
<pre><br />
bash$ sfdisfil > file.asc col=3 format="%3.1f " number=n < file.rsf<br />
bash$ cat file.asc<br />
1.0 1.5 3.0<br />
4.8 9.1 7.3<br />
</pre><br />
<br />
===Reading and writing CSV files===<br />
CSV (Comma-separated values) is a particular example of an ASCII<br />
format, where values on different rows are separated by commas or<br />
other symbols. To convert from CSV to RSF, you can use the<br />
<tt>sfcsv2rsf</tt> utility.<br />
For example, let us take an ASCII file with numbers separated by commas<br />
<pre><br />
bash&#36; cat file.csv<br />
1.0,1.5,3.0<br />
4.8,9.1,7.3<br />
</pre><br />
Converting it to RSF:<br />
<pre><br />
bash&#36; sfcsv2rsf < file.csv > file.rsf<br />
bash&#36; sfin file.rsf<br />
file.rsf:<br />
in="/tmp/file.rsf@"<br />
esize=4 type=float form=native <br />
n1=3 d1=1 o1=0 label1="unknown" unit1="unknown" <br />
n2=2 d2=1 o2=0 label2="unknown" unit2="unknown" <br />
6 elements 24 bytes<br />
</pre><br />
To convert from RSF to CSV, we can use formatting parameters in <tt>sfdd</tt>:<br />
<pre><br />
bash&#36; sfdd form=ascii --out=file.csv \<br />
line=3 strip=1 format="%3.1f," < file.rsf >/dev/null<br />
bash&#36; cat file.csv<br />
1.0,1.5,3.0<br />
4.8,9.1,7.3<br />
</pre><br />
Some CSV files contain headers with definitions for different columns.<br />
<pre><br />
bash&#36; cat file.csv<br />
height,width,weight<br />
1.0,1.5,3.0<br />
4.8,9.1,7.3<br />
</pre><br />
To read a file like that, use <tt>header=</tt> parameter in <tt>sfcsv2rsf</tt>, as follows:<br />
<pre><br />
bash&#36; sfcsv2rsf < file.csv header=y > file.rsf<br />
</pre><br />
After that, different columns can be accessed by keywords.<br />
<pre><br />
bash&#36; < file.rsf sfheaderattr segy=n<br />
3 headers, 2 traces<br />
*******************************************************************************<br />
key min max mean<br />
-------------------------------------------------------------------------------<br />
height 0 1 @ 0 4.8 @ 1 2.9<br />
width 1 1.5 @ 0 9.1 @ 1 5.3<br />
weight 2 3 @ 0 7.3 @ 1 5.15<br />
*******************************************************************************<br />
</pre><br />
<br />
===Reading LAS files===<br />
LAS (Log ASCII Standard) is a text format used for storing<br />
well-logging data (Heslop et al., 1999<ref>Heslop, K., J. Karst, S. Prensky, D. Schmitt, et al., 1999, Log ASCII standard LAS version 3.0: The Log Analyst, 40.</ref>). LAS files can be converted to the RSF format using<br />
<tt>sflas2rsf</tt> utility.<br />
Let us try an example file from one of the SEG tutorials:<br />
<pre><br />
bash&#36; tutorials=https://raw.githubusercontent.com/seg/tutorials-2014/master<br />
bash&#36; wget &#36;tutorials/1406_Make_a_synthetic/L-30.las<br />
</pre><br />
Converting to RSF, we can detect 15 different logs:<br />
<pre><br />
bash&#36; sflas2rsf L-30.las L-30.rsf<br />
(base) sergey@DESKTOP-80QRDA0:~/all/fomels/nnint&#36; sfin L-30.rsf<br />
L-30.rsf:<br />
in="/home/sergey/RSFROOT/data/L-30.rsf@"<br />
esize=4 type=float form=native<br />
n1=15 d1=? o1=?<br />
n2=25621 d2=0.5 o2=1140<br />
384315 elements 1537260 bytes<br />
</pre><br />
Individual logs are accessible by their keys and can be used in programs such as <tt>sfheadermath</tt>.<br />
<pre><br />
bash&#36; < L-30.rsf sfheaderattr segy=n desc=y<br />
15 headers, 25621 traces<br />
*******************************************************************************<br />
key min max mean<br />
-------------------------------------------------------------------------------<br />
DEPTH 0 1140 @ 0 13950 @ 25620 7545<br />
[Depth]<br />
CALD 1 -999 @ 0 19.811 @ 3909 -140.356<br />
[Caliper Caliper - Density]<br />
CALS 2 -999 @ 0 14.84 @ 23096 7.43849<br />
[Caliper Caliper - Sonic]<br />
DEPT 3 1140 @ 0 13950 @ 25620 7545<br />
[Depth]<br />
DRHO 4 -999 @ 0 0.254 @ 23667 -149.67<br />
[Drho Delta Rho]<br />
DT 5 -999 @ 0 199.263 @ 1462 90.0167<br />
[Sonic Delta-T]<br />
GRD 6 -999 @ 0 178.416 @ 21788 -100.952<br />
[GammaRay Gamma Ray - Density]<br />
GRS 7 -999 @ 0 140.148 @ 23376 53.8002<br />
[GammaRay Gamma Ray - Sonic]<br />
ILD 8 -999 @ 0 2022.95 @ 20 34.5917<br />
[DeepRes Deep Induction Standard Processed Resistivity]<br />
ILM 9 -999 @ 0 2196.26 @ 20661 40.5595<br />
[MedRes Medium Induction Standard Processed Resistivity]<br />
LL8 10 -999 @ 0 2097.76 @ 20213 35.6343<br />
[ShalRes Latero-Log 8]<br />
NPHILS 11 -999 @ 0 0.45 @ 23039 -776.522<br />
[Neutron Neutron Porosity - Ls Mtx]<br />
NPHISS 12 -999 @ 0 0.615 @ 5215 -373.244<br />
[Neutron Neutron Porosity - Ss Mtx]<br />
RHOB 13 -999 @ 0 2.811 @ 23941 -147.773<br />
[Density Bulk Density]<br />
SP 14 -999 @ 0 -19.065 @ 20570 -105.029<br />
[SP Spontaneous Potential]<br />
*******************************************************************************<br />
bash&#36; < L-30.rsf sfheadermath output=RHOB segy=n > RHOB.rsf<br />
bash&#36; < RHOB.rsf sfwindow min2=4000 max2=13000 | sfgraph title=Density<br />
</pre><br />
<br />
[[Image:rhob.png|frame|center|Density log.]]<br />
<br />
==Other documentation==<br />
This note should give you a general understanding of the RSF file format. See the [[RSF Comprehensive Description]] if you are interested in minutia. Other relevant documentation is: <br />
<br />
*[[Why Madagascar]]<br />
*[[Installation|Installation instructions]]<br />
*[http://reproducibility.org/RSF/ Madagascar self-documentation] <br />
*[[Guide to madagascar programs]]<br />
*[[Guide to madagascar API|Guide to the Madagascar programming interface]]<br />
*[[Guide to programming with madagascar]]<br />
*[[Revisiting SEP tour with Madagascar and SCons]]<br />
*[[Reproducible computational experiments using SCons]]<br />
<br />
==About this document==<br />
This page was created from the LaTeX source in [http://rsf.svn.sourceforge.net/viewvc/rsf/trunk/book/rsf/rsf/format.tex?view=markup book/rsf/rsf/format.tex] using [[latex2wiki]].<br />
<br />
==References==<br />
<references/></div>Sergeyhttps://www.reproducibility.org/wiki2020/index.php?title=Guide_to_RSF_file_format&diff=3925Guide to RSF file format2021-03-08T23:06:13Z<p>Sergey: /* Reading and writing CSV files */</p>
<hr />
<div><center><font size="-1">''This page was created from the LaTeX source in [https://github.com/ahay/src/blob/master/book/rsf/rsf/format.tex book/rsf/rsf/format.tex] using [[latex2wiki]]''</font></center><br />
<br />
<br />
[[Image:Fotolia_9592362_XS.jpg|right|]]<br />
<br />
==Principles==<br />
The main design principle behind the RSF data format is [http://en.wikipedia.org/wiki/KISS_principle KISS] ("Keep It<br />
Short and Simple"). The RSF format is borrowed from the SEPlib data format<br />
originally designed at the Stanford Exploration Project<br />
(Claerbout, 1991<ref>Claerbout, J. F., 1991, Introduction to Seplib and SEP utility software, ''in'' SEP-70, 413--436. Stanford Exploration Project.</ref>). The format is made as simple as possible for<br />
maximum convenience, transparency and flexibility.<br />
According to the Unix tradition, common file formats should be in a readable<br />
textual form so that they can be easily examined and processed with universal<br />
tools. Raymond (2004<ref>Raymond, E. S., 2004, The art of UNIX programming: Addison-Wesley.</ref>) writes:<br />
<blockquote><br />
To design a perfect anti-Unix, make all file formats binary and opaque, and<br />
require heavyweight tools to read and edit them.<br />
</blockquote><br />
<blockquote><br />
If you feel an urge to design a complex binary file format, or a complex<br />
binary application protocol, it is generally wise to lie down until the<br />
feeling passes.<br />
</blockquote><br />
Storing large-scale datasets in a text format may not be economical. RSF<br />
chooses the next best thing: it allows data values to be stored in a binary<br />
format but puts all data attributes in text files that can be read by humans<br />
and processed with universal text-processing utilities.<br />
===Example===<br />
Let us first create some synthetic RSF data.<br />
<pre><br />
bash$ sfmath n1=1000 output='sin(0.5*x1)' > sin.rsf<br />
</pre><br />
Open and read the file <tt>sin.rsf</tt>.<br />
<pre><br />
bash$ cat sin.rsf<br />
sfmath rsf/rsf/rsftour: fomels@egl Sun Jul 31 07:18:48 2005<br />
<br />
o1=0<br />
data_format="native_float"<br />
esize=4<br />
in="/tmp/sin.rsf@"<br />
x1=0<br />
d1=1<br />
n1=1000<br />
</pre><br />
The file contains nine lines with simple readable text. The first line<br />
shows the name of the program, the working directory, the user and<br />
computer that created the file and the time it was created (that<br />
information is recorded for accounting purposes). Other lines contain<br />
parameter-value pairs separated by the "=" sign. The "in"<br />
parameter points to the location of the binary data. Before we discuss<br />
the meaning of parameters in more detail, let us plot the data.<br />
<pre><br />
bash$ < sin.rsf sfwiggle title='One Trace' | sfpen<br />
</pre><br />
On your screen, you should see a plot similar to the figure below.<br />
[[Image:sin1.png|frame|center|An example sinusoid plot.]]<br />
Suppose you want to reformat the data so that instead of one trace of a<br />
thousand samples, it contains twenty traces with fifty samples each. Try<br />
running<br />
<pre><br />
bash$ < sin.rsf sed 's/n1=1000/n1=50 n2=20/' > sin10.rsf <br />
bash$ < sin10.rsf sfwiggle title=Traces | sfpen<br />
</pre><br />
or (using pipes)<br />
<pre><br />
bash$ < sin.rsf sed 's/n1=1000/n1=50 n2=20/' | sfwiggle title=Traces | sfpen<br />
</pre><br />
On your screen, you should see a plot similar to the figure below:<br />
[[Image:sin2.png|frame|center|An example sinusoid plot, with data reformatted to twenty traces.]]<br />
What happened? We used <tt>sed</tt>, a standard Unix line editing utility to<br />
change the parameters describing the data dimensions. Because of the<br />
simplicity of this operation, there is no need to create specialized data<br />
formatting tools or to make the <tt>sfwiggle</tt> program accept additional<br />
formatting parameters. Other general-purpose Unix tools that can be applied on<br />
RSF files include <tt>cat</tt>, <tt>echo</tt>, <tt>grep</tt>, etc. <br />
An alternative way to obtain the previous result is to run<br />
<pre><br />
bash$ ( cat sin.rsf; echo n1=50 n2=20 ) > sin10.rsf <br />
bash$ < sin10.rsf sfwiggle title=Traces | sfpen<br />
</pre><br />
In this case, the <tt>cat</tt> utility simply copies the contents of the<br />
previous file, and the <tt>echo</tt> utility appends new line "<tt>n1=50<br />
n2=20</tt>". A new value of the <tt>n1</tt> parameter overwrites the old value<br />
of <tt>n1=1000</tt>, and we achieve the same result as before.<br />
Of course, one could also edit the file by hand with one of the general<br />
purpose text editors. For recording the history of data processing, it is<br />
usually preferable to be able to process files with non-interactive tools.<br />
<br />
==Header and Data files==<br />
A simple way to check the layout of an RSF file is with the <tt>sfin</tt><br />
program.<br />
<pre><br />
bash$ sfin sin10.rsf<br />
sin10.rsf:<br />
in="/tmp/sin.rsf@"<br />
esize=4 type=float form=native<br />
n1=50 d1=1 o1=0<br />
n2=20 d2=? o2=?<br />
1000 elements 4000 bytes<br />
</pre><br />
The program reports the following information: the location of the data file<br />
(<tt>/tmp/sin.rsf\@</tt>), the element size (4 bytes), the element<br />
type (floating point), the element form (native), the hypercube dimensions<br />
(<math>50 \times 20</math>), axis scaling (1 and unspecified), and axis origin (0 and<br />
unspecified). It also checks the total number of elements and bytes in the<br />
data file.<br />
Let us examine this information in detail. First, we can verify that the data<br />
file exists and contains the specified number of bytes:<br />
<pre><br />
bash$ ls -l /tmp/sin.rsf@<br />
-rw-r--r-- 1 sergey users 4000 2004-10-04 00:35 /tmp/sin.rsf@<br />
</pre><br />
4000 bytes in this file are required to store <math>50 \times 20</math> floating-point<br />
4-byte numbers in a binary form. Thus, the data file contains nothing but the<br />
raw data in a contiguous binary form.<br />
===Datapath===<br />
How did the RSF program (<tt>sfmath</tt>) decide where to put the data file?<br />
In the order of priority, the rules for selecting the data file name and the<br />
data file directory are as follows:<br />
<br />
#Check <tt>--out=</tt> parameter on the command line. The parameter specifies the output data file location explicitly. <br />
#Specify the path and the file name separately. <br />
#*The rules for the path selection are: <br />
#*#Check <tt>datapath=</tt> parameter on the command line. The parameter specifies a string to prepend to the file name. The string may contain the file directory. <br />
#*#Check <tt>DATAPATH</tt> environmental variable. It has the same meaning as the parameter specified with <tt>datapath=</tt>. <br />
#*#Check for <tt>.datapath</tt> file in the current directory. The file may contain a line <pre> datapath=/path/to_file/ </pre> or <pre> machine_name datapath=/path/to_file/ </pre> if you indent to use different paths on different platforms. <br />
#*#Check for <tt>.datapath</tt> file in the user home directory. <br />
#*#Put the data file in the current directory (similar to <tt>datapath=./</tt>). <br />
#*: <br />
#*The rules for the filename selection are: <br />
#*#If the output RSF file is in the current directory, the name of the data file is made by appending \@. <br />
#*#If the output file is not in the current directory or if it is created temporarily by a program, the name is made by appending random characters to the name of the program and selected to be unique. <br />
<br />
Examples:<br />
<br />
<pre><br />
bash$ sfspike n1=10 --out=test1 > spike.rsf <br />
bash$ grep in spike.rsf <br />
in="test1" <br />
</pre> <br />
<br />
<pre><br />
bash$ sfspike n1=10 datapath=/tmp/ > spike.rsf <br />
bash$ grep in spike.rsf <br />
in="/tmp/spike.rsf@" <br />
</pre> <br />
<br />
<pre><br />
bash$ DATAPATH=/tmp/ sfspike n1=10 > spike.rsf <br />
bash$ grep in spike.rsf <br />
in="/tmp/spike.rsf@" <br />
</pre> <br />
<br />
<pre><br />
bash$ sfspike n1=10 datapath=/tmp/ > /tmp/spike.rsf <br />
bash$ grep in /tmp/spike.rsf <br />
in="/tmp/sfspikejcARVf" <br />
</pre> <br />
<br />
====Packing header and data together====<br />
While the header and data files are separated by default, it is also possible<br />
to pack them together into one file. To do that, specify the program's<br />
"<tt>--out</tt>" parameter as <tt>--out=stdout</tt>. Example:<br />
<pre><br />
bash$ sfspike n1=10 --out=stdout > spike.rsf<br />
bash$ grep in spike.rsf<br />
Binary file spike.rsf matches<br />
bash$ sfin spike.rsf<br />
spike.rsf:<br />
in="stdin"<br />
esize=4 type=float form=native<br />
n1=10 d1=0.004 o1=0 label1="Time" unit1="s"<br />
10 elements 40 bytes<br />
bash$ ls -l spike.rsf<br />
-rw-r--r-- 1 sergey users 196 2004-11-10 21:39 spike.rsf<br />
</pre><br />
If you examine the contents of <tt>spike.rsf</tt>, you will find that it<br />
starts with the text header information, followed by special<br />
symbols, followed by binary data. <br />
Packing headers and data together may not be a good idea for data processing<br />
but it works well for storing data: it is easier to move the packed file<br />
around than to move two different files (header and binary) together while<br />
remembering to preserve their connection. Packing header and data together is<br />
also the current mechanism used to push RSF files through Unix pipes.<br />
<br />
===Type===<br />
The data stored with RSF can have different types: character, unsigned<br />
character, integer, floating point, or complex. By default, single precision<br />
is used for numbers (<tt>int</tt> and <tt>float</tt> data types in the C<br />
programming language) but double precision and other<br />
integer types (<tt>short</tt> and <tt>long</tt>) are also<br />
supported. The number of bytes required for represent these<br />
numbers may depend on the platform.<br />
===Form===<br />
The data stored with RSF can also be in a different form: ASCII, native<br />
binary, and XDR binary. Native binary is often used by default. It is the<br />
binary format employed by the machine that is running the application. On<br />
Linux-running PC, the native binary format will typically correspond to the<br />
so-called little-endian byte ordering. On some other platform, it might be<br />
big-endian ordering. XDR is a binary format designed by Sun for exchanging<br />
files over network. It typically corresponds to big-endian byte ordering. It<br />
is more efficient to process RSF files in the native binary format but, if you<br />
intend to access data from different platforms, it might be a good idea to<br />
store the corresponding file in an XDR format. RSF also allows for an ASCII<br />
(plain text) form of data files. <br />
Conversion between different types and forms is accomplished with<br />
<tt>sfdd</tt> program. Here are some examples. First, let us create synthetic<br />
data.<br />
<pre><br />
bash$ sfmath n1=10 output='10*sin(0.5*x1)' > sin.rsf<br />
bash$ sfin sin.rsf<br />
sin.rsf:<br />
in="/tmp/sin.rsf@"<br />
esize=4 type=float form=native<br />
n1=10 d1=1 o1=0<br />
10 elements 40 bytes<br />
bash$ < sin.rsf sfdisfil<br />
0: 0 4.794 8.415 9.975 9.093<br />
5: 5.985 1.411 -3.508 -7.568 -9.775<br />
</pre><br />
Converting the data to the integer type:<br />
<pre><br />
bash$ < sin.rsf sfdd type=int > isin.rsf<br />
bash$ sfin isin.rsf<br />
isin.rsf:<br />
in="/tmp/isin.rsf@"<br />
esize=4 type=int form=native<br />
n1=10 d1=1 o1=0<br />
10 elements 40 bytes<br />
bash$ < isin.rsf sfdisfil<br />
0: 0 4 8 9 9 5 1 -3 -7 -9<br />
</pre><br />
Converting the data to the ASCII form:<br />
<pre><br />
bash$ < sin.rsf sfdd form=ascii > asin.rsf<br />
bash$ < asin.rsf sfdisfil<br />
0: 0 4.794 8.415 9.975 9.093<br />
5: 5.985 1.411 -3.508 -7.568 -9.775<br />
bash$ sfin asin.rsf<br />
asin.rsf:<br />
in="/tmp/asin.rsf@"<br />
esize=0 type=float form=ascii<br />
n1=10 d1=1 o1=0<br />
10 elements<br />
bash$ cat /tmp/asin.rsf@<br />
0 4.79426 8.41471 9.97495 9.09297 5.98472 1.4112 -3.50783<br />
-7.56803 -9.7753<br />
</pre><br />
<br />
===Hypercube===<br />
While RSF stores binary data in a contiguous 1-D array, the conceptual<br />
data model is a multidimensional hypercube. By convention, the<br />
dimensions of the cube are defined with parameters <tt>n1</tt>,<br />
<tt>n2</tt>, <tt>n3</tt>, etc. The fastest axis is <tt>n1</tt>.<br />
Additionally, the grid sampling can be given by parameters<br />
<tt>d1</tt>, <tt>d2</tt>, <tt>d3</tt>, etc. The axes origins are given<br />
by parameters <tt>o1</tt>, <tt>o2</tt>, <tt>o3</tt>, etc. Optionally,<br />
you can also supply the axis label strings: <tt>label1</tt>,<br />
<tt>label2</tt>, <tt>label3</tt>, etc., and axis units strings:<br />
<tt>unit1</tt>, <tt>unit2</tt>, <tt>unit3</tt>, etc. <br />
==Compatibility with other file formats==<br />
It is possible to exchange RSF-formatted data with several other popular data formats.<br />
===Compatibility with SEPlib===<br />
RSF is mostly compatible with its predecessor, the SEPlib file format.<br />
However, there are several important differences:<br />
<br />
#SEPlib programs typically use the element size (<tt>esize=</tt> parameter) to distinguish between different data types: <tt>esize=4</tt> corresponds to floating point data, while <tt>esize=8</tt> corresponds to complex data. The RSF type handling mechanism is different: data types are determined from the value of the <tt>data_format</tt> parameter. Madagascar computational programs typically output files with <tt>data_format="native_float"</tt> or <tt>native_complex</tt>.<br />
#The default data form in SEPlib programs is typically XDR and not native as it is in RSF. Thus, to make a dataset created with SEPlib readable by Madagascar programs, you would typically need to add to the history file <tt>data_format="xdr_float"</tt> or <tt>data_format="xdr_complex"</tt> . <ref group="note">For SEPlib 6.5.3 and older: Note that xdr_complex is not a valid SEPlib value, so for datasets of complex numbers encoded as pairs of floats, a dataset cannot be at the same time valid in both SEPlib and Madagascar. A valid SEPlib dataset will have esize=8 and data_format="xdr_float", but sfin will show it as having "200% of expected" data. Adding data_format="xdr_complex" to such a dataset will make sfin work as expected, but SEPlib's In or In3d will give a segmentation fault because of an unknown data type. To patch SEPlib to accept <tt>native_complex</tt> and <tt>xdr_complex</tt> data, the following changes must be made:<br />
* In <tt>$SEPSRC/seplib_base/lib/corelibs/sep/strformats.c</tt>:<br />
** Add "xdr_complex" and "native_complex" to the str_fmt_names structure<br />
** Set FMT_LENGTH to 15<br />
* In <tt>$SEPSRC/seplib_base/lib/corelibs/include/strformats.h</tt>:<br />
** Add preprocessor directives to define FMT_XDR_COMPLEX as 8 and FMT_NATIVE_COMPLEX as 9<br />
** Set NUM_FMT to 10<br />
</ref><br />
#It is possible to pipe the output of Madagascar programs to SEPlib: <pre>bash$ sfspike n1=1 | Attr want=min</pre> (output should be: <tt>minimum value = 1 at 1</tt>). However, piping the output of SEPlib programs to RSF (or, for that matter, any other non-SEPlib programs) will result in an unterminated process. For example, the command <pre> bash$ Spike n1=1 | sfattr want=min </pre> will hang. This is because SEPlib uses sockets for piping and expects a socket connection from the receiving program, while Madagascar passes data through regular Unix pipes. <br />
#SEP3D is an extension of SEPlib for operating with irregularly sampled data (Biondi et al., 1996<ref>Biondi, B., R. Clapp, and S. Crawley, 1996, SEPlib90: SEPlib for 3-D prestack data, ''in'' SEP-92, 343--364. Stanford Exploration Project.</ref>). There is no equivalent of it in RSF for the reasons explained in the beginning of this guide. Operations with irregular datasets are supported through the use of auxiliary input files that represent the geometry information.<br />
<br />
;Notes<br />
<references group="note" /><br />
<br />
===Reading and writing SEG-Y and SU files===<br />
The SEG-Y format is based on the proposal of Barry et al. (1975<ref>[http://www.seg.org/SEGportalWEBproject/prod/SEG-Publications/Pub-Technical-Standards/Documents/seg_y_rev0.pdf Barry, K. M., D. A. Cavers, and C. W. Kneale, 1975, Report on recommended standards for digital tape formats: Geophysics, '''40''', 344--352]</ref>).<br />
It was revised in 2002<ref>See http://www.seg.org/SEGportalWEBproject/prod/SEG-Publications/Pub-Technical-Standards/Documents/seg_y_rev1.pdf</ref>. The<br />
SU format is a modification of SEG-Y used in Seismic Unix<br />
(Stockwell, 1997<ref>Stockwell, J. W., 1997, Free software in education: A case study of CWP/SU: Seismic Unix: The Leading Edge, '''16''', 1045--1049.</ref>).<br />
To convert files from SEG-Y or SU format to RSF, use the <tt>sfsegyread</tt><br />
program. Let us first manufacture an example file using SU utilities<br />
(Stockwell, 1999<ref>-------- 1999, The CWP/SU: Seismic Un*x package: Computers and Geosciences, '''25''', 415--419.</ref>):<br />
<pre><br />
bash$ suplane > plane.su<br />
bash$ segyhdrs < plane.su | segywrite tape=plane.segy<br />
</pre><br />
To convert it to RSF, use either<br />
<pre><br />
bash$ sfsuread < plane.su tfile=tfile.rsf endian=0 > plane.rsf<br />
</pre><br />
or<br />
<pre><br />
bash$ sfsegyread < plane.segy tfile=tfile.rsf \<br />
hfile=file.asc bfile=file.bin > plane.rsf<br />
</pre><br />
The endian flag is needed if the SU file originated from a little-endian<br />
machine such as Linux PC.<br />
Several files are generated. The standard output contains an RSF file with the<br />
data (32 traces with 64 samples each):<br />
<pre><br />
bash$ sfin plane.rsf<br />
plane.rsf:<br />
in="/tmp/plane.rsf@"<br />
esize=4 type=float form=native<br />
n1=64 d1=0.004 o1=0<br />
n2=32 d2=? o2=?<br />
2048 elements 8192 bytes<br />
</pre><br />
The contents of this file are displayed in the figure.<br />
[[Image:plane.png|frame|center|The output of suplane, converted to RSF and<br />
displayed with <tt>sfwiggle</tt>.]]<br />
The <tt>tfile</tt> is an RSF integer-type file with the trace headers (32<br />
headers with 71 traces each):<br />
<pre><br />
bash$ sfin tfile.rsf<br />
tfile.rsf:<br />
in="/tmp/tfile.rsf@"<br />
esize=4 type=int form=native<br />
n1=71 d1=? o1=?<br />
n2=32 d2=? o2=?<br />
2272 elements 9088 bytes<br />
</pre><br />
The contents of trace headers can be quickly examined with the <br />
<tt>sfheaderattr</tt> program.<br />
The <tt>file.asc</tt> is the ASCII header file for the whole record.<br />
<pre><br />
bash$ head -c 242 file.asc<br />
C This tape was made at the<br />
C <br />
C Center for Wave Phenomena <br />
</pre><br />
The <tt>file.bin</tt> is the binary header file.<br />
<br />
To convert files back from RSF to SEG-Y or SU, use the <tt>sfsegywrite</tt><br />
program and reverse the input and output:<br />
<pre><br />
bash$ sfsuwrite > plane.su tfile=tfile.rsf endian=0 < plane.rsf<br />
</pre><br />
or<br />
<pre><br />
bash$ sfsegywrite > plane.segy tfile=tfile.rsf \<br />
hfile=file.asc bfile=file.bin < plane.rsf<br />
</pre><br />
<br />
If <tt>hfile=</tt> and <tt>bfile=</tt> are not supplied to <tt>sfsegywrite</tt>, the corresponding headers will be generated on the fly. The trace header file can be generated with <tt>sfsegyheader</tt>. Here is an example:<br />
<pre><br />
bash$ sfheadermath < plane.rsf output=N+1 | sfdd type=int > tracl.rsf<br />
bash$ sfsegyheader < plane.rsf tracl=tracl.rsf > tfile.rsf<br />
bash$ sfsegywrite < plane.rsf tfile=tfile.rsf > plane.segy<br />
</pre><br />
<br />
====Unusual trace header keys====<br />
Sometimes, SEG-Y files deviate from the standard by creating additional<br />
trace header keys. If, for example, you find out that the SEG-Y file<br />
contains an additional trace header key stored in bytes 225-226, you<br />
can either remap one of the standard two-byte keys<br />
<pre><br />
bash&#36; sfsegyread < file.segy tfile=tfile.rsf gut=224 > file.rsf<br />
</pre><br />
or create a new key<br />
<pre><br />
bash&#36; sfsegyread < file.segy tfile=tfile.rsf \<br />
key1=mykey key1_len=2 mykey=224 > file.rsf<br />
</pre><br />
Any number of additional keys can be created this way.<br />
<br />
===Reading and writing ASCII files===<br />
Reading and writing ASCII files can be accomplished with the <tt>sfdd</tt><br />
program. For example, let us take an ASCII file with numbers<br />
<pre><br />
bash$ cat file.asc<br />
1.0 1.5 3.0<br />
4.8 9.1 7.3<br />
</pre><br />
Converting it to RSF is as simple as<br />
<pre><br />
bash$ echo in=file.asc n1=3 n2=2 data_format=ascii_float > file.rsf<br />
bash$ sfin file.rsf<br />
file.rsf:<br />
in="file.asc"<br />
esize=0 type=float form=ascii<br />
n1=3 d1=? o1=?<br />
n2=2 d2=? o2=?<br />
6 elements<br />
</pre><br />
For more efficient input/output operations, it might be advantageous to<br />
convert the data type to native binary, as follows:<br />
<pre><br />
bash$ echo in=file.asc n1=3 n2=2 data_format=ascii_float | \<br />
sfdd form=native > file.rsf<br />
bash$ sfin file.rsf<br />
file.rsf:<br />
in="/tmp/file.rsf@"<br />
esize=4 type=float form=native<br />
n1=3 d1=? o1=?<br />
n2=2 d2=? o2=?<br />
6 elements 24 bytes<br />
</pre><br />
Converting from RSF to ASCII is equally simple:<br />
<pre><br />
bash$ sfdd form=ascii --out=file.asc < file.rsf > /dev/null<br />
bash$ cat file.asc<br />
1 1.5 3 4.8 9.1 7.3<br />
</pre><br />
You can use the <tt>line=</tt> and <tt>format=</tt> parameters in<br />
<tt>sfdd</tt> to control the ASCII formatting:<br />
<pre><br />
bash$ sfdd form=ascii --out=file.asc \<br />
line=3 format="%3.1f " < file.rsf > /dev/null<br />
bash$ cat file.asc<br />
1.0 1.5 3.0<br />
4.8 9.1 7.3<br />
</pre><br />
An alternative is to use <tt>sfdisfil</tt>.<br />
<pre><br />
bash$ sfdisfil > file.asc col=3 format="%3.1f " number=n < file.rsf<br />
bash$ cat file.asc<br />
1.0 1.5 3.0<br />
4.8 9.1 7.3<br />
</pre><br />
<br />
===Reading and writing CSV files===<br />
CSV (Comma-separated values) is a particular example of an ASCII<br />
format, where values on different rows are separated by commas or<br />
other symbols. To convert from CSV to RSF, you can use the<br />
<tt>sfcsv2rsf</tt> utility.<br />
For example, let us take an ASCII file with numbers separated by commas<br />
<pre><br />
bash&#36; cat file.csv<br />
1.0,1.5,3.0<br />
4.8,9.1,7.3<br />
</pre><br />
Converting it to RSF:<br />
<pre><br />
bash&#36; sfcsv2rsf < file.csv > file.rsf<br />
bash&#36; sfin file.rsf<br />
file.rsf:<br />
in="/tmp/file.rsf@"<br />
esize=4 type=float form=native <br />
n1=3 d1=1 o1=0 label1="unknown" unit1="unknown" <br />
n2=2 d2=1 o2=0 label2="unknown" unit2="unknown" <br />
6 elements 24 bytes<br />
</pre><br />
To convert from RSF to CSV, we can use formatting parameters in <tt>sfdd</tt>:<br />
<pre><br />
bash&#36; sfdd form=ascii --out=file.csv \<br />
line=3 strip=1 format="%3.1f," < file.rsf >/dev/null<br />
bash&#36; cat file.csv<br />
1.0,1.5,3.0<br />
4.8,9.1,7.3<br />
</pre><br />
Some CSV files contain headers with definitions for different columns.<br />
<pre><br />
bash&#36; cat file.csv<br />
height,width,weight<br />
1.0,1.5,3.0<br />
4.8,9.1,7.3<br />
</pre><br />
To read a file like that, use <tt>header=</tt> parameter in <tt>sfcsv2rsf</tt>, as follows:<br />
<pre><br />
bash&#36; sfcsv2rsf < file.csv header=y > file.rsf<br />
</pre><br />
After that, different columns can be accessed by keywords.<br />
<pre><br />
bash&#36; < file.rsf sfheaderattr segy=n<br />
3 headers, 2 traces<br />
*******************************************************************************<br />
key min max mean<br />
-------------------------------------------------------------------------------<br />
height 0 1 @ 0 4.8 @ 1 2.9<br />
width 1 1.5 @ 0 9.1 @ 1 5.3<br />
weight 2 3 @ 0 7.3 @ 1 5.15<br />
*******************************************************************************<br />
</pre><br />
<br />
==Other documentation==<br />
This note should give you a general understanding of the RSF file format. See the [[RSF Comprehensive Description]] if you are interested in minutia. Other relevant documentation is: <br />
<br />
*[[Why Madagascar]]<br />
*[[Installation|Installation instructions]]<br />
*[http://reproducibility.org/RSF/ Madagascar self-documentation] <br />
*[[Guide to madagascar programs]]<br />
*[[Guide to madagascar API|Guide to the Madagascar programming interface]]<br />
*[[Guide to programming with madagascar]]<br />
*[[Revisiting SEP tour with Madagascar and SCons]]<br />
*[[Reproducible computational experiments using SCons]]<br />
<br />
==About this document==<br />
This page was created from the LaTeX source in [http://rsf.svn.sourceforge.net/viewvc/rsf/trunk/book/rsf/rsf/format.tex?view=markup book/rsf/rsf/format.tex] using [[latex2wiki]].<br />
<br />
==References==<br />
<references/></div>Sergey