diff --git a/.gitignore b/.gitignore index ea9f41d52583a3c48ff19df8a753c04991399d75..db550953950efb28282c1eed2f4e211420b22b09 100644 --- a/.gitignore +++ b/.gitignore @@ -33,7 +33,7 @@ frida external_ci gcc clang -build* +build mybuild qbuild podstyle.css diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml index 653d6088458231efb88df7ab11b9235fa40d2f15..cb2ec73b91445140ee159e1e954e7fd95dad84ca 100644 --- a/.gitlab-ci.yml +++ b/.gitlab-ci.yml @@ -9,8 +9,8 @@ mac: - mkdir pub/build - cd pub/build - cmake .. - - make - - ctest --output-on-failure + - make -j4 + - ctest --output-on-failure --timeout 120 # - cpack . artifacts: paths: diff --git a/wiki/build.md b/wiki/build.md new file mode 100644 index 0000000000000000000000000000000000000000..311db98f9e00e12a4157f961bec872b5f981f3c8 --- /dev/null +++ b/wiki/build.md @@ -0,0 +1,184 @@ +Building Frida +============== + +This page is part of the [Frida](start) online handbook. It describes +the standard procedure to download, compile, and install Frida2. + +See also: + +- [frida.ini](frida.ini): local set up +- [git](git): alternative download and build procedure, using the + development tree + +### Requirements + +##### Operating systems + +- **Linux.** Our standard environment is Debian. Please report if + there are difficulties with other distributions. + +```{=html} +<!-- --> +``` +- **Mac OS.** See [Frida under MacOS](Frida under MacOS). + +```{=html} +<!-- --> +``` +- **MS Windows.** See [Frida under Windows](Frida under Windows). + +##### Required packages + +To compile the sources, one needs the compilation tools + +- make +- g++ (GNU C++ compiler) +- bison +- flex + +and the libraries (-dev packages) + +- readline (at least version 4.3) +- history (may come with package libreadline-dev) +- gsl +- blas (required by gsl) +- fftw3 +- boost (modules shared_ptr, regex, date_time) + +At runtime, one external program is needed: + +- gnuplot + +In Debian/Ubuntu, the following command installs all required packages: + + sudo aptitude install \ + make \ + g++ \ + bison \ + flex \ + libreadline-dev \ + libgsl0-dev \ + libfftw3-dev \ + libboost-regex-dev \ + libboost-date-time-dev \ + gnuplot + +Other required packages are automatically installed through +dependencies. In particular, each development package (-dev, containing +the .h files) depends on a library package that contains the +corresponding shared library. + +The libboost regex and date-time development packages depend on +libboost-dev which contains, among others, the shared pointer module +(which is implemented through .h files only; no shared library). + +There may be some incompatibility between g++ and libboost versions. +Proven combinations are + +- libboost1.40 or 1.42 with g++-4.4; +- libboost1.34 with g++-4.3. + +Please report on other combinations that work or do not work. + +The transition from libreadline5 to libreadline6 may also cause +problems; please report details. + +### Download and Build + +Download most recent source packages from +<http://apps.jcns.fz-juelich.de/src/frida/>. + +- frida2.\<version\>.tgz contains Frida proper, +- frida2libs-\<version\>.tgz contains several auxiliary libraries. + +Unpack, configure, make, install the libraries: + + tar xzvf frida2libs-<version>.tgz + cd frida2libs + ./configure + make + sudo make install + sudo ldconfig # see note below + cd .. + +Note regarding *ldconfig*: This command somehow registers new libraries. +I am surprised that it is not automatically executed by *make install*. +But this seems to be distribution specific. Users report: + + sudo ldconfig # required in Ubuntu + sudo /sbin/ldconfig /usr/local/bin # required in Suse + +Unpack, configure, make, install the main program: + + tar xzvf frida2-<version>.tgz + cd frida2 + ./configure + make + sudo make install + cd .. + +Finally, create a directory for the graphics output: + + mkdir ~/gnew + +That\'s it. The command `frida` should start a frida session. + +### Test + +Try to run the following test session: + + $ frida + ? > fm # enter command fm = file make to create a little data file + Number of points per scan [1] ? 11 # enter 11 to overwrite the default 1 + Number of scans [1] ? # just press RETURN to accept the default value + Save as [grid3] ? # again, we just accept the default value + 0 > oy x^2 # our data file becomes a parabola + 1 > p # plot it + 1 > cc p0*t^p1 # create a fit curve: some power law + 2 > cf # fit p0,p1 to the data; should say "success" + 2 > a # add the fit to the plot + 2 > quit # terminate the frida session + +### frequent errors + +If libraries are not found, check LD_LIBRARY_PATH. Possibly, +/usr/local/lib must be added. \[related commands: ldd, ldconfig\]. + +If the global configure-make-install fails, goto the subdirectories +*kww*, *lmfit*, *readplus*, *yaml-cpp-at*, and build these libraries +separately, using again the standard *./configure; make; sudo make +install* sequence. Then build *frida2*. Some of the libraries contain +`demo` directories. In case of problems, go to these subdirectories and +try to run the test programs. + +If your configure routine fails because of the missing library readline, +please check the config.log for further details. If you find the +following error messages: + + /usr/users/iff_ns/brodeck/usr/local/lib/libreadline.so: undefined reference to `PC' + /usr/users/iff_ns/brodeck/usr/local/lib/libreadline.so: undefined reference to `tgetflag' + /usr/users/iff_ns/brodeck/usr/local/lib/libreadline.so: undefined reference to `tgetent' + /usr/users/iff_ns/brodeck/usr/local/lib/libreadline.so: undefined reference to `UP' + /usr/users/iff_ns/brodeck/usr/local/lib/libreadline.so: undefined reference to `tputs' + /usr/users/iff_ns/brodeck/usr/local/lib/libreadline.so: undefined reference to `tgoto' + /usr/users/iff_ns/brodeck/usr/local/lib/libreadline.so: undefined reference to `tgetnum' + /usr/users/iff_ns/brodeck/usr/local/lib/libreadline.so: undefined reference to `BC' + /usr/users/iff_ns/brodeck/usr/local/lib/libreadline.so: undefined reference to `tgetstr' + +you have to use the library ncurses during the compilation. To do so, +set the environemnt variable LDFLAGS=\"-lncurses\" and run configure +again (don\'t forget to run \'export LDFLAGS\'). + +### Problems + +If problems persist, please attach the output of + + uname -a + +and run + + make clean + ./configure | tee configure.log + make | tee make.log + +and attach configure.log and make.log. diff --git a/wiki/built-in_functions.md b/wiki/built-in_functions.md new file mode 100644 index 0000000000000000000000000000000000000000..e59d33cd357af9e39ebdb809d33a51a64d080b0d --- /dev/null +++ b/wiki/built-in_functions.md @@ -0,0 +1,63 @@ +Built-in functions +================== + +*Top:* [Frida handbook](/start)\ +*Previous:* [Expressions](Expressions)\ +*Next:* [Curves](Curves)\ +The following, outdated list gives an idea of the rich collection of +built-in functions that comes with Frida. An up-to-date list of built-in +functions can be obtained from the running program by typing **hf** at +the main command. + + > hf + Operators by precedence: + ... + Built-in functions: + ln(x): natural logarithm of x, or 0 if x<=0 + lg(x): decadic logarithm of x, or 0 if x<=0 + sqrt(x): square root of x, or 0 if x<0 + abs(x): absolute value of x + exp(x): exponential function of x + sin(x): sine of x (where x in radian) + cos(x): cosine of x (where x in radian) + tan(x): tangent of x (where x in radian) + cot(x): cotangent of x (where x in radian) + sind(x): sine of x (where x in degrees) + cosd(x): cosine of x (where x in degrees) + tand(x): tangent of x (where x in degrees) + cotd(x): cotangent of x (where x in degrees) + asin(x): arc sine of x (result is in radian; return 0 if |x|>1) + acos(x): arc cosine of x (result is in radian; return 0 if |x|>1) + atan(x): arc tangent of x (result is in radian) + acot(x): arc cotangent of x (result is in radian) + asind(x): arc sine of x (result is in degrees; return 0 if |x|>1) + acosd(x): arc cosine of x (result is in degrees; return 0 if |x|>1) + atand(x): arc tangent of x (result is in degrees) + acotd(x): arc cotangent of x (result is in degrees) + sinh(x): hyperbolic sine of x + cosh(x): hyperbolic cosine of x + tanh(x): hyperbolic tangent of x + coth(x): hyperbolic tangent of x + gamma(x): gamma function of x (i.e. factorial of x-1) + erfP(x): ? + erfQ(x): ? + erf(x): error function of x + erfc(x): complementary error function of x + sinc(x): sinus cardinalis, sin(x)/x + ceil(x): the smallest integer i with i>=x + floor(x): the largest integer i with i<=x + nint(x): the integer nearest to x + min(x,y): the smaller of the two arguments x and y + max(x,y): the smaller of the two arguments x and y + ran(x,y): a random number between x and y + kwwc(x,b): (*) Fourier transform \int_0^\infty dq \cos(xq) \exp(q^b) + kwws(x,b): (*) Fourier transform \int_0^\infty dq \sin(xq) \exp(q^b) + (*) For more info on kwwc and kwws, see http://joachimwuttke.de/kww/ + gauss(x,s): the normalized Gaussian exp(-x^2/2/s^2)/sqrt(2 pi)/s + gnn(x,s): the unnormalized Gaussian exp(-x^2/2/s^2) + cauchy(x,w): the Cauchy-Lorentz function ?/(x^2+w^2) + q4w(x,y,z): ? + cauchy2(x,y,z): ? + skww(x,y,z): ? + rehavneg(x,y,z): real part of the Havriliak-Negami function + imhavneg(x,y,z): imaginary part of the Havriliak-Negamit function diff --git a/wiki/coding_conventions.md b/wiki/coding_conventions.md new file mode 100644 index 0000000000000000000000000000000000000000..d1ed559852ef8c09dc89a9595f6c421a8917ca01 --- /dev/null +++ b/wiki/coding_conventions.md @@ -0,0 +1,104 @@ +**[FRIDA](FRIDA) C++ coding conventions** + +As all other implementation notes, this is subject to change during the +real implementation work. + +### Appearance + +##### Indentation + +The code layout basically follows Kernigham & Ritchie style with 4 +blanks indentation. + +If in doubt, the command + + indent -kr <file> + +can be used on new code. However, this automatic layout tool shall not +be used on existing code. It\'s layout is not free of inconsistencies, +especially regarding blanks before and inside parentheses and blanks +before/after \* and & operators in function argument lists - but it\'s +much better than any automatic solution. + +##### Doxygen comments + + //! <short description>, above a class|typedef|.. in the declaration (.h) file + /** <long description>, follows short description if one line is not enough + ... */ + <variable> ///!< inline explanation of a variable + +### Variable names + +##### Cases + +class, namespace: upper case, camel case if appropriate + +singleton class instance: the\<Class\>, e.g. theWorkspace + +class member functions: lowercase. This is a change from Frida-pre-2 + +vectors and other non-elementary data: lowercase, or camel case starting +with lowercase + +pointer class: \<Class\>Ptr + +pointer and smart pointer: no explicit p.., ..prt, or whatever. Look at +the definition which is rarely far away. + +##### Variable names + +Should be more explicit than in earlier versions, as far as the generic +frame is concerned. + +In contrast, inner loops of numeric routines continue to be as concise +as possible. + +##### Letters + +Consistent use of letters is primordial in cryptic programming. Beware +however pedantic application of Hungarian convention. + +- **c..** refers to a curve, where **d..** refers to a data file +- **co..** is a coordinate, **na..** its name, **un..** its unit +- **f..** likely a file name. For file descriptors, use uppercase + **F** +- **i..** likely an integer. Specifically: **i**, **ii**, \... run + over data points +- **j..** likely an integer. Specifically: **j**, **jj**, \... run + over spectra +- **k..** likely an integer. Specifically: **k**, **kk**, \... run + over files +- **n..** likely an integer. Typically: number of \... Specifically so + in **ni**, **nj**, **nk** +- **s..** or **S..** refers to a scan (shall we change case from + pre-2) +- **t..** (text) could be used for things converted to string (because + s is take otherwise). But again, generally no Hungarian convention. + +or curves) in WorkFile / DataFile / CurvFile + +- **w..** refers to a workfile, usually through a smart pointer + +```{=html} +<!-- --> +``` +- **A** an abstract syntax tree +- **C..** class +- **D**, **DD** double in value or channel +- **E**, **EE** variance in value or channel +- **F_in**, **F_out** are external file descriptors +- **I**, **II** integer in value or channel +- **P** fit parameters +- **T** Tuple: within Spectrum, T\[0\] points to channel x, T\[1\] + to y. **nT** is the internal representation of *ni* +- **V** a vector which is a particular important member of a class; + for instance the vector of slices + +### C++ idiom + +##### const declarations + +do_something( const int size_in ): \"const\" has no effect upon calling +program (size_in won\'t be modified anyway, because it is passed *by +value* to do_something). Therefore: write \"const\" only in the +implementation, not in the declaration. diff --git a/wiki/commands.md b/wiki/commands.md new file mode 100644 index 0000000000000000000000000000000000000000..6b0cee437f6b82034b29c263a4b145467c354c51 --- /dev/null +++ b/wiki/commands.md @@ -0,0 +1,247 @@ +Commands +======== + +*Top:* [Frida handbook](/start)\ +*Previous:* [Data model](Data model)\ +*Next:* [Overwrite or Duplicate](Overwrite or Duplicate)\ +This page gives a list of Frida commands. It is only updated once in a +while. Up-to-date information can be retrieved at any moment from the +running Frida program: At the command prompt, + +- type **h** to obtain a list of *plain commands* and *command groups* +- type a *command group* initial letter (like **c**, **g**, **m**, + \...) to obtain a full list of commands within that group. + +#### Plain commands and command groups + + > h + This is Frida, version 2.3.0c + Plain commands: + a add to plot + p plot (g for list of other plot commands) + t show execution time (for development) + ! execute shell command + quit terminate Frida session + Command groups (type initial letter for help): + b backup of online files + c curves and fits + d directory (inspect internal files) + e edit (change textual parameters) + g graphics setup + f file input, output, creation + m manipulate data (select, bin, reorganise, ...) + o operate on variables, compute integrals, ... + r read raw data (currently only for SPHERES) + _ special / temporary commands + Modifier (appended to command): + ! operate in place (overwrite) + More help: + \h help on meta commands and macros + hf functions of scalar arguments + hid dimension-reducing functionals of data (replaced 'oio') + hic dimension-reducing functionals of curves + ho operators by precedence + hr handling resolution functions (convolution, Dirac delta function) + +#### Curve commands + + > c + Curve and fit commands: + cc create + cca create artificial curve (no data reference) + Basic operations: + cm modify function + cd set data file to fit + cv set file to convolute with (cv-: no convolution) + cr set fit range restriction + cf fit + cfs fit, allow slow convolution + cp print parameters + Parameter attributes: + cx fix parameters + cg make parameters global (same for all curves in the file) + cu unfix parameters (remove fixed and global attributes) + File attributes: + cwc weighing: constant + cwl weighing: logarithmic + cwv weighing: reciprocal variance (default) + cwvd weighing: reciprocal variance, data only + cwvc weighing: reciprocal variance, curve only + cgd+ graphics: take x points from data reference file + cgd- graphics: create x points by dividing the plot window + Frida-wide settings for fitting: + csf set gradient step factor + cse set sqrt of differentiation step + csc set number of calls + csv set verbosity + csm set fit metric + cst set maximum duration of fits + Frida-wide settings for numeric integration: + cn? show parameters + cnm set integration mode + cna set eps abs + cnr set eps rel + +#### Display commands + + > d + Directory commands: inspect internal files + df list of files + dz z-values and spectral ranges + dp x|y-values + dy y-values + dr real parameters + dc coordinate names and units + dd doc lines + +#### Edit commands + + > e + Edit parameters: + ef file name + ec<coord> name and unit of <coord>=x,y,z0,.. + ed documentation lines + +#### File commands + + > f + File commands: + fl load from disk + fs save to disk + fso save to disk, allow overwrite + fe1 export (only z and y, one line per spectrum) + fe2 export (one block per spectrum: z header and x-y lines) + fc copy + fx exchange (copy, then delete source) + fdel delete + fm make + fr read in + ft(h|v|c)[m][s][d] read tab + h horizontal (only y values, one line per spectrum) + v vertical (one x-y pair per line) + c choose vertical columns + m multi-block (blocks begin with z line) + s from script + d input includes dy + +#### Graphics commands + + > g + Graphics commands: + p plot + pr plot with new autoranges + pn plot next spectrum + pv plot previous spectrum + a add to plot + an add next spectrum + av add previous spectrum + gd graphic dialog (feed commands to gnuplot) + Save for print: + gp plot to complete .ps file + gf plot to short .psX file + Change window: + gw list of plot windows + g<n> switch to plot window <n> + Set axes: + gx set x range + gy set x range + ga auto-ranges + gxa x auto-range + gya y auto-range + gxl log x (toggle; or use gxl+ gxl-) + gyl log y + gxf force x into frame (toggle; or use gxf+ gxf-) + gyf force y into frame (toggle; or use gyf+ gyf-) + Other setup: + ge plot error bars (toggle, or ge+ ge-) + gnd set maximum number of data points + gnc number of equidistant base points for curve plotting + gcr allow iterative refinement for curve plotting? + Help: + gsym explain plot symbols + +#### Manipulation commands + + > m + Commands to manipulate data: + m/ protect spectra from forthcoming operations + m/- lift protections + Generic manipulations (expression driven): + md generic delete + mr generic retain + Per point: + mpd delete + mpr retain + mpa average (=bin), list controlled + mpaf average, reduce number of points by fixed factor + mpaea average according to error bound (absolute) + mpaer average according to error bound (relative) + mpo sort + mpq average when x equal + mpsym symmetrize: map -x -> x, then average + Per spectrum: + msd delete + msr retain + msa average + msb break into files + msj join + ms* spawn spectra + msx z_max <-> x + mso sort by expression + msoz sort by z-values + mzx exchange / rotate z coordinates + mz- delete z coordinate + Per file: + mfj merge ("join") + mfj+ merge, adding z=number-of-input-file + mfjp merge pointwise + mr- remove real par + Interpolation operations: + mi locate y in x' to determine y'(x'=y) + Histogram operations: + mhm bin data + + +#### Operation commands + + > o + Commands to operate on variables or spectra: + ox on x + oy on y + oz0,.. on z0,.. + oz+ add new z value + or0,.. on r0,.. + or+ add new real par + op0,.. on p0,.. (fit parameter) + op* on all fit parameters + opa take absolute value + opc copy from another curve + ofi functional integrate + ofs summation up to current index + ofd functional derivate + ofac functional autocorrelate + oi rank-reducing operations (generalized integrals) + oixy choose two columns, save them as x and y + oixyd choose three columns, save them as x, y, dy + +#### Raw data read in commands + + > r + read raw data from SPHERES: + ry load energy spectra + ry2 load also spectra for open state + ry4 load also chopper histograms + ry8 load spectra for two directions separately + ryd load only spectra for two directions + rs load series of energy spectra + rs2 load also spectra for open state + +#### Meta commands and macros + + > \h + meta commands: + \i <file> include file + \q escape to main menu + \wh write history + +*Last updated for Frida2.3.0c.* diff --git a/wiki/curves.md b/wiki/curves.md new file mode 100644 index 0000000000000000000000000000000000000000..d6402ff9dd2c1cc19359dab1d8bab97f50600345 --- /dev/null +++ b/wiki/curves.md @@ -0,0 +1,19 @@ +Curves +====== + +*Top:* [Frida handbook](/start)\ +*Previous:* [Built-in functions](Built-in functions)\ +*Next:* [External curves](External curves)\ + +#### Introductory example + + > fm 101 3 testdata # create test data set + 0 > =ox (i-50)/10 # x-scale now goes from -5 to 5 in steps of 0.1 + 0 > =oy exp(-(x-j)^2/2/(1.4^j)^2)*ran(.8,1.2+0*i) # y(x) is now a noisy Gaussian + 0 > cc p0*exp(-(t-p1)^2/2/p2^2) # create fit curve + 1 > op0 1 # set initial value of fit parameter + 1 > op1 1 # (in this case not really necessary, + 1 > op2 1 # because by default the initial value is 1 anyway) + 1 > cf # perform a fit + 1 > 0,1 p 1 # plot spectrum 1 (data and curve) + 0,1 > cp # show fit parameters diff --git a/wiki/data_model.md b/wiki/data_model.md new file mode 100644 index 0000000000000000000000000000000000000000..00bfaa534e7e51412bad4e7cb1c1b029fc2c915c --- /dev/null +++ b/wiki/data_model.md @@ -0,0 +1,49 @@ +Data Model +========== + +*Top:* [Frida handbook](/start)\ +*Next:* [Commands](Commands) + +Frida is basically a huge collection of methods that operate on data +or/and curve files. + +### Data files + +A **data file** is basically an array of »spectra«. A **spectrum** +contains *x,y* data pairs (or *x,y,dy* triples, if errors are given). +The physical meaning of *x* and *y* is stored as »coordinate« +information in the »meta data« of the file. A **coordinate** consists of +»name« and »unit«, e.g. \"mass (kg)\" or \"counts (Hz)\". + +If a data file contains more than one spectrum, it is mandatory that +there is a *z* coordinate; there may be several (denoted *z0, z1, +\...*). Each spectrum contains as many *z* entries as there are *z* +coordinates. + +There are additional **file meta data**, like the »file name« and the +»documentation« array. Furthermore, a file may contain an arbitrary +number of »file constants«. An **file constant** is a coordinate-value +pair, denoted *r0, r1, \...*. For instance, *r0* may contain the +coordinate \"T (K)\" and the value 320 to indicate a temperature of 320 +K. + +### Curve files + +A **curve file** contains additional file meta data, most importantly a +**curve definition**. This is either an »expression« or a »script +command«. Instead of a set of »spectra«, a curve file contains a set of +»curves«. Each **curve** contains an array of curve parameters *p0, p1, +\...* (as many as prescribed by the curve definition), an array of flags +that indicate whether a parameter is free or fixed, and a few numeric +values *cq0, cq1, cq2* describing fit quality. + +### The data model is stable for many years + +The data model is the most basic design feature of FRIDA. Internal and +external storage formats and all method interfaces depend on it. It has +remained very stable over the years (whereas the +[technologies](technologies) used for implementation have changed almost +completely). Major changes were the addition of multi-*z*-coordinates in +1996, the insertion of an abstraction layer for *x-y-dy* in 2006, and +full support for r pars in 2010. Backward compatibility is always +maintained by providing read-in methods for older data files. diff --git a/wiki/expressions.md b/wiki/expressions.md new file mode 100644 index 0000000000000000000000000000000000000000..bb694311d50e6410236982b0f30f3f206d038728 --- /dev/null +++ b/wiki/expressions.md @@ -0,0 +1,185 @@ +Expressions +=========== + +*Top:* [Frida handbook](/start)\ +*Previous:* [Overwrite or Duplicate](Overwrite or Duplicate)\ +*Next:* [Built-in functions](Built-in functions)\ + +#### Overview + +Expressions are used in Frida at many different places. Some important +uses are: + +In calculator mode: + + . 4+7 # yields 11 + . sin(pi/2) # yields 1 + . y[2,0,7] # yields the y value of file 2, spectrum 0, point 7 + +In operations like **ox**, **oy**, **op0**, \...: + + 3 > oy x^2 # in file 3 (a data file), set y:=x^2 + 5 > op0 y[4,0,j] # in file 5 (a curve file), initialize p0 with a value taken from file 4 + +In commands **dp**, **mr**, **md**, evaluated as boolean to select data +points: + + 3 > dp y<10 # show all points with y<10 + 3 > mr x>0 # retain points with x>0, delete all other points + 3 > md x<=0 # another way of saying the same + +Expressions are the standard means to define a fit curve: + + 8 > cc p0+p1*t # create a curve to fit file 8, using an affine-linear function + +#### Syntax + +In the following, a very basic variant of Backus-Naur form is used. A +dot as in \<an_integer_value.\> indicates that the meaning of this node +is so obvious that a formal definition is not needed. + + expr: + <arithmetic_expr> # arithmetic expression + <function_evaluation> # built-in function + resol ( <shift> ) # Dirac delta function, copies a given resolution function + conv ( <theory>, <shift> ) # convolutes 'theory' with a given resolution function + <numeric_value> # like 1.23 + <variable_reference> # like x[3,0,j] + t # running variable for curve evaluation + <curve_reference> ( <expr> ) # like c[4](x-0.2) + +The special expressions resol, conv, t are explained in +[curves](curves). + + arithmetic_expr: + ( <expr> ) # parentheses around expression change nothing + - <expr> # negative value + ! <expr> # boolean negation + <expr> <bin_op> <expr> # binary operator, see below + <expr> ? <expr> : <expr> # conditional evaluation (lower precedence than all bin_op's) + + bin_op: # ordered by decreasing precedence + +- # error ( e.g. 193+-2 ) + ^ # exponentiation + over # binomial coefficient, 5 over 3 = 5!/3!/2! = 10 + * | / | mod | div # multiplication, division, remainder, integer division + + | - # addition, subtraction + > | >= | <= | < # comparison + == | != # equality, inequality + xor # boolean exclusive or + && # boolean and + || # boolean or + + function_evaluation: + ln ( <expr> ) # natural logarithm + ... # many more functions of one argument + min ( <expr>, <expr> ) # the smaller of the two arguments + ... # some more functions of two arguments + rehavneg ( <expr>, <expr>, <expr> ) # real part of the Havrialiak-Negami function + ... # some more functions of three arguments + +for a full listing of built-in functions, see +[built-in functions](built-in functions), or use **hf** at the FRIDA +main level + + numeric_value: + <an_integer_number.> # like 8 or -17 + <a_floating_point_number.> # like .2 or 2.72 or -3.7e-3 + pi | e # these two mathematic constants are hard-coded + + variable_reference: + var_label # e.g. y, meaning y[k,j,i] with k,j,i given by context + var_label [ <expr> ] # e.g. p0[8] + jvar_label [ <expr>, <expr> ] # e.g. z2[6,3] + ivar_label [ <expr>, <expr>, <expr> ] # e.g. x[6,3,18] + +Note: above, \"\[\" and \"\]\" are *not* BNF notation to indicate an +optional argument; they stand for real brackets. Actually, all indices +*are* optional: if omitted, the default values k,j,i are filled in, +depending on position. + + var_label: + k # number of file in online memory + r<num.> # file constants r0, r1, ... + dr<num.> # error thereof (not yet implemented) + nj # number of spectra + jvar_label # j-dependent variable (j is the spectrum index) + + jvar_label: + j # index of spectrum + z<num.> # z variables z0, z1, ... + dz<num.> # error thereof (not yet implemented) + p<num.> # curve parameters p0, p1, ... + dp<num.> # error thereof (not yet implemented) + cq<num.> # curve quality cq0, cq1, cq2 + ni # number of data points + ivar_label # i-dependent variable (i is the data point index) + + ivar_label: + i # index of data point + x # x value + y # y value + dy # error thereof + + curve_reference: + c # c, meaning c[k,j], with k,j given by context + c [ <expr> ] # e.g. c[3], meaning c[3,j], with j given by context + c [ <expr>, <expr> ] # e.g. c[3,0] + +Same note as for variable_reference: \"\[\" and \"\]\" are *not* BNF +notation to indicate an optional argument; they stand for real brackets. +All indices *are* optional: if omitted, the default values k,j are +filled in, depending on position. + +#### Context-dependent evaluation: running variables k,j,i + +How expressions are evaluated depends on the context in which they are +used. For instance, the way `dirac` and `conv` are evaluated depends on +whether or not a convolutand is set. See [curves](curves) for more on +this. + +In evaluating variable references like `x` or `y[0,k-3,j]` implicit or +explicit use is made of running indices `k`, `j`, `i`. A reference +without index, like `x`, is just a shortcut for `x[k,j,i]`. Any of these +indices may be omitted: + +- `x[8]` stands for `x[8,j,i]`; +- `x[,3]` stands for `x[k,3,i]`; +- `x[,,0]]` stands for `x[k,j,0]`; and so on. + +Depending on context, loops are performed over `k`, `j`, `i`. + +##### Loop over k + +The variable `k` stands for the number of a file in the online memory. +Whenever a command is performed for a given file selection, this can be +seen as running a loop over `k`. For instance, in the simple command + + 3-5 > . y[,0,3] + +the value y\[k,0,3\] is shown, with `k` taking successively the values +3, 4, 5, as given by the preset file selection indicated in the prompt. + +A more complex example: files 10 to 13 are curve files. Then the command + + ? > 10:13 cd k-8 + +means, that curve file 10 shall be fitted to data file 10-8=2, curve +file 11 to data file 3, and so on. + +##### Loops over j and i + +A command like **oy** implies a loop over `j` and `i`: + + 4 > oy x^2 + +is executed as follows: + + for all j with 0<=j<nj: + for all i with 0<=i<ni[j]: + let y[k,j,i] := x[k,j,i]^2 + +In contrast, a command like **oz0** or **op2** or **oi** implies a loop +over `j` only. + +*Not up to date.* diff --git a/wiki/external_curves.md b/wiki/external_curves.md new file mode 100644 index 0000000000000000000000000000000000000000..1b8f2ff8efcfe455813cc97a79ba06825297a311 --- /dev/null +++ b/wiki/external_curves.md @@ -0,0 +1,36 @@ +### FRIDA: External curves + +This page is part of the [Frida online manual](/start).\ +Usually, fit [curves](curves) are constructed using +[built-in functions](built-in functions). However, it is also possible +to create a curve that is computed through an external computer program. + +The command + + > cca h + +gives up-to date information on the different ways to construct an +external curve. + +At present, the codes + + ecr, eir, ecw, eiw + +can be used to set external evaluation mode. Frida then prompts for the +name of the external program and for the number of parameters. + +The above codes bear the following meanings: + +- **c**: The external program reads parameters from the command line. +- **i**: The external program reads parameters from stdin; the first + input line must contain the number of parameters (for redundant + cross-check); the following lines contain one parameter each. +- **r**: The external program reads x values from stdin (after reading + parameters, if applicable); it first expects one line containing the + number of x values; each of the following lines contains one x + value. +- **w**: The external program sets its own x grid; Frida then + interpolates. + +In any case, the external program outputs lines containing data pairs +`x y`. diff --git a/wiki/frida.ini.md b/wiki/frida.ini.md new file mode 100644 index 0000000000000000000000000000000000000000..7b1e30396bb0451f4036266730fc2326f558b30b --- /dev/null +++ b/wiki/frida.ini.md @@ -0,0 +1,35 @@ +### frida.ini + +This page is part of the [Frida](Frida) online handbook. It describes +the local setup, using the file `frida.ini`. + +#### Introduction + +When the program Frida2 is started, a setup file `frida.ini` is read. +This file contains some macro definitions that are needed to execute +certain commands. For instance, the macro definition + + \d psdir ~/gnew/ + +is needed by the commands *gf* and *gp* that write PostScript files to +the directory `psdir`, by default defined as `~/gnew/`. + +#### Location of the .ini file + +When Frida is installed (command *make install*), the file `frida.ini` +is copied from the distributions `share/` directory to +`/usr/local/etc/frida/`. + +If you want to modify the setup, then + +- do so in the directory `/usr/local/etc/frida/` (and not in the + distribution\'s `share/` directory) +- make the directory write-protected + (`chmod a-w /usr/local/etc/frida`) so that your modifications cannot + be overwritten when you update your Frida installation +- when updating Frida, it may be necessary that you copy changes to + your modified `frida.ini` by hand. + +We could easily extend the setup mechanism to allow different users on +the same machine to have personalized ini files. However, this will only +be implemented if a convincing use case is proposed. diff --git a/wiki/frida_under_macos.md b/wiki/frida_under_macos.md new file mode 100644 index 0000000000000000000000000000000000000000..47a6b04081f65cbe0907fb3ace78ea7ef031a661 --- /dev/null +++ b/wiki/frida_under_macos.md @@ -0,0 +1,82 @@ +##### Frida installation on the Macintosh + +Notes by Rémi Busselez. + +First you need to install Xcode to behave a compiler. Afterwhat you need +to install Macports, to build a (quasi) sane unix environment. + +bison, flex and blas come with Apple XCode and does not need to be +installed with Macports + +Install readline, gnuplot, yaml-cpp, gsl, fftw3 and boost (take a long +time !) Using Macports, gnuplot install readline for his own +dependencies + +Environment should be OK now. + +You also need to link boost headers in /usr/include + +sudo ln -s /opt/local/include/boost /usr/include + +kww: + + ./configure --prefix=/opt/local (to be consistent with Macports usage install in /opt/local) + make + sudo make install + +Work Fine + +lmfit: + + ./configure --prefix=/opt/local (to be consistent with Macports usage install in /opt/local) + make + sudo make install + +Work fine well + +frida-libs: + + ./configure --prefix=/opt/local (to be consistent with Macports usage install in /opt/local) + make + sudo make install + +Work fine well + +To compile frida, it is just a little bit more complicate + +As readline dev files stand in /opt/local/include, we will have to +include this path in CPPFLAGS + +As readline library files (the real one not the symbolic link to +libedit) stand in /opt/local/lib we will have to include it in LDFLAGS + +As malloc.h is in /usr/include/malloc/malloc.h we will have to include +this directory in CPPFLAGS + +Moreover as a symbolic linc is present to simulate a fake readline +library towards the libedit library, you may have to delete it +temporarily: + +sudo rm /usr/lib/libreadline.dylib + +(do not forget to relink after the installation is performed: sudo ln -s +/usr/lib/libedit.3.dylib /usr/lib/libreadline.dyib) + +To configure all the path just: + +./configure \--prefix=/opt/local CPPFLAGS=\"-I/opt/local/include +-I/usr/include/malloc -I/opt/local/include/yaml-cpp\" +LDFLAGS=\"-L/opt/local/lib + +To define the uint in xax_lex.cpp and xax_yacc.cpp add + +\#include \<sys/types.h\> + +You can build it with + +make + +and install it It should compile fine + +Afterwhat you will have to modify frida.ini in /opt/local/etc/frida and +change path like /usr/local/share/frida with /opt/local/share/frida diff --git a/wiki/frida_under_windows.md b/wiki/frida_under_windows.md new file mode 100644 index 0000000000000000000000000000000000000000..27f51d3a8504f9f278130ff3dd373ae1ac1f3ece --- /dev/null +++ b/wiki/frida_under_windows.md @@ -0,0 +1,24 @@ +Frida under MS Windows +====================== + +Running Frida under Windows is not currently explored by ourselves; we +depend on reports from users. + +### Build + +Basic approaches: + +- So far, nobody has tried to build Frida natively under Windows + (using Visual Studio and so on) +- Using the [Cygwin](http://www.cygwin.com) shell, Frida can be + installed as in a Linux environment. +- Another solution is using [VMWare](http://www.vmware.com) to run a + Linux environment on a windows box. + +### Graphics + +In former years, there was a severe restriction: as Windows did not +support unix-like pipes, Frida could not feed data to Gnuplot for +plotting. However, according to +<http://www2.warwick.ac.uk/fac/sci/moac/currentstudents/peter_cock/cygwin/part6/>, +this problem is solved by now. Please report your experience. diff --git a/wiki/git.md b/wiki/git.md new file mode 100644 index 0000000000000000000000000000000000000000..0e685ee5acb78a9aa23772f391023e54b4f528eb --- /dev/null +++ b/wiki/git.md @@ -0,0 +1,42 @@ +frida.git +========= + +Working with the **git** development tree of [Frida](start). + +For the simpler standard build procedure, see [build](build). + +#### Requirements + +Besides the software needed for building Frida, the following is +required: + +- git +- autoconf2.13 even though it is labeled \"obsolete\" in latest Debian + +#### Download + +Then + + git clone git://apps.jcns.fz-juelich.de/jwu-varia.git/ + git clone git://apps.jcns.fz-juelich.de/frida.git/ + +will create folders *jwu-varia* and *frida* in the current directory. +The Frida source is in *frida/pub*. *jwu-varia* provides libraries in +the subdirectories *kww*, *lmfit*, *read-plus*, *yampl-cpp-at*. Other +subdirectories are of no importance for Frida. + +#### Build + +Go to each of these subdirectories and run + + ./configure + make + make install # as root + +To modify makefiles, one more software requirement: + +- GNU autotools (autoconf, automake, libtool, autotools-dev) + +Then run + + autoreconf -fi diff --git a/wiki/historic_data_formats.md b/wiki/historic_data_formats.md new file mode 100644 index 0000000000000000000000000000000000000000..7b9093cefddb11851d915cfe7045509617066e5d --- /dev/null +++ b/wiki/historic_data_formats.md @@ -0,0 +1,127 @@ +**[FRIDA](FRIDA) implementation note:** documentation of historic data +formats + +Note that these formats are largely isomorphous, since the +[data model](data model) has little changed over the years. The last +major change was the introduction of multi-*z*-coordinates (probably +along with the i96-Format). + +#### ASCII-01 + +Used in preliminary versions of Frida-2. File name extension \".a01\". A +sample file with 2 spectra: + + ASCII-01 JWu + typ D + fil grid5x2 + doc x := 2.70*i + doc z0 := 2.35*j + doc y := x^2+12.34*y + doc coordinate x renamed from 2.7*i (*) into Breite (m) + doc coordinate y renamed from 2.7*i^2+12.34*y (*^?) into Hoehe (m) + doc coordinate z0 renamed from 2.35*j (*) into Tiefe (m) + doc saved as grid5x2 on Tue Mar 21 23:16:14 2006 + x Breite m + y Hoehe m + z0 Tiefe m + #j 2 + scn 0 + Z 1 + 0 + XY 5 + 0 0 + 2.7 7.29 + 5.4 29.16 + 8.1 65.61 + 10.8 116.64 + scn 1 + Z 1 + 2.35 + XY 5 + 0 0 + 2.7 7.29 + 5.4 29.16 + 8.1 65.61 + 10.8 116.64 + eof + +Comments: + +- Very nice to read +- The many blanks in the XY sections are an enormous waste of storage. +- The usage of \"\#\" in the meaning of \"number of\" needs to be + rethought, since in the scripting world, its the comment prefix + (especially so in column 0) +- \"scn\" stood for \"scan\"; throughout Frida-2, \"scan\" shall be + renamed back to \"spectrum\" +- \"doc \" should not be repeated in every line; instead, use begin + and end tag + +#### ASCII-96 + +Used in Frida-1. Is used as a standard since 1996. A sample file with 2 +spectra: + + ASCII-96 + (a24,a56) + fil filename + tit long title + doc oosc + dir + sub instrument + &eob 2 (end of block) ------------------------ + (a24,i16) + ?det-bal-sym 0 + @sam-erg-gain -1 + plot-sy# 0 + &eob 3 (end of block) ------------------------ + (a24,a24,g20.10) + E0 meV 2.334191189 + T K 1200.000000 + cts[mon] 420564.0000 + at-mass amu 1.000000000 + FP m 2.483000040 + Cw usec 7.125000480 + &eob 4 (end of block) ------------------------ + (a4,a24,a24) + x w meV + y S(2th,w) meV-1 + z1 2th + &eob 5 (end of block) ------------------------ + (a80) + IN6: + "ht" + normalized to 27 + y=y-#, #=6e-6 + y=y/#, #=y=S(2th,w) + deleted sptra 95-97 + chs kept 1-1021 + &eob 6 (end of block) ------------------------ + (2i8) + (i16,4g16.8/5g16.8) + (3(2x,g16.8)) + 2 1 + &spectrum 1 + 5 5.000000 + -1.6909029 0.27020141 0.30771835 + -1.6895938 0.26911141 0.30647704 + -0.15219796 0.10242032 0.58738120E-01 + 0.10496248E-01 2.2390736 0.28680437 + 0.17196839 -0.27120464E-01 0.18103506E-01 + &spectrum 2 + 5 10.000000 + -1.7605379 -0.65812241E-01 0.16396864E-05 + -1.7594403 -0.65739436E-01 0.16396864E-05 + -1.7583395 0.20337768 0.26886582 + -1.7572356 -0.65237570E-01 0.16271324E-05 + -1.7561285 -0.64987736E-01 0.16208946E-05 + +Comments: + +- clear structure, very easy to read +- sectioned by type definitions (will become obsolete for scripting + languages) +- huge amount of storage capacity needed (blanks and zeros in measured + data) + +#### pre-96 diff --git a/wiki/installation.md b/wiki/installation.md new file mode 100644 index 0000000000000000000000000000000000000000..23543545aba9dadb6832ab5322c2072c1bb0705b --- /dev/null +++ b/wiki/installation.md @@ -0,0 +1,223 @@ +Installation +============ + +*Top:* [Frida handbook](start)\ +*Previous:* [Introduction](Introduction)\ +*Next:* [Tutorial](/tutorial/start) + +This page describes the standard procedure to download, compile, and +install Frida. + +*See also:* + +- [frida.ini](frida.ini): local set up +- [git](git): alternative download and build procedure, using the + development tree + +Requirements +------------ + +### Operating systems + +- **Linux.** Our standard environment is Debian. Please report if + there are difficulties with other distributions. +- **Mac OS.** See [Frida under MacOS](Frida under MacOS). +- **MS Windows.** See [Frida under Windows](Frida under Windows). + +### Required packages + +To compile the sources, one needs the compilation tools + +- pkg-config (for configuring the Fridalibs) +- make (for building the Fridalibs) +- cmake (for building Frida) +- g++ (GNU C++ compiler. Starting with Frida2.1.6a, we require basic + support for the new language standard C++11. At least g++-4.5 is + required) +- bison +- flex + +and the libraries (-dev packages) + +- readline (at least version 4.3) +- history (usually packaged as part of libreadline-dev) +- gsl +- blas (required by gsl) +- fftw3 +- boost (module filesystem, plus header-only modules format, + algorithm) +- yaml-cpp version **0.5** (Neither 0.3 nor 0.6 will work. Why are + these people breaking their API over and again?) + +At runtime, one external program is needed: + +- gnuplot (With X11 support. This comes typically as package + \"gnuplot\" or \"gnuplot-x11\". Package \"gnuplot-nox\" will not + work; typical error messages are \"unrecognized option -title\" and + \"util.c: No such file or directory\") + +In Debian/Ubuntu, the following command installs all required packages: + + sudo aptitude install \ + make \ + g++ \ + bison \ + flex \ + libcerf-dev \ + libreadline-dev \ + libgsl0-dev \ + libfftw3-dev \ + libyaml-cpp-dev \ + libboost-regex-dev \ + libboost-date-time-dev \ + gnuplot + +Other required packages are automatically installed through +dependencies. In particular, each development package (-dev, containing +the .h files) depends on a library package that contains the +corresponding shared library. + +The libboost regex and date-time development packages depend on +libboost-dev which contains, among others, the shared pointer module +(which is implemented through .h files only; no shared library). + +There may be some incompatibility between g++ and libboost versions. +With g++ version \>= 4.4, we need at least libboost1.40. + +If libcerf is not available as a package, then please report this +problem. As a workaround, install from source, as described below. + +Download Frida and dependences +------------------------------ + +Download most recent source packages: + +- <https://jugit.fz-juelich.de/mlz/kww/-/releases> +- <https://jugit.fz-juelich.de/mlz/lmfit/-/releases> +- <https://jugit.fz-juelich.de/mlz/libcerf/-/releases> (on some + platforms also available as a binary package) +- <https://jugit.fz-juelich.de/mlz/ransampl/-/releases> +- <https://jugit.fz-juelich.de/mlz/frida/-/releases> + +On the indicated \"release\" pages, go to the latest release (on top), +and download the `.tgz` archive file. + +Build library dependences +------------------------- + +Unpack the libraries kww, lmfit, ransample, and libcerf: + + tar xzvf <source>.tgz + +Build: + + cd <source directory> + mkdir build + cd build + cmake .. + make + ctest + make install + +On some systems, it may be necessary to register newly installed +libraries: + + sudo ldconfig # may be required in Ubuntu + sudo /sbin/ldconfig /usr/local/bin # may be required in Suse + +Build Frida +----------- + +Unpack and build the main program: + + tar xzvf frida2-<version>.tgz + cd frida2 + mkdir build + cd build + cmake .. + make -j<n> + +where \<n\> is the number of processors to be used in parallel. + +Now run the test suite: + + unset LD_LIBRARY_PATH + ctest -j<n> + +Unsetting the environment variable LD_LIBRARY_PATH ensures that freshly +compiled libraries are used, and not previously installed ones. + +Only proceed if all tests pass. Otherwise run + + ctest -V + +to get more verbous output, and report to the maintainer. + +If tests pass, proceed with + + sudo make install + +Finally, create a directory for the graphics output: + + mkdir ~/gnew + +That\'s it. The command `frida` should start a frida session. + +### A first interactive session + +Try to run the following test session: + + $ frida + ? > fm # enter command fm = file make to create a little data file + Number of points per spectrum [1] ? 11 # enter 11 to overwrite the default 1 + Number of spectra [1] ? # just press RETURN to accept the default value + Save as [grid3] ? # again, we just accept the default value + 0 > oy x^2 # our data file becomes a parabola + 1 > p # plot it + 1 > cc p0*t^p1 # create a fit curve: some power law + 2 > cwc # equal weight to all data points (since no error bars are set) + 2 > cf # fit p0,p1 to the data; should say "success" + 2 > a # add the fit to the plot + 2 > quit # terminate the frida session + +Frequent errors +--------------- + +### When libraries are not found + +If libraries are not found, check LD_LIBRARY_PATH. Possibly, +/usr/local/lib must be added. \[related commands: ldd, ldconfig\]. + +### When libreadline is not found + +If configure fails because libreadline is reportedly not found or +corrupted, please check the config.log for further details. If it +contains error messages like the following + + /usr/users/iff_ns/brodeck/usr/local/lib/libreadline.so: undefined reference to `PC' + /usr/users/iff_ns/brodeck/usr/local/lib/libreadline.so: undefined reference to `tgetflag' + ... + /usr/users/iff_ns/brodeck/usr/local/lib/libreadline.so: undefined reference to `tgetstr' + +then this due to a bug in outdated Linux distributions like CentOS5. See +<https://bugzilla.redhat.com/show_bug.cgi?id=499837>. The missing +symbols are provided by the library ncurses. + + export LDFLAGS="-lncurses" + +and run configure again. + +How to report bugs +------------------ + +If problems persist, please send a bug report. Attach the output of + + uname -a + +and run + + make clean + ./configure | tee configure.log + make | tee make.log + +and attach configure.log and make.log. diff --git a/wiki/introduction.md b/wiki/introduction.md new file mode 100644 index 0000000000000000000000000000000000000000..66ebac0000fb0ed38710833c9e23694513196383 --- /dev/null +++ b/wiki/introduction.md @@ -0,0 +1,38 @@ +Introduction +============ + +*Top:* [Frida handbook](start)\ +*Next:* [Installation](Installation) + +### Welcome to Frida + +**Frida** ("Fast reliable interactive data analysis") is a program for +analysing tabular data. + +It has been written for the analysis of inelastic neutron scattering +data, and it contains highly specialized routines for this purpose. Its +[data model](data model), however, is abstract and generic so that Frida +can be applied to whatever data of the form *y(x,z0,z1,\...)*. + +Frida\'s main **capabilities** are: + +- Import/export tabular data from/to various formats. +- Reorganize, bin, sort, clone data. +- Perform mathematical operations on the data. +- Automatically document all data manipulations. +- Fit the data with user-defined functions. +- Plot data and functions, generating publication-grade PostScript + graphics. + +Besides, Frida is also a powerful pocket calculator. + +Frida is operated through a **command-line interface**. Commands are +cryptic. Everything is designed to offer a maximum of computational +power and flexibility for a minimum of typing. Learning Frida is not +easy, but rewarding. + +#### Citation + +When using Frida in scientific work, please cite as follows: + + J. Wuttke: Frida - fast reliable inelastic data analysis. http://apps.jcns.fz-juelich.de/frida. diff --git a/wiki/overwrite_or_duplicate.md b/wiki/overwrite_or_duplicate.md new file mode 100644 index 0000000000000000000000000000000000000000..1cc499e8e34678fa2c5a5f2aab9974ba3c67f016 --- /dev/null +++ b/wiki/overwrite_or_duplicate.md @@ -0,0 +1,28 @@ +Overwrite or duplicate ? +======================== + +*Top:* [Frida handbook](/start)\ +*Previous:* [Commands](Commands)\ +*Next:* [Expressions](Expressions)\ +Almost all Frida methods that modify files can be chosen to overwrite or +to duplicate the input file. Overwriting the input is also designated as +\"in-place operation\". + +There is no difference in execution speed between overwriting and +duplicating mode. It is merely a question of workspace management. +Duplicating files to often makes it difficult to keep track. When +working on very large files, it may also exceed RAM, so that the system +begins swapping. On the other hand, many operations are irreversible +within Frida. If a file is overwritten accidentally, it may be necessary +to redo an entire Frida session. + +For these reasons, every Frida method has a suitably chosen *default +policy*: Irreversible data modifications duplicate the input by default; +methods that can easily be reverted overwrite their input. + +The default behaviour can be overwritten by a command postfix. Use **!** +to force overwriting. Example: + +- 2 oi! y\[,,0\] + +Without the postfix *!*, *oi* would duplicate. diff --git a/wiki/principles.md b/wiki/principles.md new file mode 100644 index 0000000000000000000000000000000000000000..ed4b5c1badd933ea11414af6ed46ac442b43314d --- /dev/null +++ b/wiki/principles.md @@ -0,0 +1,44 @@ +### FRIDA: basic design ideas and guidelines + +(Joachim Wuttke 2006:) At the time of this writing, promising +developments take place in the field of neutron scattering data +analysis. So far, however, no software has emerged that fully fulfills +my personal requirements. In my opinion, a data analysis program should +satisfy the following requirements: + +1. It must be open source, because + 1. I want to be able to quickly add new modules, and + 2. I want to be able to check the algorithms I am using in key + steps of the data reduction and transformation. +2. It shall not depend on commercial licenses, because I want the + freedom to give the program to future users of my instrument, from + whatever institution they come. +3. It must have routines that are very specific for neutron scattering + (reading different raw data formats, transformation to S(2th,w), + interpolation to S(q,w) &c.). +4. At the same time, it must have very flexible facilities for various + operations on genery x\[-z\]-y data (selective deletion, rebinning, + arithmetic operations, curve fitting &c.) +5. Routine operations on data fields must be optimised for speed (which + almost inevitably implies coding in FORTRAN or in C). +6. It must have a line-oriented input mode designed for the ease of the + expert user. +7. It should have a GUI input facility for the occasional or novice + user. +8. It should have documentation (tutorial, manual, faq). + +If I missed points you consider important, please let me know. Anyway, +even if there are only eight requirements, I guess it is safe to say +that no program on earth comes close to fulfilling all of them. After +inquiring about various alternatives, I think IDA (now Frida-1) is still +an excellent starting point. IDA has the big advantage that fundamental +decisions regarding my requirements (2) and (5) have been made right - +these choices are hard to correct ex post. IDA basically fulfills +requirements (1) through (6) - and it fails completely with (7) and (8). +So, the development priorities are clear: + +- consolidate the existing code base; +- redesign the overall programming framework, uncompromisingly + maintaining the current strengths of IDA, and preparing the ground + for adding a GUI and scripting layer; +- simultaneously, work on the documentation diff --git a/wiki/start.md b/wiki/start.md new file mode 100644 index 0000000000000000000000000000000000000000..d7b2e30998831f33a91f529de64ba765ba2920be --- /dev/null +++ b/wiki/start.md @@ -0,0 +1,80 @@ +Frida +===== + +**Frida** (\"Fast reliable interactive data analysis\") is a versatile +data analysis program with special routines for inelastic neutron +scattering. It is actively maintained by [Joachim +Wuttke](http://www.fz-juelich.de/SharedDocs/Personen/JCNS/EN/Wuttke_J.html). + +News +---- + +12may20: Bug fix release 2.4.3b. + +See the +[changelog](https://jugit.fz-juelich.de/mlz/frida/-/blob/master/CHANGELOG) +for details. + +Documentation +------------- + +- [Introduction](Introduction) +- [Installation](Installation) + - [frida.ini](frida.ini) (the configure file) +- [Tutorial](/tutorial/start) + - [Session 1](/tutorial/basic usage): Command-line basics, load + file, inspect, plot + - [Session 2](/tutorial/simple manipulations): Select, Bin, + reorganize data. + - [Session 3](/tutorial/elementary fitting): Fit. + - [Session 4](/tutorial/fitting with convolution): Convolute + theory with experimental resolution. +- Online manual + - [Data model](Data model) + - [Commands](Commands) + - [Overwrite or Duplicate](Overwrite or Duplicate) + - [Expressions](Expressions) + - [Built-in functions](Built-in functions) + - [Curves](Curves) + - [External curves](External curves) +- Implementation notes (for developers) + - [Design principles](principles) + - [technologies and libraries used](technologies) + - [coding conventions](coding conventions) +- Old Versions + - [historic data formats](historic data formats) + - [](/frida1/) + +```{=html} +<!-- --> +``` +- Offline manual: + -  + +Resources +--------- + +Download location: + +- <https://jugit.fz-juelich.de/mlz/frida/-/tags>: latest release, file + frida\<version\>.tgz + +Or clone from + +- <https://jugit.fz-juelich.de/mlz/frida/-/tree/master>: \"Clone\" + button on the upper right of the main window + +Contact +------- + +### Bug reports + +Please send bug reports to Joachim. The git development snapshot +contains a bug list (\~/TODO). + +### School on Data Analysis + +The first JCNS School on Data Analysis for Quasielastic Neutron +Scattering with Frida took place on 28-29 April 2010. The next school +will be organized as soon as a sufficient number of potential +participants have indicated their interest. diff --git a/wiki/technologies.md b/wiki/technologies.md new file mode 100644 index 0000000000000000000000000000000000000000..fe09077b3d40cdcab8730f1c9f5c3cfefce530c3 --- /dev/null +++ b/wiki/technologies.md @@ -0,0 +1,44 @@ +**[FRIDA](FRIDA) implementation note** + +#### Technologies used in Frida + +Programming language and programming tools: + +- C++ *so far exclusively with gcc compiler* +- flex and bison (yacc implementation) + +Generic (non-mathematical) libraries: + +- standard template library +- [boost libraries](http://www.boost.org), especially + - [shared_ptr](http://www.boost.org/libs/shared_ptr/) Smart + pointer class templates +- GNU readline + +Mathematical libraries: + +- GSL GNU Scientific Library +- [lmfit](http://joachimwuttke.de/lmfit) Levenberg-Marquardt + least-squares fit +- fftw3 very fast Fourier transform +- [kww](http://joachimwuttke.de/kww) Kohlrausch-WIlliams-Watts + function + +Graphics: + +- gnuplot display through a pipe +- editable postscript graphics + +Build tools: + +- GNU make +- GNU autotools + +Documentation: + +- doxygen for in-code documentation, so far only weakly supported +- dokuwiki for online manual + +Version control: + +- git version control diff --git a/wiki/tutorial/basic_usage.md b/wiki/tutorial/basic_usage.md new file mode 100644 index 0000000000000000000000000000000000000000..2a3463141785155b036be8b36384ccd5073a12d8 --- /dev/null +++ b/wiki/tutorial/basic_usage.md @@ -0,0 +1,87 @@ +Basic Usage +=========== + +*Top:* [Frida handbook](/start)\ +*Up:* [Tutorial](start)\ +*Next:* [Simple manipulations](Simple manipulations) + +This is the first session of the Frida tutorial. We will familiarize +ourselves with the command-line interface. We will read a few demo files +from disk, and inspect their contents. Then we will learn how to plot. + +#### Load a file into Frida\'s memory + +Download and unpack the test data, as described in the +[introduction](start) to the tutorial. Launch Frida in the directory +that contains the test data set, and perform the following actions one +by one. Take your time to understand Frida\'s reaction to each single +command. + + command action + -------------- ------------------------------------------------------------------------------------------- + \> h help: print program version, explain one-letter commands, list initials of other commands + \> fl gly??? file-load: load given data files, using bash pattern expansion + 0:4 \> + +At this point, the prompt has changed. It indicates that the next +command will operate by default on the internal files numbered 0-4. +These are of course the five files we have just read from disk. + +#### Inspect files + +We continue the session. + + ----------- ------------------------------------------------------------------------------------ + 0:4 \> d list all commands starting with d + 0:4 \> df directory-files: list the files in Frida\'s internal memory + 0:4 \> dr directory-rpar: list sample temperatures (one real number per file, called *rpar*) + 0:4 \> 0 change default, next command will operate on file 0 + 0 \> (ok, the prompt has changed) + 0 \> dd directory-doc: show the file history (right now rather short) + 0 \> 1 dd same for file 1 (we could also enter \"1\" and \"dd\" as separate commands) + 1 \> dc directory-coordinates: explain the meaning of *x*, *y*, *z0* + ----------- ------------------------------------------------------------------------------------ + +In this example, the fundamental tabular structure *y(x,z0)* is filled +with a neutron scattering law *S(E,q)*. The energy *E* is variable *x*, +and the wavenumber *q* is variable *z0* (in a more complex data set, we +could also have *z1*, *z2*, \...). Functional values *S* are represented +by variable *y*. + + --------- ------------------------------------------------------------------ + 1 \> dz display-z: for each spectrum, show *z* entries and the *x* range + --------- ------------------------------------------------------------------ + +In this example, we have fourteen wavenumbers *q*, and for each *q* we +have a spectrum *S(w)* extending from -14 to 14 ueV and consisting of +351 data triples *x*-*y*-*dy* (*dy* is the standard error of *y*). + +#### Plot spectra + +File 0, gly180, is basically the instrumental resolution function. We +will use it as our first example for plotting. + + -------------- ----------------------------------------------------------------------------- + 1 \> 0 p 0 plot spectrum number 0 of file 1 + (use the mouse or Ctrl-Space to come back to the dialog terminal) + 0 \> a 1 add spectrum 1 + 0 \> a 2 add spectrum 2 (the elastic peak will be cut, the y scale is inappropriate) + 0 \> ga graphic autorange (let plot range be determined by next plot data) + 0 \> p 0:2 (now all data fit into the plot range) + 0 \> pr 5:9 plot with new autoranges (shorthand for ga and p) + 0 \> pr : plot all spectra of file 0 + 0 \> gx -5 5 set *x* range + 0 \> p : + 0 \> gya reset *y* range so that it will be redetermined for the next plot + 0 \> 4 p : plot all spectra of file 4 + 0 \> gy 0 40 set *y* range + 0 \> 0:4 p 6 plot spectrum 6 for all files + 0:4 \> g list graphics commands + 0:4 \> gw list graphic windows (active window is marked by \*) + 0:4 \> g2 goto graphic window 2 (linear in *x*, logarithmic in *y*) + 0:4 \> p 6 plot spectrum 6 + 0:4 \> gp replot current image into a postscript file + 0:4 \> quit terminate the Frida session + -------------- ----------------------------------------------------------------------------- + +*Last updated for Frida2.3.0c.* diff --git a/wiki/tutorial/elementary_fitting.md b/wiki/tutorial/elementary_fitting.md new file mode 100644 index 0000000000000000000000000000000000000000..119ce6ca8ea3207431b971ff74aa083a923f9e7f --- /dev/null +++ b/wiki/tutorial/elementary_fitting.md @@ -0,0 +1,47 @@ +Elementary fitting +================== + +*Top:* [Frida handbook](/start)\ +*Up:* [Tutorial](start)\ +*Previous:* [Simple manipulations](Simple manipulations)\ +*Next:* [Fitting with convolution](Fitting with convolution) + +In this tutorial session, we learn how to fit parametrized curves to +given data, and how to extract the fitted parameters. + +#### Basic example + +Let us fit the instrumental resolution `gly180` by a Gaussian: + + command action + ------------------------------------- ----------------------------------------------------------------------------------------------------- + ? \> fl gly180 load data file + 0 \> mr! abs(x)\<2 restrict frequency range (the modified \'!\' means: operate \"in place\", overwrite the input file) + 0 \> p 7 plot spectrum 7 + 0 \> cc p0\*exp(-(t-p1)\^2/2/p2\^2) create the fit curve (will be file 1) + 1 \> a 7 add to plot, with default initial values p0=p1=..=1 + 1 \> cf fit (by default, the curve refers to data file 0) + 1 \> 0,1 p 7 replot data and fit + 0,1 \> 1 cp inform about curve, print parameters + 1 \> oi p2 create file containing p2 vs z0 + 2 \> g4 new lin-lin plot window + 2 \> p plot p2 vs z0 + +Instead of the explicit formula p0\*exp(-(t-p1)\^2/2/p2\^2), one could +use the shorthand p0\*gnn(t-p1,p2), where gnn stands for \"gauss, not +normalized\". The expression gauss(t,p2) is equivalent to the normal +form 1/sqrt(2\*pi)/p2\*exp(-t\^2/2/p2\^2). + +#### Fixing and setting parameters + +Assume file 1 is a curve. + + command action + ---------------- --------------------------------------------------------- + 1 \> cx 2 fix parameter p2 to the current value + 1 \> op2 1.234 set parameter p2 to 1.234 + 1 \> op2 pi/z0 arbitrary expressions may be used + 1 \> cf now fit, with p2 kept fixed at the previously set value + 1 \> cu 2 unfix p2 + +*Last updated for Frida2.3.0c.* diff --git a/wiki/tutorial/fitting_with_convolution.md b/wiki/tutorial/fitting_with_convolution.md new file mode 100644 index 0000000000000000000000000000000000000000..6af23a40ae9962d48b6afda92855db48892e07b6 --- /dev/null +++ b/wiki/tutorial/fitting_with_convolution.md @@ -0,0 +1,51 @@ +Fitting with convolution +======================== + +*Top:* [Frida handbook](/start)\ +*Up:* [Tutorial](start)\ +*Previous:* [Elementary fitting](Elementary fitting)\ +*Next:* \... tell me what most urgently should be put here + +In this tutorial session, we learn how to fit data with a curve computed +as a convolution of a theory and an instrumental resolution. + +#### Basic example + +We choose the data file `gly275` and the resolution file `gly180`. Our +theory is function kwwc(frequency,p1,p2), i.e. the Fourier transform of +the stretched exponential exp(-(time/p1)\^p2). Note a slight risk of +confusion: the dummy variable t will be frequency, not time. + + command action + --------------------------------- ---------------------------------------- + ? \> fl gly180 load resolution file + 0 \> fl gly275 load data file + 1 \> cc p0\*conv(kwwc(t,p1,p2)) create the fit curve (\*1) + 2 \> cf fit without convolution + 2 \> g2 choose lin-log window + 2 \> 1,2 p 7 plot data and fit + 1,2 \> 2 cv 0 let curve file 2 use resolution file 0 + 2 \> cf fit again, now with convolution + 2 \> 0:2 p 7 plot resolution, data and fit + +(\*1) This requires Frida version 2.1.5. In older version, `conv` +required a second argument, indicating a possible shift: +`conv(theory,shift)`. With 2.1.5, this argument has become optional, +with default value 0. + +#### Background, Convolution, Delta-Peak + +The most generic form of a curve is + + p0 + p1*resol(p2) + p3*conv( theory(t,p4,p5,...), p2 ) + +The first term is a flat background. For performance reasons, it should +be computed outside the convolution. + +The second term is a Dirac delta function. When the curve is set to +refer to a resolution file, then resol(p2) is a copy of the resolution +function. Since Frida 2.1.5, the shift parameter p2 is optional, with +default value 0. + +The third term is the convolution, as introduced in the above basic +example. Again, the shift parameter p2 has become optional. diff --git a/wiki/tutorial/simple_manipulations.md b/wiki/tutorial/simple_manipulations.md new file mode 100644 index 0000000000000000000000000000000000000000..aeaf00558e2c89c90037adc801a52f03ae013ffd --- /dev/null +++ b/wiki/tutorial/simple_manipulations.md @@ -0,0 +1,91 @@ +Simple manipulations +==================== + +*Top:* [Frida handbook](/start)\ +*Up:* [Tutorial](start)\ +*Previous:* [Basic usage](Basic usage)\ +*Next:* [Elementary fitting](Elementary fitting) + +In this tutorial session, we will learn some basic data manipulations: +selecting, binning, reorganizing. We use the same test data set as +before (see the [introduction](start) of the tutorial). + +#### A simple manipulation: delete some spectra + +Launch Frida in the directory that contains the test data set, and +perform the following actions one by one. Take your time to understand +Frida\'s reaction to each single command. To do so, use the inspection +and plot commands you learned in session 1. + + command action recommended inspection + ---------------- -------------------------------------------------- ------------------------ + ? \> fl gly180 load one data file + 0 \> msd 0:3 manipulate_spectra_delete: delete spectra 0 to 3 + 1 \> 0:1 dz inspect old and new file + +In general, manipulations leave the original file intact; they act upon +a copy of the file that is stored at the end of the internal memory, +with a new internal file number. In this example, a manipulation on file +0 yielded a new file 1. + +#### Average spectra + + -------------------- ------------------------------------------------------------------------------------------------------------------------------------------------------------------------- + 0:1 \> 1 msa 0,3,6 manipulate_spectra_add: average over groups of spectra: spectra 0 to 2 become new spectrum 0; spectra 3 to 5 become new spectrum 1; spectra 6 to 8 become new spetrum 2 + 2 \> 1 msa ::3 short-hand for the same: ::3 means every third, starting from 0 + -------------------- ------------------------------------------------------------------------------------------------------------------------------------------------------------------------- + +Use dz to see the effect of the binning. Use plot commands to verify +that the averaged spectra have better statistics than the original ones. + +#### Delete files from internal memory + +From now on, directory and plot commands are no longer prescribed in the +listings. Use them on your own to investigate the effects of the +manipulations introduced in the following. Try also variations of the +commands presented here. + +Your activities will create new files in the internal memory. To clean +up, and to restore standard conditions for the next example, use the +deletion command: + + ------------ ----------------------------------------------- + 1: \> fdel deletes all inline files starting with file 1 + ------------ ----------------------------------------------- + +In the following, we always assume that file 0 is gly180 as read from +disk. + +#### Bin channels + + -------------- ----------------------------------------------------------------------------- + 0 \> mpa ::3 manipulate-points-add: within each spectrum, bin three data points into one + -------------- ----------------------------------------------------------------------------- + +Plot input and output to see the improved statistics after binning. + +#### Reduce dimension + + -------------- ----------------------------------------------------------------- + ? \> 0 msa 0 average all spectra into one + ? \> 0 msr 6 manipulate-spectra-retain: delete all spectra except spectrum 6 + -------------- ----------------------------------------------------------------- + +Each of these two commands creates a file that contains just one +spectrum. Use the inspection commands df, dc, dr to see the effect of +this dimension reduction: *z0* is automatically converted into a *rpar*. + +#### Merge files, and save to disk + + ---------------- -------------------------------------------------------- + ? \> : fdel empty the internal memory (or start new Frida session) + ? \> fl gly??? load five test files + 0:4 \> msr 6 retain spectrum 6 of each file + 5:9 \> mfj manipulate-file-join: merge five files into one + 10 \> dz old *rpar* has become *z0* + 10 \> fs save to disk, using the autogenerated file name gly + 10 \> mpa ::3 channel binning, as explained above + 11 \> fso save, overwriting the existing file gly + ---------------- -------------------------------------------------------- + +*Last updated for Frida2.3.0c.* diff --git a/wiki/tutorial/start.md b/wiki/tutorial/start.md new file mode 100644 index 0000000000000000000000000000000000000000..c8ab2ef595a7d67bbf80f4b4c7748c34bd4971d0 --- /dev/null +++ b/wiki/tutorial/start.md @@ -0,0 +1,37 @@ +Tutorial +======== + +*Top:* [Frida handbook](/start)\ +*Previous:* [Installation](/installation)\ +*Continue with:* [Tutorial session 1](basic usage) + +### Usage + +This tutorial explains step by step how to use Frida. It consists of +several independent session listings. The listings contain only user +commands and a few comments. It is intended for interactive use: +**Execute all commands in Frida, and observe the program\'s output.** + +##### Test data + +All tutorial sessions use the same test data set, +<http://apps.jcns.fz-juelich.de/src/frida/frida-testdata1.tgz>. Change +to an empty directory and unpack the tgz archive. The five data files +are from a test measurement at the neutron backscattering spectrometer +[SPHERES](http://apps.jcns.fz-juelich.de/spheres). They contain +incoherent neutron scattering spectra of a glass-forming liquid, +glycerol. Each file corresponds to one sample temperature, as expressed +by the file names gly180 etc. + +##### The tutorial sessions + +The first tutorial sessions should be studied in the given order: + +- [Basic Usage](Basic Usage): Command-line basics, load file, inspect + contents, plot. +- [Simple manipulations](Simple manipulations): Select, Bin, + reorganize data. +- [Elementary fitting](Elementary fitting): Fit a parametric function + to a data set; extract amd manipulate fit parameters. +- [Fitting with convolution](Fitting with convolution): The fit curve + is computed by convoluting a theory with an experimental resolution.