Table
Tables are the basical metadata system in Dynamo. They are just matrices that describe the properties of sets of particles. The property coded on each column is fixed.
A table can be seen as the metadata of a given data folder. The first column in a table is just an integer number (which we call a tag) that identifies a particle. In the data folder, this particle is identified by the name of the file that contains it, i.e.
<myData>/particle_00003.em
would be the particle with tag 3 in the data folder myData. A table will be able to operate on this particle if it has a row whose first column is the integer 3.
As Tables are matrices, Matlab tools (like find,plot,hist) are easily customizable to explore and visualize the contents of a table. Additionally, Dynamo includes many specific utilities. dapropos table will give you a list of basic commands that you can use for table manipulation.
Contents
Uses of table objects
Inside Dynamo, tables appear in mainly three scenarios. The [#Format convention in a table|formatting convention] is however always the same.
Initial metadata for alignment projects
Assumed to code the initial shifts and angles of the particles (if known a priori)
Results of alignment iterations
They are automatically produced by Dynamo at the end of each iteration of a project.
Extracting particles from volumes
- Main article: Cropping table
Any table can be used to extract particles from a volume or set of volumes. The spacial coordinates need to be coded in columns 24 to 26. The same table can be used for several volumes when used jointly with a volume-table mapping file.
Format convention in a table
- Main article: Table column convention
Rows correspond to particles, columns correspond to predefined properties. The convention used in the definition of each column of the table can be consulted each time by the command dthelp
1 : tag tag of particle file in data folder 2 : aligned value 1: marks the particle for alignment 3 : averaged value 1: the particle was included in the average 4 : dx x shift from center (in pixels) 5 : dy y shift from center (in pixels) 6 : dz z shift from center (in pixels) 7 : tdrot euler angle (rotation around z, in degrees) 8 : tilt euler angle (rotation around new x, in degrees) 9 : narot euler angle (rotation around new z, in degrees) 10 : cc Cross correlation coefficient 11 : cc2 Cross correlation coefficient after thresholding II 12 : cpu processor that aligned the particle 13 : ftype 0: full range; 1: tilt around y ( ... dhelp dtutorial for more options) 14 : ymintilt minimum angle in the tilt series around tilt axis (i.e. -60) 15 : ymaxtilt maximum angle in the tilt series around tilt axis (i.e. 60) 16 : xmintilt minimum angle in the tilt series around second tilt axis (i.e. -60) 17 : xmaxtilt maximum angle in the tilt series around second x (i.e. 60) 18 : fs1 free parameter for fourier sampling #1() 19 : fs2 free parameter for fourier sampling #2() 20 : tomo tomogram number 21 : reg for arbitrary assignations of regions inside tomograms 22 : class class number 23 : annotation use this field for assigning arbitrary labels 24 : x x coordinate in original volume 25 : y y coordinate in original volume 26 : z z coordinate in original volume 27 : dshift norm of shift vector 28 : daxis difference in axis orientation 29 : dnarot difference in narot 30 : dcc difference in CC 31 : otag original tag (subboxing) 32 : npar number of added particles (compactions) / subunit label (subboxing) 34 : ref reference. Used in multireference projects 35 : sref subreference (i.e. generated by Dynamo PCA) 36 : apix angstrom per pixel 37 : def defocus (in micron) 41 : eig1 "eigencoefficient" #1 42 : eig2 "eigencoefficient" #2 more columns can be added for user-defined properties.
Reading and writing tables
Tables are customarily stored as text files with extension '.tbl'.
Tables can be read and write into the matlab worspace with the standard commands dwrite and dread.
For specially long tables, you can use the extension '.tbl', which will write the files in binary format, allowing considerable speedup in I/O.
Visualization of tables
Basic info
A summary of contents in a table can be displayed by dtinfo. Typing the command:
dinfo mytable.tbl
on a table file mytable.tbl you will get the mean and std values for the contents of each column of the table.
Plots through command line
dtplot allows the creation of graphs like: positions of particles in a volume, orientations, distributions of orientations. For instace, you can check if all the particles in a reference (column = 34) have a similar missing wedge orientation (the tilt angle is coded in column 8).
For instance:
dtplot example.tbl -pf oriented_positions
shows the locations and orientations of all the particles in a table.
Basic plots through GUI
The basic GUI is dtshow. Typing:
dtshow file.tbl
for a file containing a table, or
dtshow(myTable)
for a table defined in the workspace, opens a lightweight GUI that allows quick plotting of the contents of the table, and also a GUI access to all the plotting options of dtplot. This GUI is one of options that the dcp GUI offers when exploring the results of an alignment through browsing the generated tables.
GUI for complex plots dtview
dtview is a more complicated browser for table contents, allowing the joint visualization of pairs of properties for particle sets inside a table.
Table manipulation
Creating tables
In general, Dynamo itself creates correctly formatted tables as they are needed. If a table needs to be explicitly created, an easy way to create a blank table compatible with a given data folder is with the command dynamo_table_blank.
Selection of parts of a table
Use dtgrep. It's applicability ranges from just selecting an explicitely defined range of particle tags to the computation of particles pointing in a direction.
Merging tables
Low level manipulation
Tables are just matrices. As such, they can be easily merged with simple Matlab manipulation tools. For instance, to merge the tables in files table file 1 and table file 2 and set the result into file merged table file, you just read the contents into the workspace and use the cat commnad for matrix concatenation:
t1 = dread(<table file 1>); t2 = dread(<table file 2>); tMerged = cat(1,t1,t2); % concatenate rows of t1 and t2 dwrite(tMerged,<merged table file>);
When you merge tables manually, please make sure than the merged tables do not have overlapping tags (column 1).
You can check this by displaying the result of the operation.
length(unique(tMerged(:,1)))
Merging commands
Dynamo also includes a dynamo_table_merge command, available from Matalb or the standalone
Conversion tools for other software packages
dapropos compatibility will print a selection of functions that can be used to import and export particle metadata with the format expected by other packages, as AV3, jsubtomo and XMIPP