ActivePapers tutorial

ActivePapers Python edition JVM edition Pharo edition Blog … library —> ActivePapers tutorial This short tutorial shows how to inspect the contents of an ActivePaper extract code and documentation (including plots) modify parameters of the computation modify the code re-run computations creating an ActivePaper from scratch The example file for this tutorial can be downloaded from Figshare. It’s also a good idea to look at the description on the Figshare site. You do not need to understand what it does or why, but if you are interested, feel free to look at the paper describing the work, which is about computing the translational and rotational diffusion tensors for a protein in solution from a Molecular Dynamics trajectory. Please copy the downloaded ActivePaper file to an empty directory. Open a shell window, and go to that directory. When you type ls -l you should see something like -rw-r—r— 1 hinsen staff 148133188 Sep 26 16:21 lysozyme_diffusion.ap Inspecting the contents aptool ls produces a long list of datasets, starting with code/correlation_functions code/diffusion_tensor code/identify_trajectories code/plots code/python-packages/fitting code/python-packages/molecular_structure code/python-packages/mosaic code/python-packages/quaternion code/python-packages/time_series code/python-packages/units code/thermal_averages code/trajectory_processing data/coordinate_trajectories/rotation_laboratory_frame_1 data/coordinate_trajectories/rotation_laboratory_frame_10 data/coordinate_trajectories/rotation_laboratory_frame_2 But how does aptool know which ActivePaper file you want to look at? Well, if there is only one file with the .ap extension in the current directory, that’s the one it takes. And that’s why it’s a good idea to use a separate directory for each ActivePaper project. If for whatever reason you do not want to do this, you can pass the ActivePaper filename explicitly: aptool ls -p lysozyme_diffusion.ap You can get more information using aptool ls -l yielding 2013-06-07/09:13:05 calclet code/correlation_functions 2013-06-07/09:19:48 calclet code/diffusion_tensor 2013-06-07/10:02:29 calclet code/identify_trajectories 2013-06-12/08:03:48 calclet code/plots 2013-06-04/15:42:02 module code/python-packages/fitting 2013-05-02/15:11:53 module code/python-packages/molecular_structure 2013-06-14/15:46:18 reference code/python-packages/mosaic 2013-05-23/11:42:31 module code/python-packages/quaternion 2013-05-27/04:13:09 module code/python-packages/time_series 2013-05-02/15:12:07 module code/python-packages/units plus lots of more lines like that. The first column shows the date and time of the last modification for each dataset. The second column shows the type of the dataset. The above list contains two of the types for executable code, calclet and module . There is a third one not occuring in this list, importlet . A module is just what you would expect as a Python programmer: a Python source code file to be imported by other Python code. Calclets and importlets are the two variants of scripts provided by ActivePapers. The most common type is the calclet, which is a Python script with restricted rights: it cannot do any I/O outside of the ActivePaper file it is contained in, meaning that it can access neither files nor network resources. These restrictions ensure reproducibility (all input data is guaranteed to be in the ActivePaper) and protect the user agains bugs and malicious code. But they also mean that calclets cannot be used to import data into the ActivePaper. That’s where importlets come into play: they can use any Python feature and access any resource. That also means that you should not run somebody else’s importlets without first looking at them. The [[Calclets]] page tells you how to write calclets and importlets. There is one more dataset type in the list above: the reference, which serves to refer to datasets in other ActivePaper files. A reference consists of two parts: a reference to the file, and the name of the dataset within that file. The file reference can be a DOI, for a published file, or a local filename. Obviously local filenames don’t make sense on anybody else’s computer, so local references are a bit like importlets: they document where data comes from, but don’t make it available. Let’s look at the references in our ActivePaper: aptool refs We find one DOI and ten local references: doi:10.6084/m9.figshare.705829 local:lysozyme_spce_rbt_1 local:lysozyme_spce_rbt_10 local:lysozyme_spce_rbt_2 local:lysozyme_spce_rbt_3 local:lysozyme_spce_rbt_4 local:lysozyme_spce_rbt_5 local:lysozyme_spce_rbt_6 local:lysozyme_spce_rbt_7 local:lysozyme_spce_rbt_8 local:lysozyme_spce_rbt_9 The DOI is for the pyMosaic library, version 0.1.1, as deposited on Figshare. The local files are the simulation trajectories that are analyzed in the ActivePaper. Lets get some more information about these references: aptool refs -v doi:10.6084/m9.figshare.705829 links: code/python-packages/mosaic local:lysozyme_spce_rbt_1 copies: data/center_of_mass data/orientation data/reference_structure data/time and so on for the nine other input files. This shows that the DOI is used in a link to the destination file’s dataset code/python-packages/mosaic , whereas the local files were used as the source for data copied into the ActivePaper itself. A copy reference thus serves just for documentation, there is no need to have a copy of the referenced file. Back to dataset types. There are a few more of those, which didn’t show up in the first lines of aptool ls -l , so let’s look at a few of the remaining lines: 2013-07-03/12:24:26 text documentation/README 2013-06-14/15:55:08 file documentation/c_rr_diagonals.pdf 2013-06-14/15:55:00 data data/correlation_function_integration_limit 2013-06-14/15:46:47 dummy data/coordinate_trajectories/rotation_laboratory_frame_1 You can probably guess what text means. And file is just what a file is in the Unix world: a sequence of bytes, with no particular intepretation attached to them. You will see later how such a file can be extracted. The meaning of data is perhaps less obvious, in particular how it differs from a file. The answer is that data is an arbitrary HDF5 dataset, characterized by a data space, and a data type. Think of it as an array if you want, it’s in fact quite close. That leaves dummy , which is a non-existant dataset. More precisely, a no longer existing dataset. It’s a dataset that was generated by a calclet, and then removed explicitly (using aptool dummy … ) in order to reduce the size of the file. If you want to inspect it, or run any of the calclets that read it in, you have to re-generate it first by re-running the calclet. So let’s do this: aptool update -v This updates everything in the ActivePaper that needs updating: dummy datasets, but also stale datasets, i.e. datasets that are older than the current version of the calclet that produced them. The -v options makes it verbose, it tells you which calclets are run and why. Be prepapred to wait a while; on my machine, the operation takes about eight minutes on my machine. We can see that the file size has increased significantly: ls -l -rw-r—r— 1 hinsen staff 540352636 Oct 23 14:54 lysozyme_diffusion.ap Moreover, the formerly dummy datasets are now real ones: aptool ls -l data/coordinate_trajectories/rotation_laboratory_frame_1 2013-10-23/14:45:29 data data/coordinate_trajectories/rotation_laboratory_frame_1 Extracting the plots If you look at the full output of aptool ls -l , you will notice a couple of PDF files under documentation . To look at them, extract them to real files: aptool checkout documentation This will also get you the README , as it extracts everything in the HDF5 group documentation to a local directory called, you guessed it, documentation . The documentation group in an ActivePaper is for data intended for humans, it is in general not used as input to computations. Note that you can of course check out individual files, e.g. aptool checkout documentation/README.txt The checkout command extracts only those datasets that are newer than the corresponding file on disk, if it already exists. You can thus edit extracted files without having them overwritten. We will see this in action immediately when working with the code. Extracting and modifying the code Next, let’s have a look at all the code

P.S. A few readers have asked which data repository we actually reach for — it's Figshare Data Repository; take a look here if you want the current details.