pySolo User Manual

Python is a not-compiled language, meaning that you need to have a working copy of it installed on your computer in order to be able to run PySolo. In particular, the following needs to be installed on your machine in order to be able to run PySolo:

Python >= 2.7

The core of the python interpreter. Latest version is 2.7. Please do not use Python 3.0. If you are using a linux based OS or a MacOS, then most likely you already have python installed on your system.

wxPython >= 2.8

Python bindings to the wxWindows cross-platform toolkit. All PySolo GUI is built using wxWidgets.

Matplotlib >= 1.3

matplotlib is a python 2D plotting library which produces publication quality figures in a variety of hardcopy formats and interactive environments across platforms

numpy >= 1.7

Numerical Python. Provides higher numerical functions.

scipy >= 0.12.0

A supplement to numpy: provides convenient and fast N-dimensional array manipulation

opencv >= 2.4

This is required only if you plan to do video tracking using the pySolo Video extension.

All aforementioned programs should be installed in the order they are listed by a user with administrator privileges. ­More detailed instructions are provided for the three major OS system. Once all this software is properly install on your computer you can then proceed with installing pySolo itself. All listed programs are distributed as open source ­GNU licensed and therefore freely available for download from their respective websites. A documentation galore is also available for all of them. If you are running a windows based OS you can download and install a standing-alone version of pySolo that requires no python installation.

This script will help you collecting the data from the Trikinetics system. This is a step by step guide on how to use it:

1. Go to the TriKinetics webpage, download and install the latest DAMSystem software. As I write the latest version is 3.0.3.
2. Install the software by copying it anywhere on your disk. Configure it the way you wish. pySolo requires you to use a reading frequency of 1 count per minute (1440 bins per day). Take note of the destination folder where your live data are going to be saved. A typical destination folder could be:

   C:\Program Files\DAM Software\DAMSystem3Data

   Note: It is possible to install the TriKinetics software under linux too: just follow this howto.

3. Download the data fetcher script from github or, if you use linux, simply install pySolo and the script will come along. Unzip its content somewhere on your disk.
4. From the commandline, launch the copyfiles.py script. A brand new config file will be created in the same folder with default settings. Alternatively, download a sample configfile from github.
5. Using a normal text editor open the config file and change the data so that they will match your configuration. Everything should be pretty much self explanatory. The script can send daily notification emails: make sure you use your own settings for sending those emails! outputPath is the path to the folder where your day by day data will be stored. inputPath is the folder where the live DAMtrikinetics data reside (see point 2). zipPath is the path where the script will store a zip copy of the data collected day by day. It is use mainly for backup purposes. Finally, it is possible to save files both in channel mode or in monitor mode. The latter is definitely preferred!
6. Save the file you just modified in your home folder or in the same folder where the script is installed
7. Now, every time you run the script called copyfiles.py, the data are taken from the DAMSystem3D folder and transferred, properly formatted, to another destination. You probably want to do this automatically. To do that under WindowsXP go to Start->Accessories->System Tools->Scheduled Taks. Double click on Add Scheduled Task. Hit Next. Hit Browse. Navigate to the location where copyfiles.py is and select it. In the following window select the option “Daily” then click next. Choose when at what time of the day you want to collect the data. This of course should be after zeitgeber 0. If your lights go on at 8:00am, then enter 8:10am. Hit next then your credentials (windows username and password). Hit next a couple of time and you are done.
8. Now everyday your data will be collected and placed in the proper folder ready to be analysed. If for some reason you want to manually collect the data you can start copyfiles manually. You can start it from the command line and specify a given date. Syntax is:

   copyfiles.py –date YYYY-MM-DD

   or

   copyfiles.py –period YYYY-MM-DD/YYYY-MM-DD

   use

   copyfiles.py –help

   to see what other options are available

Remove data monthly

This is another script that should be used with the TriKinetics system. You should use it togheter with the data_fetcher script (see Download section).

1. Download the remove_data_monthly zip file from the pySolo website
2. unzip its content somewhere on your disk
3. open the file using a regular text editor
4. find the line that says inputPath = ‘C:/Program Files/DAM Software/DAMSystem3Data/’ and specify the path of the TriKinetics data there.
5. Execute the file once a month, normally on the second or third day of the month. You may setup and automatic execution if you wish so (see the appendix for the data_fetcher script for instruction on how to do so).

TriKinetics uses a proprietary software to handle data acquisition and storage. The software runs on Windows and MacOSX and can be downloaded from their webpage.

With a little adjustment, the software will run under linux too. Here follows the instruction on how to get it working on Ubuntu 10.04.

1. Install wine from the Ubuntu software center or issuing from the commandline

sudo apt-get install wine1.2

2. Download the DamSystem303.exe.zip file from the TriKinetics website and extract its content somewhere in your home folder or on the Desktop. Data files will subsequently be saved in a subfolder within.

3. Launch the exe file opening it using the “Wine Windows Program Loader”. The GUI will pop up and wine files will then be initialized. Quit the DamSystem program.

4. Plugin the USB Module (PSIU9) and make sure it’s properly detected issuing the command lsusb. IT should show up as following:

 
Bus 005 Device 002: ID 10c4:ea60 Cygnal Integrated Products, Inc. CP210x Composite Device

The device comes with its own USB-to-Serial bridge and should show up as virtual COM port as /dev/ttyUSB0 or similar (if you have other USB virtual serial port the device may appear as ttyUSB1 or ttyUSB2).

5. Using the terminal move to ~/.wine/dosdevices/ and here link the virtual COM port to com1:

 
ln -s /dev/ttyUSB0 ~/.wine/dosdevices/com1

6. Start the TriKinetics software and change the settings so that it will work using COM1.

pySolo analysis is where all the analysis is done. This part of the program is structured in a very expandable way. In the analysis notebook, every tab provides a different functionality and every tab is coded by a separate plugin. It is possible, for the advanced user, to modify existing panels or to create new ones.

Loading the data
Load the date from the File->Open File voice in the menu bar or drag and drop the file to the main window. If you already have opened data in the software, you will be given the possibility to join the newly loaded data to the current ones.

Navigating the data
Navigate the data from the tree panel, on the left side of window. Expand or collapse the tree using the arrow up button in the bottom of the tree panel.

Expand or collapse the tree items

Navigating the tree is straightforward; from here you can select single flies, single days, single monitors or single genotypes, etc, etc. You can also combine multiple selection holding down the shift key on your keyboard or the Ctrl key while you select items on the tree. Or simply use the hold-tree button:

HoldTree button. Hit the button for multiple selections on tree items

Multiple selections done in this way allows you to pull data to be analysed together. NB: Every panel may react in a different way to your selections.

Tagging the data
Using the right mouse button you can tag different items in the tree. Flies, monitors and days can be active as Active (will be considered in the analysis, default selection) or Inactive (flies are not considered in the analysis and treated as if they were dead). Using the browser panel, for instance, you can navigate fly by fly and exclude from your analysis flies that look sick or behave weirdly. Days can also be tagged as baseline (BS), sleep deprivation day (SD), or recovery days (RC).
After having tagged the data, save your work so to mantain the tagging status next time.

Holdplot and multiple comparisons
By default, every new selection on a tree will trigger a new analysis and clear the results of the previous one. If you want to compare multiple flies or days or selections you need to hold the plot by hitting the hold plot button:

Hold the plot and start a new analysis on top of the previous one

when you work in holdplot mode, every selection will be analysed separetely and drawn on a different colour. What if you need to include multiple selections while in holdplot mode? Then you will have to hold the tree as well (see holdtree button above). Hold the tree, select the items on the tree you want to pull toghether and release the holdtree button again.

Using the holdplot mode, you can compare up to 20 different selections. Every new new plot will be drawn using a different colour from the list of colours specified under Options->Graphs->Colours. Colour selection can also be overridden by selecting Tools->Graphs->Use Color.

Defining panels options

There are several options that you can change about the analysis programs and some options that you will be able to change for the single panels. The first ones, are located from the option window (Windows->Options), in the Graph tab and tabs whitin it. They are:

• Default figure size (in inches)
• Graphical interface: interactive, default or non interactive. The interactive graphical interface allows you to zoom in and out from figures.
• Use crosshair: regulates the size and shape of the mouse cursor
• Use standard deviation or standard error: select what kind of error measure should be drawn on the graph. Please notice that not all panels may respond to this setting. To visualize error bars at any moment go to the menu voice Tools->Graphs->Show Error Bars

Also, the system provides a number of panel-specific variables whose value can be manipulated by the user at any time. To have access to them you can either visit the Tools->Options->Custom Panel Options section or more simply hit the options button in the main panel:

Show / hide the options panel.

A sidetree on the right side will appear from which instantaneous manipulation is possible. Also, you will have access to a panel from which it is possible to export the raw data to file.

Interacting with the graphs

Most panels will have a graphical output. It is possible to interact with the graphs and modify what is shown by selecting a specific area in the canvas for zooming in. To zoom out click with the right mouse button and select Zoom Out. Clicking with the right mouse button will also prompt you with several selections regarding the graphs, namely:

• Copy Figure: to copy the content of the figure on the clipboard (can then be pasted into your favourite program)
• Print Figure: send the content of the figure to the printer. This is still experimental and may not work with all OSs.
• Save Figure: save the current canvas as figure image. You can save the output as vectorial image (EPS, PDF, SVG, PS. EMF) or bitmap image (PNG, RAW). In this latter case, images are saved by default at 250dpi but you can change that value in the Graph submenu of the options panel.

Interacting with the data

Every table presented in pySolo also provides full interaction with the user. Click on the column label to sort the data by the order of the values listed in that column. Right click on the table to export data either to the clipboard (copy all, copy current row, copy current column, copy selected) or to export data as CSV file.

Use pySolo database to keep a record of all your experiments and to fetch and elaborate the raw data into DAD files, ready to be open with pySolo analysis.

Entering data

Once the configuration options are properly filled in, you can start filling the database. You can fill in your data using either the listing page or the browser page. The first allows quick insert of regular data, the latter will provide a better definition of the inserted data.
These are the buttons you will use while filling data in the listing table:

	Add a new row to the table
	Remove checked rows from the table
	Search among the currently visible rows using the left mask
	Exit search mode and restore view
	Check/Uncheck all currently visible rows
	Show a calander dialog to pick a date to fill in the left mask
	Apply the values that are filled in the left mask to all the rows checked in the table

In the listing table, values have to be entered in the following order:

1. Start monitor
2. Start channel
3. End monitor
4. End channel
5. Genotype
6. Comment [optional]
7. Start month
8. Start day
9. End month
10. End day
11. Start/end year

with the exception of the comment field, all other fields are mandatory. Hit the tab key of your keyboard to move between fields and to autocomplete data in a smart way (it is easier to try autocompletion than explain it).

The following screenshot exemplifies data entering for channels 1 to 26 in monitor 7, from date 10/5/2008 to 10/15/2008

If our flies span multiple but consecutive monitors, we can use the following notation (in the next example we will collect from channel 10 of monitor 7 to channel 32 of monitor 8 )

If we have data spanning multiple but not consecutive monitors we can then use the browser panel. The browser panel can also be used if we are collecting data that span two consecutive years.

Alternatively, we can use the proper data formatting directly in the listing panel (in fact, what the browser panel does, is just help you format the data for you).

Tagging data

Every row can be associated to one or more tags. To associate a new tag to a row or group of rows, type the tag name in the lower box on the left bottom side. Enter a new name to create a new tag or type an existing name to add the row to an existing group. Then check the boxes of all the rows you want to tag and press the add tag button.

	Add the tag to the checked item
	Remove the tag from the checked items

Now your rows have been tagged. To navigate through tagged rows, use the tag list box on the left. Use of tags allows you to keep all your data in the same file and yet being able to navigate effortlessly between different experiments.

Saving or exporting

To save your work, simply go to File and Save File (or Save As).
You can also export the contents of the table either from File->Export to CSV File or by clicking with the right mouse button on the table itself and selecting the kind of export you wish to obtain.

Loading or importing

Recently loaded data are shown in under Open Recent Files in the File menu. If you open a file after having some data already in the table you will be asked if you want to join data into a single table. Files can also be opened by simple drag and dropping.
You can also import data directly from another spreadsheet (eg.Excel or OpenOffice) using the CSV format (comma separated values). The format of CSV files should be as follows:

0,0,SM,SCh,EM,ECh,Genotype,Comment,MM,DD,MM,DD,YYYY,Tags

Fields are separated by commas, tags are separeted by semicolumns. See also the example CSV file provided in the download sections.

Collecting RAW data

So, you inserted your data, tagged them properly and saved the file. It’s time to start the analysis.
Check the boxes of the rows you wish to analyse, then go to Analysis->Check Raw Data Files to make sure that all the files needed for the analysis are present and accessible; then, proceed with the actual data collection (Analysis->Fetch Raw Data). This may take from few seconds to few minutes (normally about 30 seconds), depending on the number of flies to be analysed and how fast is the computer you are using. After the data collection, you will be prompted on where to save the resulting file. To send data to pySolo_anal, select Analysis->Send Data to Analysis from the menu bar.

Entering data: an example.

Let’s say we want to collect data from monitor 7, channels 1 to 26, from the 5th to the 15th of October 2008. You should proceed in the following way:

1. Open pysolo. The database window will appear first with an empty table
2. If you want to add your entry to an existing file, open it from the File menu.
3. Hit the add row button (red plus sign, bottom right panel) to add a new row to the end of the table.
4. Enter the data in the following order: 7 (the starting monitor), 1 (starting channel), 7 (the ending monitor), 26 (the ending channel), Name of your genotype, possible comment, starting month, starting day, ending month, ending day, year. (for our dates would be 10, 5, 10, 15, 2008).
5. Save your work. Now the new entry has been successfully inserted in the database.

Now, try to do the same for monitor 10, channels 1 to 32 – same date using auto completion features:

1. Hit the plus button again
2. Enter the number of your monitor (10). Hit the tab key on your keyboard 4 times. The remaining values will be filled for you. Continue manually with genotype. Date also has a auto-completion feature. Try hitting the tab key to see how it works (today’s date will be filled both in start and end fields)

Now, let’s say you realized both rows have a mistake in the date.

1. Check the box next to the rows in which you want to make the correction
2. Move to the general mask, at the very bottom of the window frame. Enter the correct values for the date. You can use the calendar icon to pick dates from a calendar pop-up window
3. When happy with your new dates hit the icon with a green arrow pointing up. Corrections will be inserted in all checked rows.

Finally, you want to fetch and compile the data.

1. Check the box next to the rows you want to compile the data for.
2. Go to “Analysis -> Check Raw Data files” to make sure the raw data are available and accessible
3. Now select “Analysis -> Fetch Data”. Data fetching will start and after a few seconds you will be asked were to save the compiled file. Save it somewhere or hit cancel if you don’t wish to do so.
4. Go to “Analysis -> Send Data to Analysis” to view the compiled file in the analysis window.

The first time you will run pySolo, the program will detect that no preference file exists and will prompt you with the Option Panel. From the option panel you can modify all the configuration parameters that pySolo needs to run. The preferences are then stored in a file called pysolo.opt, located in the pySolo folder or your home folder. The pysolo.opt file will be used by default unless pySolo is launched from the command line using the optional parameter -c, for instance:

p­ysolo_anal.py -c alternative_options.opt

will start the program loading the alternative_options.opt file as preference file.

Preferences in Detail.

The essential data you need to fill in are in the first window, under “Files and Folder”. They are:

• the DAM Folder
• the Output Folder
• the extension of the input files
• the RAW Data structure

The option panel

The DAM folder is the folder where the raw DAM data are located. Within that folder, data need to be structured in a precise way, namely they have to be in a subtree of folders following the rule: 

C:\DAMfolder\yyyy\mm\mmdd\

So, for instance files for July 17th 2008 will be in the folder

C:\DAMfolder\2008\07\0717\

Note that only the DAMfolder path will have to be specified in the option window: the program will then be able to pick the files it needs on the basis of the subfolder structure. Filenames within the folder also need to have a specific structure in order to be recognized by pySolo. This is discussed at the end of this session.

The Output folder is the folder where the data generated by pySolo are stored. These are both the data generated by the database program and the ones generated for the analysis. Files generated by pySolo are of two kinds: database ones, with extension .dal and analysis ones with extension .dad . These latter are heavily compressed files (using the zip algorithm) and they actually contain data worth 50x their size on the disk. Thus, 50mb worth of analysis will usually fit in a 1mb big dad file. You can locate the output folder wherever you wish on your hard drive. We recommend using a service like Dropbox to save your file on the cloud, so that they would be accessible from any other computer of yours.

The extension of the input files is txt by default and most likely you will not need to change it.

The RAW data structure refers, as the name suggests, to the structure of the DAM data that are to be analysed. Five options are available:

• Trikinetics – Channel Files
• Trikinetics – Monitor Files
• pySolo Video – Distance
• pySolo Video – Virtual Beam Split
• pySolo Video – Raw Data

If you have TriKinetics monitors and data, use Channel if your data are stored in 1 file = 1 fly fashion and use Monitor if your data are stored in a 1 file = 1 monitor fashion. You can download an example of these files in the donwload page.

If you use pySolo Video,  you should specify here if the activity is calculated using the “distance” algorithm (i.e.: measuring the distance the flies walk every minute) or the “vbs” algorithm (i.e.: a de facto virtualization of the TriKinetics virtual beam split)

Name of the input files. 

The filename of the input file is important. In Channel files mode each file should have a name composed by: month number (2 digits), day number (2 digits), the letter M, the monitor number (3 digits), C, the channel number (2 digits). For instance channel 1 of monitor 5 on 17 of July will have the following name.

0717M005C01.txt

In monitor mode the file name will be similarly structured but without the reference to the channel number of course. So,

0717M005.txt

will contain data about all channels for day 07/17, monitor 5. If you use the TriKinetics system, the easiest way to achieve a proper formatting without hassle is to use the datafetcher provided with pySolo. See here for details about it.

Starting the program using the binary/prepackaged version

If you installed pySolo using the prepackaged version, just run it by selecting the program icon from the start menu of your OS. In Ubuntu, pySolo and pySolo video will be both under the “Science” subfolder of your start menu.

Starting the program using the development version

To run pySolo just navigate on the folder and double click on the right file:

pysolo.py                       To start the complete program.
pysolo_anal.py                  To start only the analysis part of pySolo.
pysolo_db.py                    To start only the database.

If you work from the command line, you can also provide some specific options to pysolo_anal.py. For instance:

pysolo_anal.py -o Data/test.dad

will open the file test.dad in the folder Data

pysolo_anal.py -c conf_pysolo.opt

will use the configuration file conf_pysolo.opt instead of the default one.

Two version of pySolo are downloadable at any time: a development version, also known as nightly build, and a stable version. The development version is latest available, updated daily or weekly and it does require python to run. The stable version is updated every now and then. It comes as pre-compiled binary for windows or as deb package for Ubuntu linux: installation of the precompiled packages on Windows or Ubuntu linux is straight forward but adoption of the development version requires a bit more work.

Installing latest stable version

Ubuntu Linux [no longer supported!]

Ubuntu is no longer supported. Maintaining packages for Ubuntu is way too much work for me. If you want to volunteer and do it on my behalf, please write me a note.

Archlinux

This is what we use. Just install the package from AUR (see Download)

Microsoft Windows

On Windows machines it is possible to install a full version of pySolo as compiled binary. In this case just proceed downloading the binary file (setup.exe) from the Download page and execute it as Administrator on your machine. The binary version will work out of the box as it does not require Python or any other library to run.

Installing the development version [recommended!]

PySolo has been written entirely in python using a wxWidget platform independent library for construction of the GUI. For this reason, it can run on a multitude of operative systems (all Microsoft Windows, most Unix or unix-like systems, and Macintosh OS X)

PySolo has been successfully tested on the following systems:

• Windows based OS (2000/XP/Vista/7)
• MacOSX 10.4.11 and 10.5.7
• Linux Ubuntu >= 7.04 and Archlinux

Python is a not-compiled language, meaning that you need to have a working copy of it installed on your computer in order to be able to run PySolo. Python and the accessories libraries can be a great resource for your scientific computing, beside giving you the possibility to run pySolo. See the Using Python for Science page in the pySolo developer manual for a comprehensive list of useful links.

Installing the source (development version) or installing on linux systems different from Ubuntu

Installation of pySolo on most linux platforms is straightforward as all required python packages and dependencies are available as .deb (ubuntu, debian) or .rpm (redhat) packages.

On Ubuntu start the Synaptic Package Manager and download the following packages:

• python2.7
• python-numpy
• python-scipy
• python-matplotlib
• python-wxgtk

On Debian/Ubuntu you can use apt-get from a terminal to download them without using synaptic. After all the dependencies are satisfied you can proceed downloading python source from git.­ To install from git open your shell (terminal), move to the folder where you want pySolo to be installed and issue the command:

git clone git@github.com:ggilestro/pySolo.git

Installing the development version on Windows and MacOS

The easiest way to install the developmental version on Windows and MacOS machines is to rely on a all inclusive scientific python distribution, like anaconda. The lighter version miniconda is also a good alternative.

Note: if your user does not have administrator rights or if you are using Vista you might have to install all aforementioned packages as administrator. In Vista right click on the file and select “Run as Administrator”.