Despite the importance of using phenomic (morphological, behavioral, etc…) data for phylogenetic tree-building, scientists have often found phenomic data collection to be a slow process. The Evolution Project is an NSF-sponsored experiment by a team of scientists to make data collection go faster using crowdsourcing.
The Evolution Project operates in tandem with MorphoBank, the NSF sponsored phylogenetics databasing project. As a result, MorphoBank is the platform for scientists to stage a crowdsourcing project. If you’re unfamiliar with MorphoBank, you can visit http://morphobank.org/index.php/Documentation/Index for a detailed how-to guide. For the purposes of this project, we’ll briefly outline below how to set up a MorphoBank project and tie it to the Evolution Project.
PLEASE NOTE: This software is live in beta version for the public to use on selected MorphoBank projects associated with NSF's Assembling, Visualizing, and Analyzing the Tree of Life program. If you are a scientist and want to use it - use the Contact form and we will enable the tools for you.
Creating a project on MorphoBank and sharing it on the Evolution Project:
The Project Workspace
To create a new project, login to MorphoBank and navigate to the My Projects section. You will see an orange "Create New Project" button at the top of the page. This will display a New Project form, which includes fields for project title and abstract, publication information, etc.
"Create new project" Button
After you’ve laid this groundwork for your project, you can begin populating it with data. Remember, you can edit these data at any time.
Setting up a Matrix
Once you’ve set-up the basic details of your MorphoBank Project, you can load a matrix in Nexus or TNT format, or create a matrix by hand in the Matrices tab of the project workspace. The axes of a matrix are composed of characters and taxa. In a standard MorphoBank project, the cells of the matrix contain character state scores and, if you like, notes, citations and media that can be labeled. In MorphoBank this information is entered by a team of scientists that decides to collaborate.
By contrast, the Evolution Project is designed to save scientists time by letting a crowd score the matrix using Media (images), see further explanation below. In order to do this, the scientist only needs to define each character and state with at least one labeled Media. Each of the Media used should have a designated “View” (orientation). Media for the taxonomic sample can then be loaded to MorphoBank in batches to save time. The autopopulate tool in MorphoBank, when executed, will load Media into the cells of the matrix so the scientist does not have to do it. Now the project is ready to be crowdsourced.
The Evolution Project gives the crowd Tasks to do. For the purposes of the Evolution Project, the character column (outlined below in red) is the axis along which “Tasks” are defined. Media must be added to each cell that a scientist wishes to crowdsource, otherwise the crowd will have no information upon which to base a cell score.
Matrix Editor with Character Column Highlighted
A scientist may wish to crowdsource only part of his or her project. To do this create and use a partition of your matrix, rather than the entire dataset. Partitions are created within the matrix editor by clicking the “partitions” button along the top row of the matrix editor, as seen below. You can then create a name for your partition and drag and drop characters and taxa to add or remove them. Use this tool to choose and isolate those characters you wish to crowdsource in the Evolution Project.
"Partitions" Button in Matrix Editor
Partition Editor – Drag and Drop Characters and Taxa
Your project workspace has a single "directory" of taxonomic names that can be used to identify specimens (and by extension media) or populate a matrix. You may load your taxa by hand, one at a time, using the taxon entry forms or in a batch process using a tab-delimited file from a database, Microsoft Excel or a similar application. The taxa directory will automatically be populated if you load a matrix from a Nexus or TNT file. If you are working with extinct taxa, please remember to mark them as such using the “is extinct” checkbox for an individual taxon, or the Extinct Taxa Batch Editing tool for larger sets. You can add, delete or edit Taxa at any time and the Evolution Project will update accordingly.
As with taxa, your project has a list of characters and their associated states. Each distinct character is represented once, and only once, in your project's character list. If the character is changed in one place, the change will propagate to all other places where the character is used.
Unlike taxa and other deposited data, characters cannot exist independently of matrices. That is, to exist in your project at least one matrix must reference the character. As a corollary to that, characters are edited through the matrix itself, rather than through a separate side tab like taxa. The character editor is accessed through the matrix by clicking “chars” along the top row of buttons. The character editor is also where you can prepare characters for crowdsourcing by adding states, descriptions and media (or you can do this directly in the matrix editor). A character may have as many states as required. Each state has an associated name and number, as well as any number of exemplar images. As we’ll discuss in a moment, it is crucial to include exemplar images if you plan to crowdsource your project, as images are the entry point for users to interact with your data.
Character Button in Matrix Editor
Character List Editor
To edit an individual character (to add description, states, media, etc.) double click on it in this editor.
To further prepare your characters for the Evolution Project, you should take advantage of MorphoBank’s ontology tool. Concepts in a Morphobank ontology are rules linking characters. For example, when a mother character is set to a specific state (say, “wing absent”) then the states of dependent characters (“wing blue”) can be automatically forced to another value (in this example, the forced value would be “inapplicable”). Well-designed rule sets mean that the crowd will not be presented with confusing options.
To create an ontology, click the “ontology” button in your matrix editor:
After clicking on the "ontology" button you should see a Character ontology window like the one shown below. This window allows you to attach rules to any character in the currently opened matrix.
To attach a rule, first select the rule type (i.e. 'Set State' Rules or Media Rules) on the tab bar along the top of the window, then click on the "Add rule" button. For "set state" rules you will see a window like the one below. To establish a "set state" rule select the source character and state from the "When setting this character to this state..." lists. You can only select a single source character and state for each rule you create. Then select the characters for which you wish to set scores and the state with which to score. You can select a range of characters by clicking on the first character and shift-clicking (clicking the mouse while the shift-key is depressed) on the last character in the range.
"Set State" Rules
For "media" rules the "add" window will appear like the one below. As with "set state" rules, you first select the source character and then select one or more target characters by clicking and/or shift-clicking. Media rules can always target multiple characters.
To take advantage of the media rules, of course, your project will need to include media.
The ability to append media to your project is an important aspect of MorphoBank and a vital aspect of the Evolution Project. Without media to illustrate the characters you wish to score via crowdsourcing, your project participants will have no idea where to begin. As a result, character states not illustrated by media will not be visible in the Evolution Project interface. You can upload any number of media to MorphoBank. Initially, media should be uploaded to the “Media” tab (which includes time-saving batch tools), but these images must then be associated with characters using the character editor mentioned above. To add media to a character enter the character editor, select the “media” tab, and select “add media” at the bottom of the window to associate an image in your project with a particular character state.
As mentioned above it is particularly helpful if your Media also have a specified View (orientation). Often a View applies to many Media. You must create Views by navigating to the “Views for Media” tab on the left-hand navigation of your project workspace. From here you can define all views that may be relevant to your project media, and then connect them to as many images as you wish.
Media Tab in Character Editor
Once you’ve created a MorphoBank Project, you can begin to participate in The Evolution Project’s crowdsourcing initiative by sharing selected data through the The Evolution Project interface. Because The Evolution Project is still in beta testing (as of 2013), you must contact a MorphoBank administrator to enable crowdsourcing controls on your MorphoBank project. Email firstname.lastname@example.org for more information. Once the software is more mature this will be more widely released.
When a MorphoBank administrator has enabled crowdsourcing controls in your project, you will have access to a “Crowdsource dashboard” from the right hand side of the Project Overview page. Follow the "Crowdsource dashboard" link and press "Click to Plan your Crowdsourcing" to turn on crowdsourcing tools throughout your project. The dashboard allows you to keep track of the aspects of your project that have been marked for crowdsourcing and maintain the crowdsource specific title, description, contact information and status of your project, but you will not make many important changes from this screen. Instead, the dashboard will provide links to the pertinent editors.
Once you press "Click to Plan your Crowdsourcing", at the top of the dashboard Project Administrators will see a form to enter basic information about the project for use on TheEvolutionProject.org. Fill out the fields “TheEvolutionProject.org title” and “TheEvolutionProject.org description” as you would like them to appear on the site. Enter the email address of a project contact person in the field labeled "TheEvolutionProject.org project contact email". While in this initial planning phase, keep the status of your project set to "Not Live" so it is not available to the general public on TheEvolutionProject.org. The information on this form can also be entered by Project Administrators on the Project Info form accesses from a link on the right side of the Project Overview page.
Your dashboard will also display the names and bios of project members (which users can maintain by clicking “Profile” on the upper-right side of your screen) as well as the matrices/partitions marked for crowdsourcing, the relevant taxa, and the characters (also known as tasks):
Portion of Evolution Project Dashboard
You can crowdsource an entire matrix, or an individual partition in your matrix. Both options are provided in the matrix editor's "Crowd" tab. Only active partitions containing both characters and taxa can be crowdsourced.
Characters are the axis on which crowdsourcing tasks will be defined. Evolution Project users are “tasked” with scoring cells in your matrix, so each character column is essentially a “task” with x number of taxa to score before it is completed. Let’s look at how characters are prepared for the Evolution Project from within their editing screens. You can enter crowdsource specific information for characters within the matrix editor's character editor, accessed by clicking the "Chars" button at the top of the matrix editor or clicking on a character in your matrix. Within the tab labeled “Character” you can determine how the character, and therefore your project's task, is perceived on the Evolution Project site without compromising your matrix as it appears in MorphoBank. For example, you can use this space to write a user-friendly name and description for your character, which may be more appropriate than your internal name and notes. This is what will appear as a “task” for the Evolution Project user. You can also designate the difficulty-level of the task – how experienced does a user need to be to score this cell? Do you want to lock this task so that only more experienced users can access it? Choose beginner, intermediate, or advanced.
You can enter simplified text to describe character states in the column labeled "TheEvolutionProject.org alternate state name" in the matrix character editor. When available, this text will appear in place of the character state name on the scoring tool. This way you can use your prefered character state names in your MorphoBank project, while still offering user friendly text to the crowd.
Enter User Friendly Character State Text in the Matrix Character Editor
Provide Task Guidance with Character Images
Images associated with the character, but not a specific character state, will appear in The Evolution Project on the right side bar with the Task Info. This is a good place to put general explanatory diagrams that might be useful for the character as a whole, or even several characters. This image can be used in combination with the task description to provide task/character specific guidance to crowdsource users. For example, you can upload an image illustrating the anatomical terms used in your character states. You can choose to use MorphoBank’s image labeling tool or use image editing software like Photoshop to apply labels prior to upload.
Before you release your project to be crowdsourced, you should take a moment to preview your project on TheEvolutionProject.org to make sure your tasks are configured properly. Login to TheEvolutionProject.org with your Morphobank username and password and navigate to the Projects section of the site. At the bottom of the list of public projects you will see your "Not Live" project available for preview.
You can tell you are previewing a project by the orange box and warning messages on the screen. Select a task to preview how it will be presented to site users. You can use the next and previous navigation at the top of the scoring tool to click through the screens in your task. Scores will not be recorded in preview mode.
Preview a Task
Once you have previewed your project and determined you are ready to make your project available for crowdsourcing, the project administrator can use the "Status" drop down on the Crowdsource Dashboard to make the project live. You have the choice of making the project accessible to the general public by selecting the option "live on TheEvolutionProject.org" or making your project available by invitation only by selecting "live/ available by invitation only." When you select to make your project available by invitation only, you will be prompted to enter the email addresses of the users you would like to grant access to your project. If the individuals are not yet members of the MorphoBank Community, accounts will be created for them.
Configuring Invitation Only Projects
Once your matrix or partition has been crowdsourced, it will appear on the Evolution Project site, and users will be able to log in and complete tasks. As they score more taxa, users will rise in rank and have access to those tasks deemed more advanced by project administrators like you.
You may want to distribute a link to crowdsource your project to a specific group over a clear period of time and be able to analize these crowdsourced scores separately from others contributed to your project. For example, you might want to develop a student project around your matrix, or you may have a dedicated group of volunteers lined up for an experiment. In other words, you might want to create an Evolution Project “Event.” To create a focused crowdsourcing event with a clear intended audience, first navigate to your project’s Evolution Project Dashboard on MorphoBank. Scroll down, and click “New CS Event” under the “Project Crowdsource Events” heading. Enter a Name for public display, notes for private use, choose an intended audience, and specify the date of the event:
After you press “Save,” You’ll see a summary of the event and a link to join the project. Distribute this link to your experiment participants.
When they follow the link, they will be prompted to login or register. Once logged in, they will have access to your project and your project only. Any scores contributed under the auspices of this link will be associated with your crowdsource event. To see other EP projects, users must first log-out.
You will be able to review the scores contributed through the event as a spreadsheet or in the matrix editor.
You can evaluate the scores contributed through TheEvolutionProject.org in the matrix editor by selecting the option to show "Crowdsourcing Results." Each cell with crowd contributed scores will contain a list of the states selected and the number of time they were selected by users. You can see a breakdown of contributed scores by user by clicking on the cell and selecting the "Crowdsource" tab in the Cell Data Editor. This is also where you can see comments entered if a taxa was skipped during the scoring process. Highlight "cells with crowdsourced scores" to make cells with crowd contributed scores stand out in your matrix.
You can evaluate the scores contributed through TheEvolutionProject.org in the Crowdsource Dashboard by navigating to the Results tab. This will display a scrollable grid of characters and taxa containing a summary of the scores users have contributed, as well as any scores entered in MorphoBank. There is also a tool to highlight scores when users are over a selected percentage of agreement.
You can also export your crowdsourced scores as a spreadsheet for analysis outside of MorphoBank using the "Export crowdsource scores" button at the top of the Crowdsource Dashboard. If you have crowdsourced scores through crowdsource events, you can download the scores linked to an event by using the "Export event scores" link in the Project Crowdsource Events section of the Crowdsource Dashboard.