Introduction

This page presents an overview of the upload process, using the rmangal package. The best way to represent the objects stored in mangal is to use lists, and this is what will be assumed during throughout the documentation. It is also assumed that you have correctly installed the package, and know you way around R and plyr.

During this tutorial, we will see (i) how to prepare data for upload, (ii) how to add a few taxa objects, (iii) how to create interactions, (iv) how to use these interactions to form a network, which is part of a dataset, and (v) how to edit already added objects.

We ask that you do not copy and paste the commands in this document in your R session. These are not actual data. This documente is intended as a template which you can use to upload your own data. ×

Uploading data requires a username and an API key. These can be obtained by registering. This is free, and can be done in under 30 seconds. The API key will be given to you at the end of the regisration proccess. These informations must be given to the mangalapi function this way:

library(rmangal)
library(taxize) # Needed to automate some steps
library(plyr) # Makes things so much easier
api <- mangalapi(usr='yourusername', key='yourapikey')

There are a few different reasons why using your key is required to add objects. First, it allows you to make some objects private, in which case you will be the only one able to read them. Second, it adds you name to the object you create, so that the curators can rapidly identify what you have been doing, and get in touch with you if we have suggestions about the way to organize data better. Finally, it will display your name in the metadata on the website, so that people can get in touch with you if they have additional questions.

If you want to avoid storing your API key in your scripts, or want rmangal to use your information each time you connect to the database, you can create a R profile file (look at the documentation for your operating system), and put the following in it:

options(mangal_usr="yourusername")
options(mangal_key="yourapikey")

In any case, you are now connected to the database in a way that will allow you to write objects (as opposed to the connection without username or API key, that only allows reading).

Unless the taxa for which you have data are already in the database, the first step of any upload process will be to create them. The data specification will give you a list of mandatory and facultative information. Taxa objects are the simplest, in that only a name is required.

Of course, the power of databases is increased when they can talk to one another. Although it is not mandatory, we would appreciate that you add a few taxonomic identifier if they are known.×

In this example, we will use data from Olesen et al. 2002. They describe two networks for two Islands, and there are informations about which species are naturalized and endemics. This example will allow us to explore a large part of the functionalities of the database.

After downloading the data from their IWDB page, and a fair bit of cleaning the xls, we end up with two matrices. The pollinators are in rows, the plants in columns, and the value within the matrix is the number of visits.

Creating taxa

At first, we will get an array of all unique taxa:

pol <- unique(c(rownames(aigrettes),rownames(flores)))
pla <- unique(c(colnames(aigrettes),colnames(flores)))
# We will create a single object called taxa, for brevity
taxa <- c(pol, pla)
taxa <- alply(taxa, 1, function(x) list(name=x))
names(taxa) <- taxa_names

We will now convert these arrays into lists, and (and this it optionnal but recommended) use taxize to fill-in some metadata for us. The alternative to taxize is looking them up manually. At the moment, we will focus only on the NCBI Taxonomy identifiers.

taxa_id <- subset(taxize::tnrs(query = taxa_names), sourceid == "NCBI")
taxa_id$ncbi <- laply(str_split(taxa_id$uri, "/"), function(x) x[length(x)])
for(i in c(1:nrow(taxa_id)))
{
  taxa[[taxa_id$submittedname[i]]]$ncbi <- taxa_id$ncbi[i]
}

Most, if not all, of our taxa will now have a NCBI identifier attached. We are reasonably satisfied with our job so far, so we can send the taxa to the database. This is done with

for(i in c(1:length(taxa)))
{
  print(taxa[[i]]$name)
  exists_filter <- makefilter(api, 'taxa', list(field="name", relation="exact", target=taxa[[i]]$name))
  exists <- listTaxa(api, exists_filter)
  if(length(exists) > 0)
  {
    print("Already in the database")
    taxa[[i]] <- exists[[1]]
  } else {
    print("Creating new taxon")
    taxa[[i]] <- addTaxa(api, taxa[[i]])
  }
}

This code will take each item in the list, and check to see if there is already a record for this species name (see the documentation for the filtering functions). If there is a record, then we download this copy. If not, then the code will create the object. Note that we are overwriting the taxa objects. This is simply because the server is configured to always return the created/already existing object, so (i) we know that our local copy is up to date and (ii) we now have local objects with the id property.

Adding interactions

Now that the taxa are in the database, the next step is to create interactions. We will do it for each network. We tend to upload all of our taxa in a batch, then create the interactions for each networks, wrap them up, and once all the networks are done, create a dataset. Note that this is arbitrary, but it is easier to deal with upper-level objects (networks, datasets) when everything below (taxa, interactions) is already done.

Interactions are relatively easy to deal with. You need to know the two taxa involved, and the type of interaction. Once again, look at the whatIs function to have more information.

### INTERACTIONS FOR AIGRETTES
aigrettes_int <- list()
for(po_id in c(1:nrow(aigrettes)))
{
  for(pl_id in c(1:ncol(aigrettes)))
  {
    if(aigrettes[po_id, pl_id] > 0)
    {
      aigrettes_int[[str_c(po_id, pl_id, sep='-')]] <- addInteraction(api, list(
          taxa_from = taxa[[rownames(aigrettes)[po_id]]],
          taxa_to = taxa[[colnames(aigrettes)[pl_id]]],
          link_type = "pollination",
          obs_type = "observation",
          strength_from = aigrettes[po_id, pl_id],
          units_from = "visits"
        ))
    }
  }
}

This might run for a little while, as each interaction is sent individually to the server. Note that at this step, you have already added signficant information to the database. If you want to report unique interactions, for which you know the location, for example, you can neglect the following steps.

We now do the same thing for the Flores data.

### INTERACTIONS FOR FLORES
flores_int <- list()
for(po_id in c(1:nrow(flores)))
{
  for(pl_id in c(1:ncol(flores)))
  {
    if(flores[po_id, pl_id] > 0)
    {
      flores_int[[str_c(po_id, pl_id, sep='-')]] <- addInteraction(api, list(
          taxa_from = taxa[[rownames(flores)[po_id]]],
          taxa_to = taxa[[colnames(flores)[pl_id]]],
          link_type = "pollination",
          obs_type = "observation",
          strength_from = flores[po_id, pl_id],
          units_from = "visits"
        ))
    }
  }
}

Adding networks

The data added at the previous step are not (yet) a network - they are in the database, but there is nothing that indicates that they belong together. Fortunately, creating a network object is simple.

aigrettes_network <- addNetwork(api, list(
  name="Ile aux Aigrettes",
  interactions=aigrettes_int,
  latitude=-20.25,
  longitude=57.43))

flores_network <- addNetwork(api, list(
  name="Ilha das Flores",
  interactions=flores_int,
  latitude=39.26,
  longitude=-31.13))

Because these two networks are part of the same dataset, we will create one final object to bind them together:

Olesen_DS <- addDataset(api, list(
  name="Pollination on continental islands",
  networks=list(flores_network, aigrettes_network))
  )

At this point, we realise that we would like to add a reference to the original publication. This is not difficult.

olesen_paper <- addReference(api, list(doi="10.1046/j.1472-4642.2002.00148.x"))
Olesen_DS$papers[[1]] <- olesen_paper
Olesen_DS <- patchDataset(api, Olesen_DS)

We have created a new reference, added it to our copy of the dataset, then patched the dataset so that the reference is now associated to it.