Zotero + Word Tutorial

I mentioned Zotero in my first academic workflow post. Zotero is a great citation manager with many great features. And if you write your documents in Word, Zotero is a no brainer. A Zotero plug-in for Word lets you insert citations from your Zotero database in your Word documents, and then Zotero automatically formats in-text citations and the reference list at the end of the document. Plus, it’s free!

Most students, whether graduate students or undergrads, use Word to write their school essays, term papers, and manuscripts, so Zotero is a great, free option for easily handling citations in these documents.

In this post, I provide a tutorial for getting citations into Zotero and getting citations from Zotero into your Word documents. At the end of the post, I provide links to some additional tutorials (including screencasts) on installing and using Zotero.

Installation

Basic Installation

You can download and install Zotero from this page. There are two ways to install Zotero:

• Standalone
• Zotero for Firefox

The stand-alone option installs Zotero like any other application that runs on your computer. Note that if you install the stand-alone version, you should also install the plug-in for whichever browser you use when you do academic research (Firefox, Chrome, or Safari). Using a browser with Zotero makes it much easier to get citations into your reference database. Install the browser add-on after installing the stand-alone version.

If you install Zotero for Firefox, you will need to have Firefox open to use Zotero (including putting references into Zotero and citing references from Zotero).

Zotero Plug-in for Word

You definitely want to install the plug-in for Word so that you can create citations automatically in Word documents. If you install the stand-alone version of Zotero, the Word plug-in is included. If you install Zotero for Firefox, you need to manually add the Word plug-in (available here).

Getting Sources into Your Reference Library

There are four main ways to get citation information into your Zotero database. I give step-by-step instructions here plus a few exercises, so you can try them out.

Web Page with Embedded Metadata

The browser that is integrated with Zotero (e.g., Firefox) can quickly and easily import sources into the Zotero library. Many web pages embed metadata into the web page itself (e.g., the web pages for academic journal articles hosted on ScienceDirect). Zotero can recognize this information and import it.

Other useful websites with embedded metadata include Amazon, Google Books, and most newspapers. So, for example, when I have a hard copy of a book, I often look it up on Amazon and grab the citation information from there, so I don’t have to type it in manually.

Exercise

Open the web browser that is integrated with Zotero, and go to this random academic journal article hosted on ScienceDirect: http://www.sciencedirect.com/science/article/pii/S1053811902913303. (Note that it does not matter if you have access to the full text of the article.) Look at the right-hand portion of the address bar of your web browser. There should be a small icon that looks like a single piece of paper. If you click on that icon, Zotero will automatically import the citation into your reference library. It’s hard to imagine how it could be any easier.

External Citation Files

Citation information can also exist in external files. I usually only resort to importing from external files when importing from metadata embedded in a web page does not work. There are several citation file types (with extensions, such as *.bib and *.rdf). Fortunately, their details don’t really matter. Zotero should be able to import all of the ones you are likely to encounter. You can usually download these files from academic journal article databases (such as ScienceDirect and Google Scholar).

Exercise

Search for a famous journal article called “The Tragedy of the Commons” on Google Scholar. (Or click this link to navigate straight to the search results.) When I wrote this post, the second result was the correct article (by Garrett Hardin). Click on the link titled “Cite”. A dialogue will pop up with text that you can copy and paste for MLA, APA, and Chicago-style citations. Click on any of the links at the bottom of the dialogue: BibTeX, EndNote, RefMan, or RefWorks. The format doesn’t really matter, but RefMan is Zotero’s native format, so I recommend clicking that one. Another dialogue will pop up asking you where to download the file on your hard drive. Select a location, and download the file. (If you clicked on the RefMan link, the file will have a *.ris extension.)

Next, open Zotero, click on the gear button (which should reveal a drop-down menu containing items like “Import ...” and “Import from clipboard”). Click on the item labeled “Import ...”. Zotero will pop up a dialogue, so you can select the file from your hard drive. Do so, and Zotero will import the citation information for “The Tragedy of the Commons” by Garrett Hardin into your library.

PDF with Embedded Metadata

Just like web pages, PDFs of recent academic journal articles often contain embedded metadata. Zotero can extract this data and write a new entry in the Zotero database for the PDF. Note that this usually only works with PDFs of relatively recent academic journal articles, but you can try it with any PDF.

Exercise

Download the full text of the first result from the Google Scholar search in the previous exercise. This is an article written by Elinor Ostrom, a Noble Prize winner, who has written extensively about the tragedy of the commons and common-pool resources. Or you can directly retrieve the PDF by clicking here.

Open Zotero, and drag the PDF into the main window (which should also list other references, usually according to title and author). After doing so, the PDF will appear with an Adobe Acrobat symbol next to it, but the citation information for the PDF has not been properly entered into the database, yet. Right-click on the title of the PDF. A menu will pop up; select “Retrieve Metadata for PDF”. Another dialogue will pop up showing Zotero’s progress. If Zotero is able to retrieve the metadata for the PDF, it will automatically create a new entry and save the PDF as an attachment to the entry.

Note that you can select multiple PDFs at the same time, right-click on them, and retrieve metadata as described above for a single PDF.

Shared Libraries

Zotero comes with an amazing feature that lets Zotero users share references with each other via Zotero Groups, again, for free. See the link for instructions on setting up Zotero Groups and syncing your Zotero collection.

If you have a shared library (whether the library is owned by you or another person), sources that are added or edited by any member of the shared library will appear in your local copy of the library. If you have enough storage for the library (currently everyone gets 300 MB for free), attachments (including PDFs) will sync, as well. These sources are available to you like any other source in your library.

Using Your Reference Library to Make Citations

In Word for PC, Zotero commands appear in their own toolbar. In Word for Mac, Zotero appears under the script icon between the “Window” and “Help” menus. To insert a citation, click “Add citation”. The first time you use this command in a Word document, Zotero will ask which citation format you want to use in that document. Select your preferred citation format from the pop-up menu, and I recommend to leave the “Store references in document” option checked. Doing so ensures that when you share your document with others, they will also be able to see and work with your Zotero citations.

Anytime you insert a citation from Zotero, a Zotero search bar will pop up, or Zotero will bring you to a window of your Zotero library. If the search bar pops up, type in the author or title of the source you want to cite. If the window of your Zotero library pops up, click on the source(s) you want to cite. Zotero will automatically insert a properly formatted in-text citation.

You may need to edit your citations as you draft your document or to format them properly (e.g., to add a page number for an in-text citation after a direct quote). You can modify in-text citations by clicking on them (so the cursor is in the in-text citation) and then selecting “Edit citation” from Word’s Zotero menu. A Zotero dialogue will pop up. Right-click on the citation you want to modify, and select the options you want.

At any time, you can use the “Add bibliography” command in Word’s Zotero menu to insert the full reference list. The reference list will appear in the same citation format as the in-text citations.

To change the citation format for both in-text references and the reference list, use the “Set document preferences” command in Word’s Zotero menu.

Using More Citation Formats

Zotero has many citation formats (7,449 at the time of writing)—far more than the basic APA, MLA, and Chicago styles. See the Zotero Style Repository for a list. If you’re writing a manuscript for an academic journal, Zotero might even have a custom citation style for the journal you’re submitting to. Zotero does not automatically download all the available citation styles. The most popular ones are preloaded, and others need to be downloaded and installed manually. (See here for instructions.)

Additional Resources

Of course, this tutorial is not exhaustive, so here are some additional resources for your further edification:

• This page includes a tutorial for installing Zotero.
• Here is a video tutorial on installing and using the Zotero plug-in for Word.
• This page has a video tutorial that goes over the basics of using Zotero.
• Finally, this page contains links to a variety of more specific topics on using Zotero.

Automating Sente Exports

In this post, I’ll provide some guidance on getting and using Sente’s reference library without opening the application itself. In last week’s post, I wrote about fixing Sente’s export to BibTeX. Being able to fix up the *.bib file that Sente exports helps with writing documents in LaTeX. Using the fix that I demoed in last week’s post, you could also make your reference metadata more consistent (e.g., removing internal periods from US state abbreviations) and ensure the metadata conforms to your style guide (e.g., capitalization of source titles). However, the fix that I demoed requires opening up Sente and exporting the reference library both as SenteXML and as BibTeX. Doing so is not an unbearable burden, but it obstructs fully scripting (i.e., fully automating) parts of the workflow.

Sente Library Basics

The Sente reference library is an SQLite3 database. SQLite is a implementation of SQL, and, according to its website, “the most widely deployed SQL database engine in the world.” In my experience, it is lightweight and relatively easy to use.

To access Sente’e SQLite database, you first have to find it. Sente library files have the extension *.sente6lib. This file is actually a folder that contains several files including the SQLite database and PDFs for articles (if you have elected to include them in the Sente library file).

Open a terminal and navigate to the folder containing the Sente library file. Then, navigate into the sub-folder with the SQLite database. One of my library files is called ASU-References or ASU-References.sente6lib in full, so to get to the folder containing the SQLite database, I enter the following in the terminal:

cd ASU-References.sente6lib/Contents

As far as I can tell, the SQLite database is always called primaryLibrary.sente601, although it may differ in other versions of Sente. To get into the database, enter the following in the terminal:

sqlite3 primaryLibrary.sente601

This command opens SQLite and allows you to get information about the database and its tables and to perform SQL queries on the database. To get a list of tables in the database, enter

.tables

To get the schema for a given table, enter

.schema TABLENAME

For example,

.schema Reference

To get just the column headings, enter

.indices TABLENAME

For example,

.indices Reference

These are some helpful basics. For a full list of SQLite commands, simply enter

.help

To exit SQLite, enter

.q

Sente Citation Identifier

One of the most valuable pieces of identifying information about a source is its Sente citation identifier. The citation identifier is the tag that Sente reads when it scans text documents (e.g., Word documents). An example in-text citation follows:

This is a made up sentence for display purposes only {Arnstein 1969}.

The tag is “{Arnstein 1969}”. So if you’re using Sente to cite sources, your documents are full of Sente citation identifiers. And if you want to do something automatic with citations, there is a good chance you will want to retrieve information about a source based on its citation identifier.

The citation identifier appears in the SparseAttribute table, which contains a lot of other additional information about each citation, too. The table has three columns:

• ReferenceUUID
• AttributeName
• AttributeValue

Here are the table contents for one reference in the SparseAttributes table:

C914FC95|EndNote reference number|116
C914FC95|publicationCountry|
C914FC95|publisher|Routledge
C914FC95|Primary contributor role|Author
C914FC95|DOI|10.1080/01944366908977225
C914FC95|BibTeX cite tag|arnstein1969ladder
C914FC95|Citation identifier|Arnstein 1969
C914FC95|publicationStatus|Unknown
C914FC95|ISSN|0194-4363
C914FC95|Web data source|Google Scholar

In the output above and throughout the rest of the post, I have abbreviated the entry in the first column, which is the ReferenceUUID, from C914FC95-5EE5-4C2D-BC80-1621EC22C2EC to C914FC95.

This particular reference is a well-known article on community participation. Its full citation is

Arnstein, Sherry R. 1969. “A Ladder of Citizen Participation.” Journal of the American Planning Association 35 (4): 216–224. doi:10.1080/01944366908977225.

Sente References’ ReferenceUUID

As far as I can tell, the ReferenceUUID is Sente’s true key for each entry in the database. In database terms, the ReferenceUUID is the unique, primary key for a reference.

Aside from being the primary key for a reference, the ReferenceUUID is also important because Sente will serve it up to other applications that request information about highlighted Sente references. For example, to use Robin Trew’s AppleScript (which exports Sente reference notes to DevonThink), you highlight references in Sente and then run the script. The script gets the ReferenceUUID for highlighted references, uses it to grab more information (including notes) about each reference, applies some formatting, and then ships the results to DevonThink. I recommended using this AppleScript in my original post on academic workflows.

It should be possible to pull the citation identifier out of the AttributeValue column by looking up the ReferenceUUID for a given reference, selecting the row that contains “Citation identifier” in the AttributeName column, and pulling the value in the AttributeValue column for that row. For the Arnstein reference, the goal is to retrieve the seventh row from the output above. The seventh row is

C914FC95|Citation identifier|Arnstein 1969

I would expect the following SQL query to retrieve the citation identifier:

select AttributeValue from SparseAttribute 
where AttributeName='Citation identifier' 
and ReferenceUUID='C914FC95'; 

But this query fails. To figure out why it fails, I examined the schema for the SparseAttribute table, which is

CREATE TABLE SparseAttribute 
    ( ReferenceUUID varchar, 
    AttributeName varchar COLLATE SenteLocalizedNoCase, 
    AttributeValue varchar COLLATE SenteLocalizedNoCase );
CREATE INDEX SparseAttributeByName 
    ON SparseAttribute ( AttributeName );
CREATE UNIQUE INDEX SparseAttributePK 
    ON SparseAttribute 
    ( ReferenceUUID, AttributeName );

Note that this table has a collation sequence. I have to admit that I don’t know what that is or what it does, but it causes the SQL query to throw an error. The following is a corrected version of the SQL query:

select AttributeValue from SparseAttribute 
where AttributeName='Citation identifier' COLLATE NOCASE 
and ReferenceUUID='C914FC95' COLLATE NOCASE;

This SQL query gives the following result:

Arnstein 1969

This is the correct citation identifier for the reference.

Next Steps

When I was working on my dissertation, I found that Robin Trew’s AppleScript stopped working. I was relying on the script to export Sente reference notes to DevonThink, and it was a key part of my workflow. The AppleScript makes SQL queries to Sente’s reference database, processes the information it retrieves, and then sends it to DevonThink. So I used all this information about Sente’s SQLite database to update the script and get it working again. If people are interested, I can post my version of the script, although mine may have become out of date by now, too.

In general, I hope this post helps other people who want to write scripts that retrieve information about their Sente references. With access to Sente’s SQL database, it’s possible to do … pretty much anything with metadata about references in your Sente library. You could even reimplement Sente’s export commands (e.g., to fix errors in the way Sente exports to BibTeX).

Academic Workflow for the Ages (Part 2)

Over two years ago, I wrote a post about an academic workflow, mostly for literature review. I did not focus on writing, although I did recommend that people to use Scrivener to draft their documents (dissertation chapters, journal articles, and so forth). In this post, I discuss an alternate way to draft documents, which I think is much better.

Pandoc

Natural Writing

I recommend that people write their documents in pandoc markdown. Pandoc’s author describes it as “your swiss-army knife” for documents because it can convert between many document formats (e.g., HTML to MS Word).

Markdown is a syntax for plain-text documents that aims to be readable in plain text but also have enough structure that it can be parsed and translated into other formats. (Markdown’s original authors aimed to generate HTML documents from markdown.) For example, the following is a list in a markdown document:

Here is a list written in markdown: 

* Here is an item in the list.
* Here is another item in the list. 
* Here is the final item in the list. 

Even in a plain-text file, it’s clear that this is a list, so it’s easy to read this document in plain text and understand the intended formatting. Additionally, because the syntax is very simple, it’s easy to just write your thoughts, arguments, and whatever else you need to without pausing to deal with formatting. If I wanted to make a list in an e-mail, I would write it exactly as written above. At least for lists, there’s nothing new to learn, and in general, writing in markdown is very natural. Markdown achieves a tremendous separation between content and formatting. Pandoc converts the list above into the HTML shown below.

<p>Here is a list written in markdown:</p>
<ul>
<li>Here is an item in the list.</li>
<li>Here is another item in the list.</li>
<li>Here is the final item in the list.</li>
</ul> 

When writing in pandoc markdown, you can also use HTML comments to make notes to yourself and keep them right next to the material they refer to. For example, you could write an outline of a journal article you’re drafting and enclose it in an HTML comment, so it’s excluded from the output (for most formats) but still at the top of your own markdown document.

<!-- This is an HTML comment. --> 

<!-- 
Comments can also 
span 
several lines. 
--> 

Here is a list written in markdown: 

* Here is an item in the list.
* Here is another item in the list. 
* Here is the final item in the list. 

And because pandoc markdown is stored in plain-text files, you can use a variety of text editors to write them. Personally, I prefer a minimalist writing environment because there are less distractions. MS Word, for example, has so many formatting tools immediately available that it’s tempting to write something and then immediately fix up how it looks. With a plain-text editor, there are virtually no distractions. My favorite at the moment is Text Wrangler on Mac with the font set to display 24pt Helvetica. It’s free and simple, and it does what I need it to.

More Features

Pandoc expands traditional markdown with new features, such as different kinds of lists, different kinds of tables, in-text citations and reference lists, footnotes, and metadata. For example, you can make a table with the following pandoc markdown:

-------------------------------------------------------------
 Centered   Default           Right Left
  Header    Aligned         Aligned Aligned
----------- ------- --------------- -------------------------
   First    row                12.0 Example of a row that
                                    spans multiple lines.

  Second    row                 5.0 Here's another one. Note
                                    the blank line between
                                    rows.
-------------------------------------------------------------

Table: Here's the caption. It, too, may span
multiple lines.

If you want to cite a source, you can write the following:

Blah blah [@smith04; @doe99].

smith04 and doe99 are BibTeX keys in a BibTeX database.¹ Pandoc will replace them with proper in-text citations and generate a reference list at the end of the document. You can even use Zotero’s citation styles, so there are a lot of options for automatic formatting, including Chicago, APA, MLA, and formats for many academic journals.²

For all the features and how to use them, spend some time on pandoc’s readme page, and try them out for yourself. Note: The table and citation examples in this section were copied verbatim from pandoc’s readme.

More Formats

Pandoc can convert markdown into formats other than HTML, including LaTeX. Pandoc converts the markdown list above into the following LaTeX:

Here is a list written in markdown:

\begin{itemize}
\itemsep1pt\parskip0pt\parsep0pt
\item
  Here is an item in the list.
\item
  Here is another item in the list.
\item
  Here is the final item in the list.
\end{itemize}

And as I mentioned above, pandoc can also convert markdown to an MS Word document (*.docx file). Do you see where this is going? If you write your documents in pandoc markdown, you can easily convert them to other formats as needed. Lists, tables, and citations will all appear correctly in a variety of formats. If you’re a PhD student and your advisors want to review your work in MS Word (usually because of its handy track changes feature and comments feature), you can just take the current version of your markdown file, convert it to MS Word, and send it out. And when it’s time to make a final version, you can still convert your markdown to LaTeX and apply whatever template you want to (your school’s template, a journal’s article template, or your own custom template). You don’t have to change your source file at all; just apply a new template to get the formatting you need.

A Simple Example

Returning to the list example above, if I have a markdown file named temp.md with the following contents:

Here is a list written in markdown: 

* Here is an item in the list.
* Here is another item in the list. 
* Here is the final item in the list. 

I can open a terminal and convert it to HTML with the following command:

pandoc -r markdown -w html -o temp.html temp.md

Pandoc will create an HTML file called temp.html with the following contents:

<p>Here is a list written in markdown:</p>
<ul>
<li>Here is an item in the list.</li>
<li>Here is another item in the list.</li>
<li>Here is the final item in the list.</li>
</ul>

Reproducible Statistics

But wait. There’s more!

If your work involves statistics and you’re familiar with R, you can write something called R Markdown. R Markdown files can contain both markdown and R code (and R code that writes markdown). So you can write an R Markdown file with your entire statistical analysis that writes out fresh statistics, tables, and figures every time you process the file. If you find an error in your work or you want to update the way a figure looks, just change the code in the R Markdown file and reprocess it. Everything else will be the same plus your changes will have been included.

I won’t get into the details of setting up an R Markdown file here, but basically, you create a file with the *.rmd extension and write in your markdown and R code. Then, you “make” the file by running R on it. There are different ways to do this. You could call R from the command line with Rscript, for example. In any case, the result will be a pandoc markdown file (with a *.md) extension, which you can then convert to other formats (e.g., MS Word, LaTeX, etc.) as you would any other pandoc markdown file. In addition, you can easily share your R code with anyone who wants to check your work.

Disadvantages

A Command-Line Tool

For some people, the fact that pandoc is a command-line tool will be a disadvantage. Some graphical user interfaces (GUIs) for pandoc are listed here. I haven’t tried them, so I can’t make any recommendations.

The LaTeX Writer

One of the footnotes in this post discusses my complaints about the way pandoc handles citations. Another problem is the way pandoc writes out tables in LaTeX. Pandoc automatically writes out tables as longtables in LaTeX and sometimes inserts minipages in the middle of tables. For example, the following table is written in pandoc markdown and saved in a file called temp.md:

-------------------------------------------------------------
 Centered   Default           Right Left
  Header    Aligned         Aligned Aligned
----------- ------- --------------- -------------------------
   First    row                12.0 Example of a row that
                                    spans multiple lines.

  Second    row                 5.0 Here's another one. Note
                                    the blank line between
                                    rows.
-------------------------------------------------------------

Table: Here's the caption. It, too, may span
multiple lines.

Pandoc will convert the document to LaTeX with the following terminal command:

pandoc -r markdown -w latex -o temp.tex temp.md

The above command creates a file called temp.tex with the following contents (although I added some line breaks for formatting):

\begin{longtable}[c]{@{}clrl@{}}
\toprule\addlinespace
\begin{minipage}[b]{0.15\columnwidth}\centering
Centered Header
\end{minipage} & %
\begin{minipage}[b]{0.10\columnwidth}\raggedright
Default Aligned
\end{minipage} & %
\begin{minipage}[b]{0.20\columnwidth}\raggedleft
Right Aligned
\end{minipage} & %
\begin{minipage}[b]{0.31\columnwidth}\raggedright
Left Aligned
\end{minipage}
\\\addlinespace
\midrule\endhead
\begin{minipage}[t]{0.15\columnwidth}\centering
First
\end{minipage} & %
\begin{minipage}[t]{0.10\columnwidth}\raggedright
row
\end{minipage} & %
\begin{minipage}[t]{0.20\columnwidth}\raggedleft
12.0
\end{minipage} & %
\begin{minipage}[t]{0.31\columnwidth}\raggedright
Example of a row that spans multiple lines.
\end{minipage}
\\\addlinespace
\begin{minipage}[t]{0.15\columnwidth}\centering
Second
\end{minipage} & %
\begin{minipage}[t]{0.10\columnwidth}\raggedright
row
\end{minipage} & %
\begin{minipage}[t]{0.20\columnwidth}\raggedleft
5.0
\end{minipage} & %
\begin{minipage}[t]{0.31\columnwidth}\raggedright
Here's another one. Note the blank line between rows.
\end{minipage}
\\\addlinespace
\bottomrule
\addlinespace
\caption{Here's the caption. It, too, may span multiple lines.}
\end{longtable}

Notice that it’s a longtable containing minipages. This looks to me like a bit of madness. My understanding is that longtables are used by default because if they’re not, tables that are longer than a single page will not appear correctly. And I assume that authors complained about this problem in the past leading to the current setup. However, longtables could at least be implemented more elegantly with the tabu package. In general, it seems that more LaTeX formatting should be left to the LaTeX template (which can be changed easily by authors) than the pandoc writer (which cannot be changed easily by authors).

Git

Another benefit of writing documents in markdown—or really any plain-text format—is the ease of using version control. Version control systems, such as git, track changes to files, and with websites like GitHub, it’s possible to keep a remote backup of your files and their changes. Version control is especially useful on long projects, such as books or dissertations, where you may want to keep separate chapters in separate files. And if you’re collaborating with tech-savvy people, you can all work on the same set of files and track changes collectively.

I won’t give a tutorial on git here, and unfortunately, there’s a bit of a learning curve. But here are some useful resources:

• A pretty good, brief introduction: http://rogerdudler.github.io/git-guide
• A more detailed introduction: http://gitolite.com/gcs.html
• A very detailed guide book: http://git-scm.com/book
• Some interactive exercises on git branching: http://pcottle.github.io/learnGitBranching

The Setup

In summary, the setup I recommend is writing documents in either pandoc markdown or R markdown (depending on whether they contain statistics) and using git to track changes to documents. This setup works very well with long works, such as books and dissertations, where it makes sense to separate chapters into individual files.

---

Notes:

---

1. You need to tell pandoc the name of the BibTeX file when you execute the pandoc command on the command line, and the file needs to be in a folder where pandoc can find it (i.e., in the same folder as the markdown file or in the folder indicated by the --data-dir flag). Again, see the readme for details.↩

2. I actually have some gripes with the syntax for pandoc’s in-text citations. It’s convenient for quick documents, but for a formal academic work, pandoc’s citation commands are somewhat lacking, and BibLaTeX’s in-text citation commands are much more powerful. This is a tricky situation because while you can write BibLaTeX macros directly in pandoc markdown, pandoc won’t use them unless it’s converting that document to LaTeX. So if you want to create an MS Word document instead of a PDF, you’ll lose your citations. Personally, my solution was to write a parser in perl that could find pandoc citations and replace them with BibLaTeX citations, so I could still output LaTeX documents containing BibLaTeX formatting. But that might not be a solution for everyone. If there’s enough interest, I will polish and publish my perl code for others to use.↩

Our Current Methods for Literature Review may be Antiquated Soon

Reviewing academic literature has become much more efficient over the past 20 years. When I was in grade school, libraries still used card catalogs. Card catalogs are drawers of note cards that provide the library’s contents by author, title, or subject. By the time I finished high school, most libraries had converted their physical card catalogs to electronic ones. And when I was finishing my undergraduate degree in 2004, electronic databases of many academic journals were established and becoming more and more user-friendly. Now, as I work on my PhD, I can easily search massive online databases of academic journals with a few keywords and get full-text PDFs of articles in just a few seconds. It’s as easy as a Google search, and, in fact, one of the online databases that I use is Google Scholar. I occasionally go to the library to pick up a book, but I don’t think that I’ve looked at a hard copy of an academic journal since starting the PhD. And I even try to avoid hard copies of books because Kindle versions can be annotated electronically and then easily integrated into my academic workflow.

Over the past couple decades, searching for literature has gotten much faster, and many sources are now available in digital form, but the literature review process has basically remained the same: select a topic, search for sources based on keywords, review sources, and repeat.

In the next few decades, advances in computer science may allow researchers to interact with literature in new ways. The two possibilities I outline below are close to being available now, but they’re not easy to implement, and there’s plenty of room for them to improve.

First, researchers should be able to identify key literature articles on a given topic without having to do the leg work of sorting through current articles and tracing their citations back to the most influential pieces. An application could crawl academic literature databases for certain terms, pull down articles that match, parse the citation lists, and construct a network map of sources on the topic. The network map could be used to identify particularly influential pieces of literature. By taking each source’s publication date into account, the application could identify sources that are turning points in a given discourse.

This is possible now, but no one has bundled existing programs into a well-engineered, user-friendly package. Crawling is nothing new in computer science. Google has been using web crawlers to create an index of web pages since its inception. Here‘s an early paper by Google’s founders on their plans for Google in 1998. (Here are the conference proceedings.) To create a network map of sources, the program should be able to get the list of citations for each source and the list of sources that cite each source. There are many ways to get an article’s list of citations automatically. Some academic literature databases, like Google Scholar, provide sources’ citation lists. Alternatively, the program could retrieve the full-text of the article as a PDF, perform optical character recognition (OCR) on the PDF, and then parse the citations from the resulting plain text of the article. Google has a free OCR engine called Tesseract OCR that can be used with perl modules like PDF::OCR2 to automatically pull plain text out of PDFs. It would be possible to parse the plain text citation lists into individual citations (e.g., with regex statements) and feed these citations back into the web crawler to repeat the process again. Some databases of academic literature already provide the list of sources that cite a given source. Web of Knowledge provides a feature called cited reference search that allows researchers to see which articles have cited a given source. And there’s already at least one perl module (SNA::Network) that can do network analysis. Most, if not all of the pieces, for such an application already exist.

Imagine identifying a theory of interest and, within minutes, having the foundational article for that theory along with the major branches of discourse stemming from it. Perhaps, disciplinary researchers wouldn’t find much value in this because researchers in the field know which articles are in the field’s canon. As an interdisciplinary researcher, I think a tool like this one would be incredibly exciting. And even disciplinary researchers might find out that particular articles are more influential than they previously thought.

Second, researchers should be able to automatically code articles to identify patterns in academic literature. If researchers can specify the themes they’re looking for, a computer program can do the work of finding articles and coding them with researchers’ coding schemes. Such programs have already been used in biology to extract information on biodiversity from scientific articles. There’s a nice summary article here, and I recommend to look at the summary figure. Information on biodiversity is well-suited to this approach because biological classification has a clear taxonomy (e.g., species, genus, and so forth). But researchers are interested in many themes, which could be similarly specified. For example, I’m currently researching participatory approaches in international development projects, and I am coding literature according to a taxonomy of participatory approaches that I have developed. Sophisticated programs like those used in biology could automatically find and code the literature according to my own taxonomy.

There are several ways to “teach” the program how to code text. The researcher could define rules for interpreting text (called hand-crafted rules). The researcher could also code a subset of articles with the scheme and then feed the hand-coded articles into the program which then “learns” the coding system on the fly (called machine learning). Currently, developing this kind of system is very labor-intensive. But imagine what will be possible in 10 or 20 years when both computational power and computers’ ability to process natural language have increased dramatically. Perhaps a researcher could code 10 articles on his or her own, feed them into a program, and then let the program search for additional articles, code them automatically, and deliver a summary of results.

The programs I’ve described here are more suitable for certain research questions, and they may be totally worthless for some research questions. It’s easy to imagine that people in the humanities would get much less use out of the programs described above. I don’t think it will ever be possible (or desirable) to remove people from the research process, but it may be possible to make certain parts of the research process much more efficient.

Creating PDFs from Chapters of a Google Book

This post describes how Mac users can quickly create PDFs with searchable text out of Google Book chapters (comparable software is also available on PCs). Being able to do so is especially useful for people using the academic workflow described in an earlier post. Here’s what you need:

• Web browser (e.g., Firefox) with these extensions (or equivalent for a browser other than Firefox):
  • Greasemonkey extension and the Google Book Downloader Greasemonkey script
  • FlashGot
• PDF printer (e.g., included in Preview)
• OCR engine (e.g., ABBYY included in DevonThink Pro Office) 

First, install the extensions in the web browser. I’m not sure what the equivalent of FlashGot in other browsers is, but equivalent Greasemonkey extensions for different browsers follow:

• Chrome: Tampermonkey 
• Safari: SIMBL / GreaseKit (instructions)

Install the Google Book Downloader Greasemonkey script in your web browser: just click the “Install” button on the page with the script. A dialogue box with instructions will pop up.

Next, search for the book on Google Books. Let’s pretend I’m interested in The White Man’s Burden by William Easterly. (This is a great book on international development and recommended for anyone interested in the topic.) If you have the Google Book Downloader script installed, you’ll see a button in the left column named “Download this book”.

If you click the “Download this book” button, you’ll see another set of buttons in its place.

You can use the drop down menus to select the page range that you want to download. I suspect that Google detects when you load too many pages from a single book, so downloading pages for the whole book will probably not work. You can enter a page range for a single chapter at a time, though. For The White Man’s Burden, the first chapter goes from “PT13” to “PT48”.

Then, if you click “Get Download Links” the Greasemonkey script will go to work and create links for each page of the book. An easy way to grab all the image files is to right click and select “FlashGot All”. FlashGot will pop up with a dialogue asking where to download the image files. After setting the download folder, FlashGot will pull down all the files.

After downloading, all the images and a bunch of other files from Google Books will be on your computer. You can ignore or delete the extra files. If you’re using a Mac, you can use Preview to open and print all the image files into a single PDF. Select all the image files and press ⌘ + O.

Preview will open with all the image files. It would be a good idea to check through the image files to make sure that there are no duplicates and that all the pages are in order.

Then, you can print all the images in one PDF. You can select all the image files by pressing ⌘ + A and then print them by pressing ⌥ + ⌘ + P. If you want to use menus, you can click “Edit”, and then “Select All”; next, click the “File” menu, and then “Print Selected Images ...”.

Preview will pop up with a dialog box. You can select the button in the lower-left corner labelled “PDF” and then select “Save as PDF ...” from the drop down menu. Preview will produce a nice PDF of all the pages from the Google Book, but at this point, the pages are just images. You wouldn’t be able to search the text in the PDF, select text in the PDF, or copy and paste text from the PDF. In other words, the PDF is not much use to an academic researcher as a collection of images.

Finally, use an OCR engine on the PDF. If you have DevonThink Pro Office 2.0 as recommended in an earlier post, you can use its OCR engine, called ABBYY. You could also get ABBYY separately, although in my opinion you may as well pay another $50 and get DevonThink, too.¹

To use OCR on the PDF with DevonThink, click the “File” menu, then “Import”, and then “Images (with OCR) ...”. Then, ABBY will run OCR on the PDF. This step may take awhile, so be patient.

When OCR finishes, select the resulting file in DevonThink. Click the “File” menu, then “Export”, and then “Files and Folders ...”. Or you can use the keyboard shortcut: ⌥ + ⌘ + E. DevonThink will pop up a dialog box asking where you want to put the file.

After OCR, you can search the text in the PDF, select text, and use copy and paste to pull text out of the PDF. If you’re using Sente as recommended in an earlier post, you can import it like any other journal article and annotate it.

---

Notes:

---

1. There are also perl modules that can OCR images and PDFs. I have experimented with the PDF::OCR2 module and successfully extracted the text into a separate file, but I haven’t taken it further to see if it’s possible to run OCR on a PDF and keep the results in a PDF with the same appearance. If this can be done, it’s possible OCR PDFs for free instead of paying upwards of $100 for OCR software.↩

Academic Workflow for the Ages

I’ve been a connoisseur of citation software for a while now. But when people ask me which citation program they should use, my response is always: “It depends.” Aside from being a nice, safe answer (I am a well-trained graduate student at this point), it’s also true.

Do you want to prepare a manuscript for an academic journal? Do you need to share your references with colleagues? Are you willing to pay for the software? What word processor do you use?

Asking which citation software is the best is the wrong question. The right question is “What software do I need for a complete academic workflow?”

And in this case again, there are several possibilities depending on your specific situation, but at least you will arrive at a good answer. In this post, I first outline some of the strengths of different citation programs, and then I outline two good options for an academic workflow, starting with the one that I personally use (the aforementioned “Academic Workflow for the Ages.”)

Some Citation Programs

First, for the love of your own sanity, use citation software! I cry a little every time I encounter a grad student nearing the end of his or her thesis without using any citation software. I shout to the heavens, “Why?!” and curse the gods for allowing this tragedy to occur. Please don’t do this to yourself. And talk to your peers: Friends don’t let friends do grad school without citation software. Second, you should never have to write references into a paper for class or a publication. To avoid this tedious work, make sure your citation program is compatible with your word processor. If you use Microsoft Word, make sure that whichever citation program you’re using can automatically insert in-text citations and a bibliography at the end (most can). If you’re using OpenOffice or Pages or LaTeX, similarly make sure that your citation software is compatible. Third, don’t worry about the file format that the program uses. (For example, Zotero uses *.ris files; JabRef uses *.bib files and so on.) In my experience, programs can import and export any format you need, so you can throw this criterion out. Finally, with the exception of Sente, the programs I mention below will work on both PCs and Macs.

Programs for research

These programs are better suited to doing literature reviews because they have some functions for taking notes on references in addition to managing citations.

Zotero versus Mendeley

Zotero and Mendeley are largely comparable: They’re both free; you can share libraries with people; they can automatically import citations from web pages and PDFs; and they both have cite-while-you-write plug-ins for Microsoft Word. Those are the basics, and both of these programs have them for free. I always recommend Zotero because, in my personal experience, it’s more stable than Mendeley and more user friendly. I’ve had Mendeley crash and delete references and do various weird things to my reference libraries. I don’t like that. Also, I like the way that Zotero grabs citation information from web pages. You simply click the icon in your browser’s address bar and in comes the reference information. Mendeley requires you to use a link (which you should save as a bookmark) that takes you to a new page and so forth. It’s awkward. So for my money, I’ll take Zotero every time.

Mendeley’s one saving grace is its PDF annotation feature. If you’re looking for free software that annotates PDFs, Mendeley has the advantage here, but if you’re willing to pay (and if you’re working on a PhD you probably should be), then there are better options. Also, Zotero can save notes on references which isn’t as good as proper annotation but helps nonetheless.

Sente

Sente is paid software, but you also get something for the investment: the best PDF annotation software available (more details on this below). It also has the same features as these other programs: automatically import reference information, share reference libraries, and so forth. It doesn’t have a cite-while-you-write feature for the latest version of Microsoft Word, but it has a document scanning feature, which serves the same purpose. I actually like document scanning better than cite-while-you-write because you can copy and paste references across documents in different formats (e.g., from a *.rtf file to a *.docx file). There’s also an iPad version of Sente, so you can sync your library across your Mac and iPad and then review PDFs on the tablet. Personally, I think the iPad is overpriced, but if I had one, I would love to review PDFs on it.

Preparing Manuscripts

These are candidates for drafting manuscripts for academic journals because they have large libraries of citation formats that cover most journals. This feature is valuable because many journals have custom citation styles. You might know Chicago or APA style, but there’s a very small chance that you know Journal of Industrial Ecology-style. So instead of manually writing in-text citations and a bibliography in a new citation style, these programs will do that tedious work for you.

Endnote versus RefWorks

I recommend EndNote because the one time I used RefWorks, its format for the journal was incorrect! So I had fix all the references by hand. Thanks RefWorks. The downside of using Endnote is that it costs money. Keep in mind that in some fields, like mathematics, authors typically submit articles in LaTeX. If you’re writing manuscripts in LaTeX, a BibTeX program is best.

BibTeX Programs

By BibTeX programs, I mean software like JabRef (which is cross platform) and BibDesk (which is Mac specific). These programs make the most sense if you’re writing your papers in some version of TeX (e.g., LaTeX). But you can make some of them work with other word processors. For example, you can export your references from JabRef and import them directly into an MS Word document. In my experience, BibTeX programs require a bit more manual work, too. Using JabRef, I had to manually link each PDF to the reference in the library, and if I wanted to move the *.bib file to a new computer, I would need to remake all those links manually.

Workflows

I use the notecard method for reading literature and writing papers. Steps in the notecard method are the following: review a source; record each important quote or thought on its own notecard along with that source’s citation key and the page number; repeat for all your sources; organize notecards into piles; and finally turn notecard piles into the paragraphs of your paper. That’s the rough idea.

So for me, the goal is to create an electronic workflow that tracks citation information, that allows me to make electronic notecards for important quotes and thoughts, and that allows me to organize those notes and draft paragraphs based on them.

Academic Workflow for the Ages

This workflow uses the following software:

• Sente 6 ($89.95): bibliography database and note taking
• Sente 6 Notes to DevonThink 2 AppleScript (free): export Sente notes to DevonThink
• DevonThink ($112.46): searchable database of notes
• Scrivener ($38.25): organizing notes and drafting
• Microsoft Word ($119.99 for Office Suite): preparing the final version

Unfortunately, all this software costs money. Academic licenses cut down on costs, but getting all this software will cost over $300. To run the software, you also need a Mac, and those aren’t cheap. I personally think that a PhD is such a massive investment of your own time that paying for the right setup is worth a little investment, too.

Consider the amount of money that you spend on a car (if you have one) and compare the amount of time you spend in your car to the amount of time you spend in front of your computer. I’d be surprised if buying the car plus insurance costs less than owning a computer, and that calculation doesn’t even take into account the fact that you’re almost certainly using the computer far more often. I use my computer more than anything else, and it’s also the most important tool I have for getting my research done. That warrants some investment in my opinion. Also, these programs all have free trial periods, so you can test all this out before making the investment.

Sente 6 is a good bibliography program, and it’s the best PDF annotation software available. When you’re annotating a PDF, Sente gives you a big view of the PDF and a sidebar for note taking. Each Sente note has four fields: title, page number, quote, and comment. You can highlight text in a PDF, and Sente can automatically create a note with a title (the first couple words of the quote), the quote itself, and the page number. It leaves the comment field blank, so you can enter your own thoughts. You don’t need to type anything except your own thoughts. That sounds like a pretty damn efficient way to make notes on sources to me. And Sente automatically tracks citation information (more on that later). After reviewing a source, I write an annotated bibliography and save it as a note for that source, as well.

Next is the most beautiful part of this workflow: use Robin Trew’s AppleScript to export your notes from Sente into DevonThink. The script gives each source its own folder. Each note is a text file stored in that source’s folder. The text file contains the note’s title, quote, comment and citation information. The citation information is a tag, like {Goldman 2009@375}. “Goldman 2009” points to the reference, and “@375” refers to the page number. Sente can read these citation tags in documents and replace them with properly formatted citations (more on that later). If you assign keyword tags to sources in Sente, those will be transferred into DevonThink, as well. You can also create new keyword tags in DevonThink. These text files are the equivalent of electronic notecards. And with Trew’s applescript, you can have a searchable database of them. Imagine what it would be like to have a searchable database of three years of literature review.

There are different versions of DevonThink, and I recommend to get the most expensive version because it comes with an OCR engine. OCR stands for optical character recognition. With OCR, you can import an image into DevonThink, and DevonThink will convert any text in the image into selectable and searchable text. You don’t need this feature for the PDFs that contemporary academic journals produce. Those PDFs are high quality, and you can select text in them and copy and paste easily. That’s important because in order to annotate the PDFs in Sente, the text in the PDF needs to be selectable.

There are two instances where OCR is valuable: (1) old journal articles and (2) selections from Google Books. Old journal articles tend to be PDFs as images without selectable text, so you won’t be able to annotate them in Sente. If you run them through DevonThink’s OCR engine, the text becomes selectable, and you can annotate them. Similarly with pages from Google Books, the pages are actually image files (*.png, I believe), so if you want to import them into Sente and take notes, you’ll need to run them through an OCR program.

When you’re ready to start writing, you can search your database of notes in DevonThink and drag and drop the most promising ones into Scrivener. Then, you can use Scrivener to write your first draft. Keep in mind that each notecard contains the citation (including the page number) of the source. So as you write your draft, you simply carry over the citation tags.

You can then export your draft from Scrivener and copy it into a Word document for formatting. When you’re done writing the document, you can scan it with Sente. Sente will go through the document, replace citation tags with properly formatted in-text citations and then put a properly formatted bibliography at the end. (Have a look at Sente’s guidance on citation tags.) Sente supports many citation formats but not as many as EndNote.

Another Elegant Solution

Using BibTeX software along with LaTeX is an efficient way to create documents. Unfortunately, I don’t know a good way to annotate PDFs and store notes with this workflow. The advantage is that LaTeX PDFs that are far more attractive than anything Word can produce. Sente can generate BibTeX tags, so Sente may offer a good solution. For certain fields, like mathematics, LaTeX is required. And more journals are allowing authors to submit manuscripts using LaTeX, and with its beautiful PDFs, LaTeX is worth considering.

Conclusion

These are some of the broad strokes of citation software and creating an integrated academic workflow. In future posts, I plan to provide more specifics, for example, on turning a chapter of a Google Book into a PDF with selectable text.