Creating and Working with a Corpus
require(quanteda)
## Loading required package: quanteda
## quanteda version 0.9.9.65
## Using 7 of 8 cores for parallel computing
##
## Attaching package: 'quanteda'
## The following object is masked from 'package:utils':
##
## ViewCurrently available corpus sources
quanteda has a simple and powerful companion package for loading texts: readtext. The main function in this package, readtext(), takes a file or fileset from disk or a URL, and returns a type of data.frame that can be used directly with the corpus() constructor function, to create a quanteda corpus object.
readtext() works on:
- text (
.txt) files; - comma-separated-value (
.csv) files; - XML formatted data;
- data from the Facebook API, in JSON format;
- data from the Twitter API, in JSON format; and
- generic JSON data.
The corpus constructor command corpus() works directly on:
- a vector of character objects, for instance that you have already loaded into the workspace using other tools;
- a
VCorpuscorpus object from the tm package. - a data.frame containing a text column and any other document-level metadata.
Example: building a corpus from a character vector
The simplest case is to create a corpus from a vector of texts already in memory in R. This gives the advanced R user complete flexbility with his or her choice of text inputs, as there are almost endless ways to get a vector of texts into R.
If we already have the texts in this form, we can call the corpus constructor function directly. We can demonstrate this on the built-in character object of the texts about immigration policy extracted from the 2010 election manifestos of the UK political parties (called data_char_ukimmig2010).
myCorpus <- corpus(data_char_ukimmig2010) # build a new corpus from the texts
summary(myCorpus)
## Corpus consisting of 9 documents.
##
## Text Types Tokens Sentences
## BNP 1126 3330 88
## Coalition 144 268 4
## Conservative 252 503 15
## Greens 325 687 21
## Labour 296 703 29
## LibDem 257 499 14
## PC 80 118 5
## SNP 90 136 4
## UKIP 346 739 27
##
## Source: /private/var/folders/46/zfn6gwj15d3_n6dhyy1cvwc00000gp/T/Rtmpa9NmmJ/Rbuild90965ea698d9/quanteda/vignettes/* on x86_64 by kbenoit
## Created: Thu May 25 10:37:46 2017
## Notes:If we wanted, we could add some document-level variables – what quanteda calls docvars – to this corpus.
We can do this using the R’s names() function to get the names of the character vector data_char_ukimmig2010, and assign this to a document variable (docvar).
docvars(myCorpus, "Party") <- names(data_char_ukimmig2010)
docvars(myCorpus, "Year") <- 2010
summary(myCorpus)
## Corpus consisting of 9 documents.
##
## Text Types Tokens Sentences Party Year
## BNP 1126 3330 88 BNP 2010
## Coalition 144 268 4 Coalition 2010
## Conservative 252 503 15 Conservative 2010
## Greens 325 687 21 Greens 2010
## Labour 296 703 29 Labour 2010
## LibDem 257 499 14 LibDem 2010
## PC 80 118 5 PC 2010
## SNP 90 136 4 SNP 2010
## UKIP 346 739 27 UKIP 2010
##
## Source: /private/var/folders/46/zfn6gwj15d3_n6dhyy1cvwc00000gp/T/Rtmpa9NmmJ/Rbuild90965ea698d9/quanteda/vignettes/* on x86_64 by kbenoit
## Created: Thu May 25 10:37:46 2017
## Notes:If we wanted to tag each document with additional meta-data not considered a document variable of interest for analysis, but rather something that we need to know as an attribute of the document, we could also add those to our corpus.
metadoc(myCorpus, "language") <- "english"
metadoc(myCorpus, "docsource") <- paste("data_char_ukimmig2010", 1:ndoc(myCorpus), sep = "_")
summary(myCorpus, showmeta = TRUE)
## Corpus consisting of 9 documents.
##
## Text Types Tokens Sentences Party Year _language
## BNP 1126 3330 88 BNP 2010 english
## Coalition 144 268 4 Coalition 2010 english
## Conservative 252 503 15 Conservative 2010 english
## Greens 325 687 21 Greens 2010 english
## Labour 296 703 29 Labour 2010 english
## LibDem 257 499 14 LibDem 2010 english
## PC 80 118 5 PC 2010 english
## SNP 90 136 4 SNP 2010 english
## UKIP 346 739 27 UKIP 2010 english
## _docsource
## data_char_ukimmig2010_1
## data_char_ukimmig2010_2
## data_char_ukimmig2010_3
## data_char_ukimmig2010_4
## data_char_ukimmig2010_5
## data_char_ukimmig2010_6
## data_char_ukimmig2010_7
## data_char_ukimmig2010_8
## data_char_ukimmig2010_9
##
## Source: /private/var/folders/46/zfn6gwj15d3_n6dhyy1cvwc00000gp/T/Rtmpa9NmmJ/Rbuild90965ea698d9/quanteda/vignettes/* on x86_64 by kbenoit
## Created: Thu May 25 10:37:46 2017
## Notes:The last command, metadoc, allows you to define your own document meta-data fields. Note that in assiging just the single value of "english", R has recycled the value until it matches the number of documents in the corpus. In creating a simple tag for our custom metadoc field docsource, we used the quanteda function ndoc() to retrieve the number of documents in our corpus. This function is deliberately designed to work in a way similar to functions you may already use in R, such as nrow() and ncol().
Example: loading in files using the readtext package
require(readtext)
# Twitter json
mytf1 <- readtext("~/Dropbox/QUANTESS/social media/zombies/tweets.json")
myCorpusTwitter <- corpus(mytf1)
summary(myCorpusTwitter, 5)
# generic json - needs a textfield specifier
mytf2 <- readtext("~/Dropbox/QUANTESS/Manuscripts/collocations/Corpora/sotu/sotu.json",
textfield = "text")
summary(corpus(mytf2), 5)
# text file
mytf3 <- readtext("~/Dropbox/QUANTESS/corpora/project_gutenberg/pg2701.txt", cache = FALSE)
summary(corpus(mytf3), 5)
# multiple text files
mytf4 <- readtext("~/Dropbox/QUANTESS/corpora/inaugural/*.txt", cache = FALSE)
summary(corpus(mytf4), 5)
# multiple text files with docvars from filenames
mytf5 <- readtext("~/Dropbox/QUANTESS/corpora/inaugural/*.txt",
docvarsfrom = "filenames", sep = "-", docvarnames = c("Year", "President"))
summary(corpus(mytf5), 5)
# XML data
mytf6 <- readtext("~/Dropbox/QUANTESS/quanteda_working_files/xmlData/plant_catalog.xml",
textfield = "COMMON")
summary(corpus(mytf6), 5)
# csv file
write.csv(data.frame(inaugSpeech = texts(data_corpus_inaugural),
docvars(data_corpus_inaugural)),
file = "/tmp/inaug_texts.csv", row.names = FALSE)b
mytf7 <- readtext("/tmp/inaug_texts.csv", textfield = "inaugSpeech")
summary(corpus(mytf7), 5)How a quanteda corpus works
Corpus principles
A corpus is designed to be a “library” of original documents that have been converted to plain, UTF-8 encoded text, and stored along with meta-data at the corpus level and at the document-level. We have a special name for document-level meta-data: docvars. These are variables or features that describe attributes of each document.
A corpus is designed to be a more or less static container of texts with respect to processing and analysis. This means that the texts in corpus are not designed to be changed internally through (for example) cleaning or pre-processing steps, such as stemming or removing punctuation. Rather, texts can be extracted from the corpus as part of processing, and assigned to new objects, but the idea is that the corpus will remain as an original reference copy so that other analyses – for instance those in which stems and punctuation were required, such as analyzing a reading ease index – can be performed on the same corpus.
To extract texts from a a corpus, we use an extractor, called texts().
texts(data_corpus_inaugural)[2]
## 1793-Washington
## "Fellow citizens, I am again called upon by the voice of my country to execute the functions of its Chief Magistrate. When the occasion proper for it shall arrive, I shall endeavor to express the high sense I entertain of this distinguished honor, and of the confidence which has been reposed in me by the people of united America.\n\nPrevious to the execution of any official act of the President the Constitution requires an oath of office. This oath I am now about to take, and in your presence: That if it shall be found during my administration of the Government I have in any instance violated willingly or knowingly the injunctions thereof, I may (besides incurring constitutional punishment) be subject to the upbraidings of all who are now witnesses of the present solemn ceremony.\n\n "To summarize the texts from a corpus, we can call a summary() method defined for a corpus.
summary(data_corpus_irishbudget2010)
## Corpus consisting of 14 documents.
##
## Text Types Tokens Sentences year debate
## 2010_BUDGET_01_Brian_Lenihan_FF 1949 8733 374 2010 BUDGET
## 2010_BUDGET_02_Richard_Bruton_FG 1042 4478 217 2010 BUDGET
## 2010_BUDGET_03_Joan_Burton_LAB 1621 6429 307 2010 BUDGET
## 2010_BUDGET_04_Arthur_Morgan_SF 1589 7185 343 2010 BUDGET
## 2010_BUDGET_05_Brian_Cowen_FF 1618 6697 250 2010 BUDGET
## 2010_BUDGET_06_Enda_Kenny_FG 1151 4254 153 2010 BUDGET
## 2010_BUDGET_07_Kieran_ODonnell_FG 681 2309 133 2010 BUDGET
## 2010_BUDGET_08_Eamon_Gilmore_LAB 1183 4217 201 2010 BUDGET
## 2010_BUDGET_09_Michael_Higgins_LAB 490 1288 44 2010 BUDGET
## 2010_BUDGET_10_Ruairi_Quinn_LAB 442 1290 59 2010 BUDGET
## 2010_BUDGET_11_John_Gormley_Green 404 1036 49 2010 BUDGET
## 2010_BUDGET_12_Eamon_Ryan_Green 512 1651 90 2010 BUDGET
## 2010_BUDGET_13_Ciaran_Cuffe_Green 444 1248 45 2010 BUDGET
## 2010_BUDGET_14_Caoimhghin_OCaolain_SF 1188 4094 176 2010 BUDGET
## number foren name party
## 01 Brian Lenihan FF
## 02 Richard Bruton FG
## 03 Joan Burton LAB
## 04 Arthur Morgan SF
## 05 Brian Cowen FF
## 06 Enda Kenny FG
## 07 Kieran ODonnell FG
## 08 Eamon Gilmore LAB
## 09 Michael Higgins LAB
## 10 Ruairi Quinn LAB
## 11 John Gormley Green
## 12 Eamon Ryan Green
## 13 Ciaran Cuffe Green
## 14 Caoimhghin OCaolain SF
##
## Source: /home/paul/Dropbox/code/quantedaData/* on x86_64 by paul
## Created: Tue Sep 16 15:58:21 2014
## Notes:We can save the output from the summary command as a data frame, and plot some basic descriptive statistics with this information:
tokenInfo <- summary(data_corpus_inaugural)
## Corpus consisting of 58 documents.
##
## Text Types Tokens Sentences Year President FirstName
## 1789-Washington 626 1540 23 1789 Washington George
## 1793-Washington 96 147 4 1793 Washington George
## 1797-Adams 826 2584 37 1797 Adams John
## 1801-Jefferson 716 1935 41 1801 Jefferson Thomas
## 1805-Jefferson 804 2381 45 1805 Jefferson Thomas
## 1809-Madison 536 1267 21 1809 Madison James
## 1813-Madison 542 1304 33 1813 Madison James
## 1817-Monroe 1040 3696 121 1817 Monroe James
## 1821-Monroe 1262 4898 129 1821 Monroe James
## 1825-Adams 1004 3154 74 1825 Adams John Quincy
## 1829-Jackson 517 1210 25 1829 Jackson Andrew
## 1833-Jackson 499 1271 29 1833 Jackson Andrew
## 1837-VanBuren 1315 4175 95 1837 Van Buren Martin
## 1841-Harrison 1893 9178 210 1841 Harrison William Henry
## 1845-Polk 1330 5211 153 1845 Polk James Knox
## 1849-Taylor 497 1185 22 1849 Taylor Zachary
## 1853-Pierce 1166 3657 104 1853 Pierce Franklin
## 1857-Buchanan 945 3106 89 1857 Buchanan James
## 1861-Lincoln 1075 4016 135 1861 Lincoln Abraham
## 1865-Lincoln 362 780 26 1865 Lincoln Abraham
## 1869-Grant 486 1243 40 1869 Grant Ulysses S.
## 1873-Grant 552 1479 43 1873 Grant Ulysses S.
## 1877-Hayes 829 2730 59 1877 Hayes Rutherford B.
## 1881-Garfield 1018 3240 111 1881 Garfield James A.
## 1885-Cleveland 674 1828 44 1885 Cleveland Grover
## 1889-Harrison 1355 4744 157 1889 Harrison Benjamin
## 1893-Cleveland 823 2135 58 1893 Cleveland Grover
## 1897-McKinley 1236 4383 130 1897 McKinley William
## 1901-McKinley 857 2449 100 1901 McKinley William
## 1905-Roosevelt 404 1089 33 1905 Roosevelt Theodore
## 1909-Taft 1436 5844 159 1909 Taft William Howard
## 1913-Wilson 661 1896 68 1913 Wilson Woodrow
## 1917-Wilson 549 1656 59 1917 Wilson Woodrow
## 1921-Harding 1172 3743 148 1921 Harding Warren G.
## 1925-Coolidge 1221 4442 196 1925 Coolidge Calvin
## 1929-Hoover 1086 3895 158 1929 Hoover Herbert
## 1933-Roosevelt 744 2064 85 1933 Roosevelt Franklin D.
## 1937-Roosevelt 729 2027 96 1937 Roosevelt Franklin D.
## 1941-Roosevelt 527 1552 68 1941 Roosevelt Franklin D.
## 1945-Roosevelt 276 651 26 1945 Roosevelt Franklin D.
## 1949-Truman 781 2531 116 1949 Truman Harry S.
## 1953-Eisenhower 903 2765 119 1953 Eisenhower Dwight D.
## 1957-Eisenhower 621 1933 92 1957 Eisenhower Dwight D.
## 1961-Kennedy 566 1568 52 1961 Kennedy John F.
## 1965-Johnson 569 1725 93 1965 Johnson Lyndon Baines
## 1969-Nixon 743 2437 103 1969 Nixon Richard Milhous
## 1973-Nixon 545 2018 68 1973 Nixon Richard Milhous
## 1977-Carter 528 1380 52 1977 Carter Jimmy
## 1981-Reagan 904 2798 128 1981 Reagan Ronald
## 1985-Reagan 925 2935 123 1985 Reagan Ronald
## 1989-Bush 795 2683 141 1989 Bush George
## 1993-Clinton 644 1837 81 1993 Clinton Bill
## 1997-Clinton 773 2451 111 1997 Clinton Bill
## 2001-Bush 622 1810 97 2001 Bush George W.
## 2005-Bush 772 2325 100 2005 Bush George W.
## 2009-Obama 939 2729 110 2009 Obama Barack
## 2013-Obama 814 2335 88 2013 Obama Barack
## 2017-Trump 582 1662 88 2017 Trump Donald J.
##
## Source: /home/paul/Dropbox/code/quanteda/* on x86_64 by paul
## Created: Fri Sep 12 12:41:17 2014
## Notes:
if (require(ggplot2))
ggplot(data=tokenInfo, aes(x=Year, y=Tokens, group=1)) + geom_line() + geom_point() +
scale_x_discrete(labels=c(seq(1789,2012,12)), breaks=seq(1789,2012,12) )
## Loading required package: ggplot2
# Longest inaugural address: William Henry Harrison
tokenInfo[which.max(tokenInfo$Tokens),]
## Text Types Tokens Sentences Year President
## 1841-Harrison 1841-Harrison 1893 9178 210 1841 Harrison
## FirstName
## 1841-Harrison William HenryTools for handling corpus objects
Adding two corpus objects together
The + operator provides a simple method for concatenating two corpus objects. If they contain different sets of document-level variables, these will be stitched together in a fashion that guarantees that no information is lost. Corpus-level medata data is also concatenated.
library(quanteda)
mycorpus1 <- corpus(data_corpus_inaugural[1:5], note = "First five inaug speeches.")
## Warning in corpus.character(data_corpus_inaugural[1:5], note = "First five
## inaug speeches."): Argument note not used.
mycorpus2 <- corpus(data_corpus_inaugural[53:58], note = "Last five inaug speeches.")
## Warning in corpus.character(data_corpus_inaugural[53:58], note = "Last five
## inaug speeches."): Argument note not used.
mycorpus3 <- mycorpus1 + mycorpus2
summary(mycorpus3)
## Corpus consisting of 11 documents.
##
## Text Types Tokens Sentences
## 1789-Washington 626 1540 23
## 1793-Washington 96 147 4
## 1797-Adams 826 2584 37
## 1801-Jefferson 716 1935 41
## 1805-Jefferson 804 2381 45
## 1997-Clinton 773 2451 111
## 2001-Bush 622 1810 97
## 2005-Bush 772 2325 100
## 2009-Obama 939 2729 110
## 2013-Obama 814 2335 88
## 2017-Trump 582 1662 88
##
## Source: Combination of corpuses mycorpus1 and mycorpus2
## Created: Thu May 25 10:37:47 2017
## Notes:subsetting corpus objects
There is a method of the corpus_subset() function defined for corpus objects, where a new corpus can be extracted based on logical conditions applied to docvars:
summary(corpus_subset(data_corpus_inaugural, Year > 1990))
## Corpus consisting of 7 documents.
##
## Text Types Tokens Sentences Year President FirstName
## 1993-Clinton 644 1837 81 1993 Clinton Bill
## 1997-Clinton 773 2451 111 1997 Clinton Bill
## 2001-Bush 622 1810 97 2001 Bush George W.
## 2005-Bush 772 2325 100 2005 Bush George W.
## 2009-Obama 939 2729 110 2009 Obama Barack
## 2013-Obama 814 2335 88 2013 Obama Barack
## 2017-Trump 582 1662 88 2017 Trump Donald J.
##
## Source: /home/paul/Dropbox/code/quanteda/* on x86_64 by paul
## Created: Fri Sep 12 12:41:17 2014
## Notes:
summary(corpus_subset(data_corpus_inaugural, President == "Adams"))
## Corpus consisting of 2 documents.
##
## Text Types Tokens Sentences Year President FirstName
## 1797-Adams 826 2584 37 1797 Adams John
## 1825-Adams 1004 3154 74 1825 Adams John Quincy
##
## Source: /home/paul/Dropbox/code/quanteda/* on x86_64 by paul
## Created: Fri Sep 12 12:41:17 2014
## Notes:Exploring corpus texts
The kwic function (KeyWord In Context) performs a search for a word and allows us to view the contexts in which it occurs:
options(width = 200)
kwic(data_corpus_inaugural, "terror")
##
## [1797-Adams, 1327] fraud or violence, by | terror | , intrigue, or venality
## [1933-Roosevelt, 112] nameless, unreasoning, unjustified | terror | which paralyzes needed efforts to
## [1941-Roosevelt, 289] seemed frozen by a fatalistic | terror | , we proved that this
## [1961-Kennedy, 868] alter that uncertain balance of | terror | that stays the hand of
## [1981-Reagan, 821] freeing all Americans from the | terror | of runaway living costs.
## [1997-Clinton, 1055] They fuel the fanaticism of | terror | . And they torment the
## [1997-Clinton, 1655] maintain a strong defense against | terror | and destruction. Our children
## [2009-Obama, 1646] advance their aims by inducing | terror | and slaughtering innocents, we
kwic(data_corpus_inaugural, "terror", valuetype = "regex")
##
## [1797-Adams, 1327] fraud or violence, by | terror | , intrigue, or venality
## [1933-Roosevelt, 112] nameless, unreasoning, unjustified | terror | which paralyzes needed efforts to
## [1941-Roosevelt, 289] seemed frozen by a fatalistic | terror | , we proved that this
## [1961-Kennedy, 868] alter that uncertain balance of | terror | that stays the hand of
## [1961-Kennedy, 992] of science instead of its | terrors | . Together let us explore
## [1981-Reagan, 821] freeing all Americans from the | terror | of runaway living costs.
## [1981-Reagan, 2204] understood by those who practice | terrorism | and prey upon their neighbors
## [1997-Clinton, 1055] They fuel the fanaticism of | terror | . And they torment the
## [1997-Clinton, 1655] maintain a strong defense against | terror | and destruction. Our children
## [2009-Obama, 1646] advance their aims by inducing | terror | and slaughtering innocents, we
## [2017-Trump, 1119] civilized world against radical Islamic | terrorism | , which we will eradicate
kwic(data_corpus_inaugural, "communist*")
##
## [1949-Truman, 838] the actions resulting from the | Communist | philosophy are a threat to
## [1961-Kennedy, 519] -- not because the | Communists | may be doing it,In the above summary, Year and President are variables associated with each document. We can access such variables with the docvars() function.
# inspect the document-level variables
head(docvars(data_corpus_inaugural))
## Year President FirstName
## 1789-Washington 1789 Washington George
## 1793-Washington 1793 Washington George
## 1797-Adams 1797 Adams John
## 1801-Jefferson 1801 Jefferson Thomas
## 1805-Jefferson 1805 Jefferson Thomas
## 1809-Madison 1809 Madison James
# inspect the corpus-level metadata
metacorpus(data_corpus_inaugural)
## $source
## [1] "/home/paul/Dropbox/code/quanteda/* on x86_64 by paul"
##
## $created
## [1] "Fri Sep 12 12:41:17 2014"
##
## $notes
## NULL
##
## $citation
## NULLMore corpora are available from the quantedaData package.
