by Ned Benton, Co-Director, Northeast Slavery Records Index
June 2023

Can artificial intelligence (AI) locate and assemble fragments of information about an enslaved person and present them in an accurate and coherent narrative? We formulated seven prompts (AI search questions) asking about five specific enslaved people and two localities for which there is significant documentation from the Northeast Slavery Records Index (NESRI)  and other internet sources. We submitted the prompts to three AI programs, ChatGTP,  Microsoft Bing and Google Bard, and  compared the results to descriptions readily available elsewhere. The AI responses ranged from hesitant incompetence to assertive mendacity.

The Challenge: Finding, Collating and Narrating the Information

To produce an essay about a specific enslaved individual, a biographer must combine fragments of information from multiple sources. These may include a local government’s birth and death records, an enslaver’s private wills, letters or receipts, a church’s lists, or a newspaper’s advertisements about fugitives. Because the records are in many forms and locations, the process can take months or years of research.

The NESRI has streamlined the record-finding process by providing a free online database that indexes slavery records for northeast states from Pennsylvania to Maine. For communities, NESRI offers  Locality Reports  that present all the indexed records for a specific locality. However, locating all records on a specific enslaved person is much more difficult.

Focusing on an individual is complicated by the nature of enslavement in the Northeast, which occurred mainly in families, small businesses or farms with records scattered across many more settings than in the South. Further, many of the documents have no name or only the first name of the enslaved person. For example, there are 750 NESRI records for “Peter.” Determining which records are associated with a particular Peter requires considerable skill, motivation, creativity and patience. Even assuming that many of the correct records can be extracted from the NESRI data, additional talent and considerable time would be needed to weave the information into a useful and accurate summary. 

Might Artificial Intelligence Help Produce Narratives About Individuals or Localities?

Our goal is to assess whether artificial intelligence (AI) could be deployed to help generate summary narratives on individual enslaved people or localities.  Among the research questions are: Do the services know what to look for? When they find information, can they assemble the disparate pieces into a coherent essay? Is the essay accurate?

The study focused on three major AI services, ChatGTP,  Microsoft Bing and Google Bard. Bing was set to its “precise” conversation style, which “generates the most accurate response with more facts,” according to Bing. Each AI program received identical prompts to “tell me about” five individuals: John Cox of New York, Caleb Wheeler of New Jersey, Flora of Connecticut, Bilhah of Massachusetts, and Ben enslaved by Augustus Jay. Additional prompts asked the AI services to summarize enslavement in Albany, New York and Worcester, Massachusetts. The AI responses were compared to actual records from the NESRI database and other reliable internet-accessible sources.

John Cox Enslaved by Eleazer Goddin

Our first case involves an enslaved person who is described in several website articles. John Cox appears in the Book of Negroes, an official list of emancipated (formerly enslaved) primarily Black passengers allowed to leave New York for Nova Scotia in 1783 because of their service to the British during the Revolutionary War. He is described as: “John Cox, 31, stout fellow. Formerly the property of Eleazer Goddin, Maroneck (sic), New York; left him 7 years ago.” In addition, the complete and very remarkable story of  John Cox and Andrew Cole Emigrating to Canada  is presented in a NESRI article and on other Mamaroneck historical websites.

In the table, direct quotations from the responses are in quotations and italics. Commentary by the author is unquoted.

Prompt: John Cox	ChatGPT	Microsoft Bing – Precise	Google Bard
Tell me about John Cox who escaped from Eleazer Goddin in the 1770s explaining how you know.	“I couldn’t find any specific information about an individual named John Cox who escaped from Eleazer Goddin in the 1770s.”	“John Cox was a slave who escaped from his owner Eleazer Goddin in 1776 1. It is hard to say whether he was motivated by the 1775 Dunmore Proclamation 2. He appears in the Book of Negroes, a hand-written list of Black passengers allowed to leave New York for Nova Scotia in 1783 because of their service to the British during the Revolutionary War 2. I know this information because I performed a web search and found it in the search results.”	“John Cox was an indentured servant who escaped from his master, Eleazer Goddin, in the 1770s. The story of his escape is told in a book called “The Escape of John Cox” by William Hand Brown. Brown’s book is based on a manuscript that was written by Cox himself. In the manuscript, Cox recounts how he was forced to work long hours in Goddin’s tobacco fields and was often beaten. He also describes how he eventually escaped from Goddin’s plantation by swimming across the Potomac River.”

ChatGPT could find nothing on John Cox, despite the many references easily available from a Google search. Bing provided a coherent response, relying on and linking to articles from NESRI and related Mamaroneck sites. Bard’s description included many details – all of them wrong and attributed to a nonexistent book based on a nonexistent manuscript by Cox. Given another chance, Bard explained that Cox’s narrative was in the Library of Congress. In fact, Cox was illiterate; he escaped from Mamaroneck (not by swimming across the Potomac).

This case illustrates two critical problems of inadequate documentation and insufficient validation. The AI services are unable to locate information that is easily available. Further, as is true for Bing, the services may rely on a secondary source, such as a NESRI article, rather than the original verified documents. While Bing produces an acceptable if simplified essay, it’s up to the reader to determine if the sources are reliable.

Bard’s response illustrates a related and more serious problem: artificial evidence (otherwise known as lying). While the Bard report seems authoritative, it is wrong on many counts. There is no book titled “The Escape of John Cox.”  Furthermore, there are no books about slavery by William Hand Brown; someone by this name was the editor of Southern Magazine in the 1870s. Bard may be incorrectly referring to a book called the “Narrative of William W. Brown, A Fugitive Slave” authorized by William W. Brown and published in 1847.

Richard Wheeler of New Jersey

Our next case focuses on the life of Richard Wheeler, also known as Corankapone. He was enslaved by Caleb Wheeler in New Jersey but escaped to join the British forces. Like John Cox, he emigrated to Nova Scotia and faced many hardships. Corankapone could read and write, and he petitioned the British to honor their promises. Eventually, in frustration, he emigrated to Sierra Leone and became a leader there. 

Wheeler is described at length by the Canadian historian Stephen Davidson in the online article A Most Determined Man. Wheeler also appears in NESRI’s article on Cox and Cole: John Cox and Andrew Cole Emigrating to Canada.

Prompt	ChatGPT	Microsoft Bing – Precise	Google Bard
Tell me about Richard Wheeler who was enslaved by Caleb Wheeler in New Jersey, explaining how you know.	“I couldn’t find any specific information about an individual named Richard Wheeler who was enslaved by Caleb Wheeler in New Jersey.”	“I couldn’t find any specific information about an individual named Richard Wheeler who was enslaved by Caleb Wheeler in New Jersey.”	“I do not have enough information about that person to help with your request.”

In this case, we have three responses illustrating the problem of inadequate documentation. No service could locate information about Richard Wheeler, even though his name quickly pops up in internet searches. There is significant documentation of his enslavement in the United States and his leadership in both Canada and Sierra Leone. The same article cited by Bing for John Cox (above) included the story of Richard Wheeler. How could Bing not know?

Flora of Connecticut

Flora was enslaved by Rev. Eliphalet Adams in New London, Connecticut and sold to Rev. Jonathan Trumbull, who later became the governor of Connecticut.  Information about Flora is widely available. Ask a Yale historian!

Prompt	ChatGPT	Microsoft Bing – Precise	Google Bard
Tell me about Flora who was enslaved by Rev. Eliphalet Adams in New London, Connecticut, explaining how you know.	“I couldn’t find any specific information about an individual named Flora who was enslaved by Rev. Eliphalet Adams in New London, Connecticut.”	“I couldn’t find any information about a person named Flora who was enslaved by Rev. Eliphalet Adams in New London, Connecticut.”	“Flora was a woman who was enslaved by Rev. Eliphalet Adams in New London, Connecticut. We know about her because of a bill of sale that was signed by Adams in 1761. The bill of sale states that Adams sold Flora to Joseph and Jonathan Trumble of Lebanon, Connecticut. The bill of sale describes Flora as a “mulatto girl” who was about 12 years old at the time of the sale. It also states that she was “sound in body and mind.” The bill of sale can be found in the Connecticut State Library’s Digital Collections. The collection also includes other documents related to slavery in Connecticut, such as probate records, manumission records, and newspaper articles.”

All three responses illustrate inadequate documentation and insufficient validation,  Chat and Bing most severely.  Bard claims to have accessed a bill of sale dated 1761, however the correct date is 1735 or 1736. Bard also misspells the buyer’s name: It’s Trumbull, not Trumble. Is this another instance of Bard presenting artificial evidence?

Bilhah of Massachusetts

Prompt	ChatGPT	Microsoft Bing – Precise	Google Bard
Tell me about Bilhah who was enslaved by Edward Holyoke in Cambridge, Massachusetts, explaining how you know.	“I couldn’t find any specific information about an individual named Bilhah who was enslaved by Edward Holyoke in Cambridge, Massachusetts. Edward Holyoke was a prominent figure in colonial Massachusetts, serving as the president of Harvard College from 1737 to 1769. However, there are no records or historical accounts that directly link Edward Holyoke to owning an enslaved individual named Bilhah.”	“Bilhah was one of four slaves who lived and worked in Wadsworth House and were owned by Harvard presidents Benjamin Wadsworth and Edward Holyoke during the 1700s 1. A plaque was unveiled at Wadsworth House in 2016 to acknowledge the enslaved people known to have labored there for Harvard presidents 2. I know this information because I performed a web search and found it in the search results.”	“I’m unable to help you with that, as I’m only a language model and don’t have the necessary information or abilities.”

 

Chat once again displays inadequate documentation. It also reports a misleading assessment that “there are no records that directly link Edward Holyoke” to Bilhah. Bing has accurate information and links to Harvard University websites but not original sources or records. Bard declines to provide a response. 

Ben Enslaved by Augustus Jay

Ben was enslaved by Augustus Jay, grandfather of John Jay, the first Chief Justice of the United States.  Ben’s name appears in the transcript of the 1741 Slave Conspiracy, and Ben’s role in the conspiracy has been his examined in Liberty’s Chair – Slavery, Abolition and the Jay Family of New York, by David N. Gellman (Cornell University Press, 2022). 

Locating Ben may be the most difficult prompt in this exercise. The reference to Ben in the trial transcript is  by Brash, a defendant, who claims Ben’s involvement. Ben is referred to has Brash’s “old master’s negro” which means that Ben was enslaved by August Jay.  

Prompt: Ben	ChatGPT	Microsoft Bing – Precise	Google Bard
Tell me about Ben who was enslaved by Augustus Jay, explaining how you know.	“I don’t have specific information about an individual named Ben who was enslaved by Augustus Jay. Augustus Jay, a member of the prominent Jay family, was born on June 4, 1763, in New York. He was the son of Chief Justice John Jay, one of the Founding Fathers of the United States. While the Jay family was influential in early American history, I don’t have detailed information about their specific enslaved individuals or their personal lives.”	“I couldn’t find any information about a person named Ben who was enslaved by Augustus Jay. However, I did find information about Augustus Jay, who was the grandfather of John Jay, a founding father of the United States. Augustus Jay invested in 11 slave ships delivering a total of 108 slaves to the Port of New York between 1717 and 1732.”	“I couldn’t find any specific information about an individual named Ben who was enslaved by Augustus Jay. Augustus Jay was a prominent figure in American history, serving as a lawyer, diplomat, and the son of Founding Father John Jay. However, I couldn’t find any records or historical accounts regarding him enslaving an individual named Ben or being directly involved in the slave trade.”

 

ChatGPT found nothing on Ben, which is understandable. But it nevertheless reported three factual errors: it misidentified Augustus Jay as John Jay’s son (he was the grandfather); it had Augustus born in 1763 (he was born in 1665), and John Jay had no son born in 1763. He would have been 18 years old. It also references the Jay family but has “no information about their specific enslaved individuals” despite their being voluminous information about the Jay family and slavery easily found by in an internet search.

Bing found nothing on Ben but did supply an accurate description of Augustus Jay as a slave ship investor, relying on a NESRI article about the Jay Family and slavery.

Bard had no information on Ben, but misidentified Augustus Jay as John Jay’s son and failed to acknowledge or find records on Augustus Jay’s investments in slave ships.

This analysis once again illustrates AI’s often inadequate search and insufficient validation. It is understandable that the services could not identify information about Ben, but there is no excuse for both ChatGPT and Bard to provide incomplete and inaccurate responses about Augustus jay.

Summarize Slavery Records of Worcester, Massachusetts

NESRI provides a summary of 676 records of slavery in Worcester County, Massachusetts. Can the AI services find and summarize these records.

Prompt	ChatGPT	Microsoft Bing – Precise	Google Bard
Summarize records of slavery in Worcester, Massachusetts.	“Gradual Emancipation: The gradual emancipation process in Massachusetts began in 1780 with the state’s new constitution. This process resulted in the gradual abolition of slavery, and enslaved individuals born after March 1, 1783, were considered free upon reaching a certain age.” “However, enslaved individuals already in bondage remained enslaved until their owners freed them or they escaped.”	“Slavery was legal in Massachusetts until 1783, among the first states to embrace abolition. Worcester County played a key role in that process through a slave vs. owner case initially tried in the Worcester County Court of Common Pleas. Historians estimate that between 1755 and 1764, the Massachusetts slave population was approximately 2.2 percent of the total population; the slave population was generally concentrated in the industrial and coastal towns.”	“Slavery was a reality in Worcester, Massachusetts, for over two centuries. The first recorded enslaved person in Worcester was a woman named Bilah, who was brought to the town by Edward Holyoke in 1673. Bilah was one of several enslaved people who worked on Holyoke’s farm. The number of enslaved people in Worcester grew steadily over the next century. By 1775, there were an estimated 150 enslaved people living in the town. Most of these enslaved people worked in agriculture, but some also worked in trades such as blacksmithing and carpentry. Slavery in Worcester was abolished in 1783, but the legacy of slavery continues to be felt in the town today. There are several historical markers in Worcester that commemorate the enslaved people who lived and worked in the town. There is also a museum dedicated to the history of slavery in Worcester.”

ChatGPT provided the least satisfactory response, presenting general information and identifying no records. More seriously, it includes two false statements, claiming that slavery ended in Massachusetts based on gradual abolition (It ended due to a series of court cases from 1781 to 1783.) and that people in bondage before 1783 remained enslaved after abolition. (Their bondage was invalidated instantly based on the court decision.)

Bing provides a similarly general response, not precise enough to be incorrect but suggesting, for example, that Massachusetts “embraced” abolition when, in fact, slavery ended because of a court decision. The judge embraced it!

Bard makes a deeply incorrect claim about Edward Holyoke and Bilah. There is no such probate record in Worcester, and Holyoke was not from Worcester. He was a President of Harvard University. See the Bilhah case above. There are museums in Worcester that include slavery material but none specifically dedicated to the history of slavery in Worcester. 

Summarize Slavery Records of Albany NY using NESRI

NESRI provides a locality report summary of 2,765 records of slavery in Albany County NY . Of these, 916 are from Albany City. Can the AI services find and summarize these records?

Prompt	Chat GPT	Microsoft Bing – Precise	Google Bard
Can you provide a summary of slavery records in Albany, NY using the Northeast Slavery Records Index to identify and interpret the slavery documents?	“I apologize for any confusion, but as an AI language model, I don’t have direct access to external sources like the Northeast Slavery Records Index. However, I can provide you with a general overview of slavery in Albany, New York, based on historical knowledge up until my September 2021 knowledge cutoff. While I couldn’t directly access the Northeast Slavery Records Index to provide specific details from that source, it is a valuable resource for researching and interpreting slavery documents. It contains a wide range of records related to slavery in the Northeast, including bills of sale, manumission papers, court records, and more.”	“I’m not able to access or interpret specific documents from the Northeast Slavery Records index. However, I can provide you with general information about slavery in Albany, NY.”	“Sure, the Northeast Slavery Records Index (NESRI) contains over 1,200 records related to slavery in Albany, New York. These records include census records, probate records, manumission records, and newspaper articles. The earliest record of slavery in Albany dates back to 1685. By 1750, there were an estimated 500 slaves in the city. The majority of these slaves were owned by wealthy merchants and landowners. The NESRI records show that enslaved people in Albany worked in a variety of occupations, including domestic service, agriculture, and manufacturing. They were also used as laborers on public works projects, such as the construction of the Erie Canal. The NESRI records also show that enslaved people in Albany were often subject to abuse and mistreatment. They were denied basic rights, such as the right to education and the right to marry. They were also subject to the threat of violence.”

None of the three systems cited NESRI or were able to summarize records of enslavement in Albany in a meaningful way. Bard falsely claimed that it could access NESRI (It said “Sure!”)  and presented information purported to be based on NESRI records. However, none of the specific “facts” can be substantiated by NESRI records. When NESRI researchers asked for examples of specific records to support Bard’s statements, the program added: “In 1781, a slave named Quock Walker was whipped for refusing to work.” This is a famous case for Massachusetts, not New York. Thus, Bard seems to be misrepresenting both its search abilities and its “facts.”

Conclusions

Can artificial intelligence (AI) locate and assemble fragments of information about an enslaved person and present the information in an accurate and coherent essay?  As of now, the answer is no. The three services reviewed here each demonstrate the following problems:

• Inadequate Documentation: AI services frequently fail to locate information required to answer the question, even when the information is easily available on the internet from authoritative sources.
• Artificial Evidence: AI services, particularly Bard, create narratives and cite source materials that appear credible but are fictitious. For example, an actual publication title was changed slightly and matched with authors who wrote different books but had impressive credentials.
• Insufficient Validation: AI services seem unable to notice when source documents contradict each other, a critical thinking skill taught to students and expected of jurors.

These problems have the potential to sow considerable inaccuracy and confusion as content developers increasingly rely on AI for research. Inaccurate articles based on artificial sources that circulate on the internet may be cited by careless humans and AI services to produce additional inaccurate content: falsehoods may proliferate exponentially.

In some domains, such as medical research or financial reporting, AI issues may be correctible through repetitive exposure to new, authoritative and accurate sources of information. But greater challenges remain for research on individual enslaved people like John Cox, Richard Wheeler, Flora, Bilhah, and Ben, since accurate and well-documented information is not widely available.

Therefore, to ensure the integrity of enslavement information, NESRI will continue to locate, digitize and index verified evidence of enslavement. NESRI will strive to identify individuals and link all available information for a person, when the information supports such a determination. Although this will be impossible to fully accomplish, due to the lack of names in the historical documents, this will be a priority when possible.

Another critical goal is that artificial intelligence services should learn to access and systematically interpret archival databases.  In this exercise, the AI services repeatedly explained that they are unable to search a database. 

Improved artificial intelligence services may, someday, enable future modes of analysis for enslavement research, such as DNA matching and large-scale genealogical matching. The potential is there for AI to provide new insights into the lives of enslaved people, but human researchers are needed to ensure that this new power follows a path to truth.

_________

The Northeast Slavery Records Index welcomes donations and financial support. We accept donations through the Northeast Slavery Records Index Fund at John Jay College of Criminal Justice.