How to make a dictionary: WS 2008-9

Evaluating lexical resources

Overview

Credibility of results
Perspectives on lexical resources
Common evaluation criteria
Citing dictionaries
Dictionary evaluation matrix

Credibility of results

Science: results have to be reproducible
Conditions need to be defined
Plagiarism and fraud in science
Publish not only interpretation but also results

Perspectives on lexical resources

Human users: wanting to read something
Human users: showing off
Computer applications: providing lexical service to humans
Computer applications: providing lexical service to computer applications
Production oriented
Marketing
Lexical resources: often belonging to more than one class
Perspectives: orthogonal dimensions of description

Common evaluation criteria

"240,000 references, 170,000 numbered definitions"

"15,000 stems with all derivations, all defined by genus proximum et differentia specifica"

"15,000,000 information nodes"

What does this mean?
Which one of those resources is the best?
Where do you see problems with these figures?
Which one do you buy?

Counting in lexicons

Unclear counting:

colour

restricted wavelength of light reflected from an object and perceived by the eye

color

see colour
Polysemous words or homonyms?

bank

1. financial institution for persons to burn money
2. financial institution issuing money

bank

part of the river
Lexemes: Derivations in different articles?

Evaluation criteria

User profile
Size
Reliability testing/sample testing
Democratic test
Methodology
Age
Blurb

Evaluation by user profile

Background and knowledge of potential users
Problem: often schism between claim and provided information
Example: Duden and orthography
Often knowledge required which is not explicitly mentioned
Example: U.S. print dictionaries and "pronunciation"
Semantic fields: not intersection free

Evaluation by size

Number of pages
Average length of definition
Number of pictures
Number of articles
Problem: Methodology of counting: homonyms/polyemous words, derivations
Figures provide only a vague idea of the size and should not (for researchers: must not) be overestimated.

Evaluation by sample tests/reliability

Take a small set of words (selected on which grounds?)
Comparative evaluation with reference to testset
In formal terms: using a test set of words, drawing a sample from a bag of words
Useful for understanding the structure and content of a given resource

Democratic test

Ridiculous (?)
Selection in a bookshop based on sales expectation
Sales figures
Taste of the salesperson

Evaluation by methodology

Method of lexicon creation
Introspection
Empirical:
- Corpus based
- Questionnaire based
Not a single method in real word dictionaries

Evaluation by Age

The newer the better?
But: diachronic perspective
Representation of language change
Outdating of print resources
Resources with constant/undisputed content

Evaluation by blurb

Most often used evaluation strategy (to be disputed)
Text usually written by marketing person (!)
Hints on some of the other criteria (without methodological soundness)

Citing dictionaries

Why would you need to cite dictionaries?
How do you cite a paper/book/chapter?
Quote or paraphrase?
Why do you think scientist cite?

Citing dictionaries

Purpose:
- crediting other person's ideas
- Using other's authority and expertise
- Making an argument reproducible
- Pointing to the credibility of somebody else's results
Who do you quote when citing a dictionary article?
- Transitive trust: the article is trusted because the user trusts the editorial board, which trusted a subject matter expert to know the state of art
- Who deserves this trust?

Citing dictionaries (cont)

Lexicon articles should not be cited unless
- The authorship of the article is given
- They are used anecdotically
- They illustrate the structure/content of a lexicon
Consensus among experts: authorship attributed to a group of authors
If you really need to cite, include in your reference:
- Title
- Editor
- Edition
- Year
- Publisher
- Place of publishing
- Lexical headword/lexeme
- URL

Dictionary evaluation matrix

Contrastive approach: form of comparing analysis
One axis: criteria
Other axis: items to be compared
Example criteria: Number of words in controlled vocabulary, number of pages/lexion articles, salsesrank, ...
Example items to compare: Oxford Advance Learner's Dictionary, Longman Pronunciation Dictionary, Collins German dictionary, wikipedia.org, ...

Task

Which one of these dictionaries is the best?
Why?
Why not?

Evaluation Matrix

Dictionary/Criterion	Oxford Advanced Learners Dictionary	Oxford English Dictionary	Duden Deutsche Rechtschreibung	Longman Pronunciation Dictionary	Collins German dictionary	en.wikipedia.org	en.wiktionary.org	www.beolingus.de	IATE (iate.europa.eu)
Required language/controlled vocabulary	English/3500 words	English/na	German/na	English	English or German	English, available in other languages In independent projects/na	English/na	English or German	One of 24 EU languages
Writing system/Pronunciation	Latin/IPA	Latin/na	Latin/na	Latin/IPA	Latin/IPA	Latin/na	Latin/na	Latin/idiosyncratic transcription system	standard script of the languages, most Latin/na
User Model	Some knowledge of English, used for definitions, syntactic use, orthography, some pronunciation	References for a word in literature	Principles of German orthography required to create hypothesis	Only for pronunciation, orthography has to be clear	Orthography of source language, disambiguation of homonyms requires knowledge of target language	Advanced knowledge of English	Advanced knowledge of English	Orthography of source language, disambiguation of homonyms requires knowledge of target language	Fluency in source and target language, some experience of subject fields or time to get into subject field
Authors	List of editors but no authorship for individual articles.	List of editors but no authorship for individual articles.	Reference to an editorial board, but no authorship for individual articles.	John C. Wells, qualification given	List of editors but no authorship for individual articles.	Every article has an editorial history: all contributors are listed in the order of participation. Some authors use pseudonyms; characterization of some authors and clear names available.	Every article has an editorial history: all contributors are listed in the order of participation. Real names available for most contributors; contributor characterization available.	List of contributors without profile	Technical translators of European institutions, not explicitly mentioned
Age of article	na (only publishing date of book)	na (only publishing date of book)	na (only publishing date of book)	na (only publishing date of book)	na (only publishing date of book)	Available with date and time in the editing history of each article	Available with date and time in the editing history of each article	na	na
Languages included	en	en	de	en	de/en	en	en	de/en, others available	official languages of the EU, currently 24 languages
Number of Articles as given by the source (online sources accessed: 2008-07-16 about 11 am, GMT)	about 63000		about 130,000		280,000	2,458,045	859,654		about 8.4 million headwords
Amazon.com sales rank/as of July 16th 2008 (hardcover if available)	PB 131,921	189,588	6,535,371	298,303	1,608,142	na	na	na	na

Last homework

Create an evaluation matrix for dictionaries! Evaluate 5 of your dictionaries.
Find examples of "pseudo objective" blurb information. To what does this refer in the dictionary? Where are the problems?
Write an essay about

Class website